This document summarizes an agenda for a Pinterest Engineering meeting. It includes discussions on mobile growth and monetization, deploying and shipping code. Specific topics that will be covered include scaling user education on mobile, growth strategies like user education, monetization through data, and how Pinterest deploys and ships code. Speakers will discuss mobile features, how user growth is driven through education, monetizing user data, and ensuring smooth code deployment.
Band of brothers, building scalable social web apps on windows azure with asp...Marjan Nikolovski
The presentation will be deep dive into how to build scalable social web apps on Windows Azure IAAS by utilizing latest technologies based on document based storage
The last few years have seen a dramatic increase in the number of PowerShell-based penetration testing tools. A benefit of tools written in PowerShell is that it is installed by default on every Windows system. This allows us as attackers to “”live off the land””. It also has built-in functionality to run in memory bypassing most security products.
I will walk through various methodologies I use surrounding popular PowerShell tools. Details on attacking an organization remotely, establishing command and control, and escalating privileges within an environment all with PowerShell will be discussed. You say you’ve blocked PowerShell? Techniques for running PowerShell in locked down environments that block PowerShell will be highlighted as well.
Simple Solutions for Complex Problems - Boulder MeetupApcera
At the NATS June Meetup in Boulder, CO, Tyler Treat of Workiva gives and updated talk on how to embrace simplicity to solve complex infrastructure problems, and how shares more information on how Workiva uses NATS for microservices communication.
You can learn more about NATS at http://www.nats.io
Azure Day Rome Reloaded 2019 - Reactive Systems with Event Gridazuredayit
Event Grid Può essere usato in modo estremamente pervasivo e versatile per costruire architetture serverless reattive, ad esempio nel mondo IoT delle Smart Things, a costo “quasi zero”. Con Event GRid è possibile creare sistemi potenzialmente giganteschi (e impossibili da ricreare on premises), che si autogovernano, espandono (e cambiano!!!) sulla base delle logiche di campo.
Python is a great programming language that works great with Cassandra. If your goal is to get your project into production quickly and iterate fast, Python is a great solution.
These slides are an introduction to the hands on portion from GitHub. https://github.com/rustyrazorblade/python-presentation
Attendees will learn the best web application security practices used by major US government entities. The presentation will cover network configuration, caching, replication, common web application vulnerabilities, and how making these changes will result in better web site performance and user satisfaction. The five most common types of web application attacks will be explained, along with simple ways to prevent them.
Pentest Apocalypse-That's when you hire a pentester, and they walk all over your network. To avoid this, organizations need to be prepared before the first packet is sent in order to get the most value from the tester. There is no excuse for pentesters to find critical vulnerabilities that are six years old on an assessment. And who needs a zero-day when employees leave credentials on wide-open shares? Just like how Doomsday Preppers helps you prepare for the apocalypse, this presentation will help you prepare for, and avoid, a pentest apocalypse by describing common vulnerabilities found on many assessments. Being prepared for common pentester activities will not only help add value to a pentest but will also help prevent attackers from using the same tactics to compromise your organization.
For More Information Please Visit:- http://bsidestampa.net
http://www.irongeek.com/i.php?page=videos/bsidestampa2015/104-pentest-apocalypse-beau-bullock
Band of brothers, building scalable social web apps on windows azure with asp...Marjan Nikolovski
The presentation will be deep dive into how to build scalable social web apps on Windows Azure IAAS by utilizing latest technologies based on document based storage
The last few years have seen a dramatic increase in the number of PowerShell-based penetration testing tools. A benefit of tools written in PowerShell is that it is installed by default on every Windows system. This allows us as attackers to “”live off the land””. It also has built-in functionality to run in memory bypassing most security products.
I will walk through various methodologies I use surrounding popular PowerShell tools. Details on attacking an organization remotely, establishing command and control, and escalating privileges within an environment all with PowerShell will be discussed. You say you’ve blocked PowerShell? Techniques for running PowerShell in locked down environments that block PowerShell will be highlighted as well.
Simple Solutions for Complex Problems - Boulder MeetupApcera
At the NATS June Meetup in Boulder, CO, Tyler Treat of Workiva gives and updated talk on how to embrace simplicity to solve complex infrastructure problems, and how shares more information on how Workiva uses NATS for microservices communication.
You can learn more about NATS at http://www.nats.io
Azure Day Rome Reloaded 2019 - Reactive Systems with Event Gridazuredayit
Event Grid Può essere usato in modo estremamente pervasivo e versatile per costruire architetture serverless reattive, ad esempio nel mondo IoT delle Smart Things, a costo “quasi zero”. Con Event GRid è possibile creare sistemi potenzialmente giganteschi (e impossibili da ricreare on premises), che si autogovernano, espandono (e cambiano!!!) sulla base delle logiche di campo.
Python is a great programming language that works great with Cassandra. If your goal is to get your project into production quickly and iterate fast, Python is a great solution.
These slides are an introduction to the hands on portion from GitHub. https://github.com/rustyrazorblade/python-presentation
Attendees will learn the best web application security practices used by major US government entities. The presentation will cover network configuration, caching, replication, common web application vulnerabilities, and how making these changes will result in better web site performance and user satisfaction. The five most common types of web application attacks will be explained, along with simple ways to prevent them.
Pentest Apocalypse-That's when you hire a pentester, and they walk all over your network. To avoid this, organizations need to be prepared before the first packet is sent in order to get the most value from the tester. There is no excuse for pentesters to find critical vulnerabilities that are six years old on an assessment. And who needs a zero-day when employees leave credentials on wide-open shares? Just like how Doomsday Preppers helps you prepare for the apocalypse, this presentation will help you prepare for, and avoid, a pentest apocalypse by describing common vulnerabilities found on many assessments. Being prepared for common pentester activities will not only help add value to a pentest but will also help prevent attackers from using the same tactics to compromise your organization.
For More Information Please Visit:- http://bsidestampa.net
http://www.irongeek.com/i.php?page=videos/bsidestampa2015/104-pentest-apocalypse-beau-bullock
Chaz Lever, Georgia Institute of Technology
Both the operational and academic security communities have used dynamic analysis sandboxes to execute malware samples for roughly a decade. Network information derived from dynamic analysis is frequently used for threat detection, network policy, and incident response. Despite these common and important use cases, the efficacy of the network detection signal derived from such analysis has yet to be studied in depth. This paper seeks to address this gap by analyzing the network communications of 26.8 million samples that were collected over a period of five years.
Using several malware and network datasets, our large-scale study makes three core contributions. (1) We show that dynamic analysis traces should be carefully curated and provide a rigorous methodology that analysts can use to remove potential noise from such traces. (2) We show that Internet miscreants are increasingly using potentially unwanted programs (PUPs) that rely on a surprisingly stable DNS and IP infrastructure. This indicates that the security community is in need of better protections against such threats, and network policies may provide a solid foundation for such protections. (3) Finally, we see that, for the vast majority of malware samples, network traffic provides the earliest indicator of infection—several weeks and often months before the malware sample is discovered. Therefore, network defenders should rely on automated malware analysis to extract indicators of compromise and not to build early detection systems.
Covert Attack Mystery Box: A few novel techniques for exploiting Microsoft “f...Beau Bullock
Does the blue team got you feeling down because they are on you like Windows Defender on a Mimikatz binary? Have you lost sleep at night because their logging and alerting levels are so well tuned that if they were vocals, auto-tune couldn’t make them any better? Do you like surprises? Well you are in luck!
Over the last few months we’ve been doing a bit of research around various Microsoft “features”, and have mined a few interesting nuggets that you might find useful if you’re trying to be covert on your red team engagements. This talk will be “mystery surprise box” style as we’ll be weaponizing some things for the first time. There will be demos and new tools presented during the talk. So, if you want to win at hide-n-seek with the blue team, come get your covert attack mystery box!
Undine: Turnkey Drupal Development EnvironmentsDavid Watson
Undine is a cross-platform, fully-featured development VM (virtual machine) for Drupalistas of all experience levels. Sponsored by Stevens Institute of Technology, it is a turnkey solution to many of the common pain points encountered in developing for Drupal.
Download Undine: http://drupal.org/project/undine
Using probabilistic data structures in sessions to power personalization and customization in real-time. Examples in Redis and Node.js
Demo code at: https://github.com/stockholmux/qcon-redis-session-store-demo
Presented at QCon SF 2017.
War stories from building the Global Patent Search Network, and why Data folks need to think more about UX and Discovery, and UX folks need to think more about Data.
Abstract:
Reactive applications need to be able to respond to demand, be elastic and ready to scale up, down, in and out—taking full advantage of mobile, multi-core and cloud computing architectures—in real time.
In this talk we will discuss the guiding principles making this possible through the use of share-nothing and non-blocking designs, applied all the way down the stack. We will learn how to deliver systems that provide reactive supply to changing demand.
I gave this talk at React Conf 2014 in London. Recording available here: https://www.youtube.com/watch?v=mBFdj7w4aFA
HBaseCon 2015 General Session: Zen - A Graph Data Model on HBaseHBaseCon
Zen is a storage service built at Pinterest that offers a graph data model of top of HBase and potentially other storage backends. In this talk, Zen's architects go over the design motivation for Zen and describe its internals including the API, type system, and HBase backend.
Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandr...DataStax
The Anti-Entropy process used by nodetool repair is the way of ensuring consistency of data on disk. Over the many years of the Apache Cassandra project it has also been the biggest pain point for teams running Cassandra. With a solid repair process in place you can be confident that deleted data will not come back to life, and that data is fully distributed when nodes fail.
In this talk Alexander Dejanovski, Consultant at The Last Pickle, will explain how Anti-Entropy works and why it should be run on your cluster. He will discuss the different options such as ""primary range"" repair, sub-range repairs, and incremental repair introduced in version 2.1.
He will also introduce additional tools such as the Spotify Reaper and the range repair script, and future optimisations incremental repair could bring to the read path.
About the Speaker
Alexander DEJANOVSKI Consultant, The Last Pickle
Alexander has been working as a software developer for the last 18 years, mainly for the french leader of express shipments. He's been leading there the effort to build a Cassandra based architecture and migrate services to it from traditional RDBMS. He is involved in the Cassandra community through the development of a JDBC wrapper for the DataStax Java Driver. Recently, he joined The Last Pickle as a Cassandra consultant and now helps customers to get the best out of it.
Apache Cassandra is a popular choice for a wide variety of application persistence needs. There are many design choices that can effect uptime and performance. In this talk we'll look at some of the many things to consider from a single server to multiple data centers. Basic understanding of Cassandra features coupled with client driver features can be a very powerful combination. This talk will be an introduction but will deep dive into the technical details of how Cassandra works.
A presentation of Apache TinkerPop's Gremlin language with running examples over the MovieLens dataset. Presented August 19, 2015 at NoSQL NOW in San Jose, California.
This presentation was given on January 17, 2016 at the GraphDay conference in Austin, Texas. The slides demonstrate the use of wave dynamics in graph structures. Moreover, they demonstrate how to implement quantum processes on graph structures.
There is an associated article available at http://arxiv.org/abs/1511.06278 (Quantum Walks with Gremlin).
Chaz Lever, Georgia Institute of Technology
Both the operational and academic security communities have used dynamic analysis sandboxes to execute malware samples for roughly a decade. Network information derived from dynamic analysis is frequently used for threat detection, network policy, and incident response. Despite these common and important use cases, the efficacy of the network detection signal derived from such analysis has yet to be studied in depth. This paper seeks to address this gap by analyzing the network communications of 26.8 million samples that were collected over a period of five years.
Using several malware and network datasets, our large-scale study makes three core contributions. (1) We show that dynamic analysis traces should be carefully curated and provide a rigorous methodology that analysts can use to remove potential noise from such traces. (2) We show that Internet miscreants are increasingly using potentially unwanted programs (PUPs) that rely on a surprisingly stable DNS and IP infrastructure. This indicates that the security community is in need of better protections against such threats, and network policies may provide a solid foundation for such protections. (3) Finally, we see that, for the vast majority of malware samples, network traffic provides the earliest indicator of infection—several weeks and often months before the malware sample is discovered. Therefore, network defenders should rely on automated malware analysis to extract indicators of compromise and not to build early detection systems.
Covert Attack Mystery Box: A few novel techniques for exploiting Microsoft “f...Beau Bullock
Does the blue team got you feeling down because they are on you like Windows Defender on a Mimikatz binary? Have you lost sleep at night because their logging and alerting levels are so well tuned that if they were vocals, auto-tune couldn’t make them any better? Do you like surprises? Well you are in luck!
Over the last few months we’ve been doing a bit of research around various Microsoft “features”, and have mined a few interesting nuggets that you might find useful if you’re trying to be covert on your red team engagements. This talk will be “mystery surprise box” style as we’ll be weaponizing some things for the first time. There will be demos and new tools presented during the talk. So, if you want to win at hide-n-seek with the blue team, come get your covert attack mystery box!
Undine: Turnkey Drupal Development EnvironmentsDavid Watson
Undine is a cross-platform, fully-featured development VM (virtual machine) for Drupalistas of all experience levels. Sponsored by Stevens Institute of Technology, it is a turnkey solution to many of the common pain points encountered in developing for Drupal.
Download Undine: http://drupal.org/project/undine
Using probabilistic data structures in sessions to power personalization and customization in real-time. Examples in Redis and Node.js
Demo code at: https://github.com/stockholmux/qcon-redis-session-store-demo
Presented at QCon SF 2017.
War stories from building the Global Patent Search Network, and why Data folks need to think more about UX and Discovery, and UX folks need to think more about Data.
Abstract:
Reactive applications need to be able to respond to demand, be elastic and ready to scale up, down, in and out—taking full advantage of mobile, multi-core and cloud computing architectures—in real time.
In this talk we will discuss the guiding principles making this possible through the use of share-nothing and non-blocking designs, applied all the way down the stack. We will learn how to deliver systems that provide reactive supply to changing demand.
I gave this talk at React Conf 2014 in London. Recording available here: https://www.youtube.com/watch?v=mBFdj7w4aFA
HBaseCon 2015 General Session: Zen - A Graph Data Model on HBaseHBaseCon
Zen is a storage service built at Pinterest that offers a graph data model of top of HBase and potentially other storage backends. In this talk, Zen's architects go over the design motivation for Zen and describe its internals including the API, type system, and HBase backend.
Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandr...DataStax
The Anti-Entropy process used by nodetool repair is the way of ensuring consistency of data on disk. Over the many years of the Apache Cassandra project it has also been the biggest pain point for teams running Cassandra. With a solid repair process in place you can be confident that deleted data will not come back to life, and that data is fully distributed when nodes fail.
In this talk Alexander Dejanovski, Consultant at The Last Pickle, will explain how Anti-Entropy works and why it should be run on your cluster. He will discuss the different options such as ""primary range"" repair, sub-range repairs, and incremental repair introduced in version 2.1.
He will also introduce additional tools such as the Spotify Reaper and the range repair script, and future optimisations incremental repair could bring to the read path.
About the Speaker
Alexander DEJANOVSKI Consultant, The Last Pickle
Alexander has been working as a software developer for the last 18 years, mainly for the french leader of express shipments. He's been leading there the effort to build a Cassandra based architecture and migrate services to it from traditional RDBMS. He is involved in the Cassandra community through the development of a JDBC wrapper for the DataStax Java Driver. Recently, he joined The Last Pickle as a Cassandra consultant and now helps customers to get the best out of it.
Apache Cassandra is a popular choice for a wide variety of application persistence needs. There are many design choices that can effect uptime and performance. In this talk we'll look at some of the many things to consider from a single server to multiple data centers. Basic understanding of Cassandra features coupled with client driver features can be a very powerful combination. This talk will be an introduction but will deep dive into the technical details of how Cassandra works.
A presentation of Apache TinkerPop's Gremlin language with running examples over the MovieLens dataset. Presented August 19, 2015 at NoSQL NOW in San Jose, California.
This presentation was given on January 17, 2016 at the GraphDay conference in Austin, Texas. The slides demonstrate the use of wave dynamics in graph structures. Moreover, they demonstrate how to implement quantum processes on graph structures.
There is an associated article available at http://arxiv.org/abs/1511.06278 (Quantum Walks with Gremlin).
Deletes Without Tombstones or TTLs (Eric Stevens, ProtectWise) | Cassandra Su...DataStax
Deleting data from Cassandra has several challenges, and existing solutions (tombstones or TTLs) have limitations that make them unusable or untenable in certain circumstances. We'll explore the cases where existing deletion options fail or are inadequate, then describe a solution we developed which deletes data from Cassandra during standard or user-defined compaction, but without resorting to tombstones or TTL's.
About the Speaker
Eric Stevens Principal Architect, ProtectWise, Inc.
Eric is the principal architect, and day one employee of ProtectWise, Inc., specializing in massive real time processing and scalability problems. The team at ProtectWise processes, analyzes, optimizes, indexes, and stores billions of network packets each second. They look for threats in real time, but also store full fidelity network data (including PCAP), and when new security intelligence is received, automatically replay existing network history through that new intelligence.
This talk argues that the future of data query/analytic languages will be all about embedding the language into the native programming language of the developer. As an example of this style, the Gremlin graph traversal language is presented. Gremlin can be represented in any programming language that supports function composition and function nesting. The language representation is then compiled to Gremlin bytecode to ultimately be executed by the/a Gremlin graph traversal machine. This enables both the Gremlin language and machine to be agnostic to the execution language.
HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBaseHBaseCon
As the operator of the dominant messenger application in South Korea, KakaoTalk has more than 170 million users, and our ever-growing graph has more than 10B edges and 200M vertices. This scale presents several technical challenges for storing and querying the graph data, but we have resolved them by creating a new distributed graph database with HBase. Here you'll learn the methodology and architecture we used to solve the problems, compare it another famous graph database, Titan, and explore the HBase issues we encountered.
Clock Skew and Other Annoying Realities in Distributed Systems (Donny Nadolny...DataStax
You write with QUORUM, you read with QUORUM. You're safe, right?
Although it may seem that way, you could read a different value than the one you wrote - even if nobody else wrote after you. One way this can happen is if the time on the machines in your cluster is not synchronized closely enough. This is called clock skew, and is just one of the ways you'll see that this anomaly can occur.
In this talk we'll dive in to how Cassandra handles conflicting data, walk through several weird and seemingly impossible situations that can happen (both with and without clock skew), and see what we can do to work around them.
About the Speaker
Donny Nadolny Senior Developer, PagerDuty
Donny Nadolny is a Scala developer at PagerDuty, working on improving the reliability of their backend systems. He spends a large amount of time investigating problems experienced with distributed systems like Cassandra and ZooKeeper.
Most Cassandra usages take advantage of its exceptional performance and ability to handle massive data sets. At PagerDuty, we use Cassandra for entirely different reasons: to reliably manage mutable application states and to maintain durability requirements even in the face of full data center outages. We achieve this by deploying Cassandra clusters with hosts in multiple WAN-separated data centers, configured with per-data center replica placement requirements, and with significant application-level support to use Cassandra as a consistent datastore. Accumulating several years of experience with this approach, we've learned to accommodate the impact of WAN network latency on Cassandra queries, how to horizontally scale while maintaining our placement invariants, why asymmetric load is experienced by nodes in different data centers, and more. This talk will go over our workload and design goals, detail the resultant Cassandra system design, and explain a number of our unintuitive operational learnings about this novel Cassandra usage paradigm.
Reactive Development: Commands, Actors and Events. Oh My!!David Hoerster
Distributed applications are becoming more popular with the increasing popularity of microservices (however you want to define that term). But the principles of distributed application development are key if you want to build a system that is resilient, responsive, elastic and maintainable. In this workshop, we’ll review the principles of CQRS and the Reactive Manifesto, and how they complement each other. We’ll build an application that can handle a large stream of data, and allow users to still have a responsive experience while interacting with real-time and near-real-time data.
We’ll look at Akka.NET as the workhorse inside your services, and how the principles of CQRS can help with your service-to-service communications.
We’ll also look at how Event Sourcing can aid in managing your domain state, and how an event stream can be used to project data for your system for a number of different uses. We’ll build our own simple event store, but also look at commercially available stores, too.
This session will focus on using Akka.NET along with a few other tools and technologies, such as EventStore and MongoDB. The concepts learned in this session will be applicable to a number of different tools, technologies and languages.
Attackers don’t just search for technology vulnerabilities, they take the easiest path and find the human vulnerabilities. Drive by web attacks, targeted spear phishing, and more are commonplace today with the goal of delivering custom malware. In a world where delivering custom advanced malware that handily evades signature and blacklisting approaches, and does not depend on application software vulnerabilities, how do we understand when are environments are compromised? What are the telltale signs that compromise activity has started, and how can we move to arrest a compromise in progress before the attacker laterally moves and reinforces their position? The penetration testing community knows these signs and artifacts of advanced malware presence, and it is up to us to help educate defenders on what to look for.
The Dirty Little Secrets They Didn’t Teach You In Pentesting Class Chris Gates
Derbycon 2011
This talk is about methodologies and tools that we use or have coded that make our lives and pentest schedule a little easier, and why we do things the way we do. Of course, there will be a healthy dose of Metasploit in the mix.
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Mike Broberg
Use Apache Spark Streaming in with IBM Watson on Bluemix to perform sentiment analysis and track how a conversation is trending on Twitter.
By David Taieb: https://twitter.com/DTAIEB55
Video: https://youtu.be/KLc_wazud3s
Tutorial: https://developer.ibm.com/clouddataservices/sentiment-analysis-of-twitter-hashtags/
Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013Amazon Web Services
Providing a great media consumption experience to customers is crucial to maximizing audience engagement. To do that, it is important that you make content available for consumption anytime, anywhere, on any device, with a personalized and interactive experience. This session explores the power of big data log analytics (real-time and batched), using technologies like Spark, Shark, Kafka, Amazon Elastic MapReduce, Amazon Redshift and other AWS services. Such analytics are useful for content personalization, recommendations, personalized dynamic ad-insertions, interactivity, and streaming quality.
This session also includes a discussion from Netflix, which explores personalized content search and discovery with the power of metadata.
Deep Dive into Amazon ElastiCache Architecture and Design Patterns (DAT307) |...Amazon Web Services
Peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns of our Memcached and Redis offerings and how customers have used them for in-memory operations and achieved improved latency and throughput for applications. During this session, we review best practices, design patterns, and anti-patterns related to Amazon ElastiCache. We also include a demo where we enable Amazon ElastiCache for a web application and show the resulting performance improvements.
Basic PowerShell Toolmaking - Spiceworld 2016 sessionRob Dunn
PowerShell is everywhere. Admit it, even if you don't like change, you've probably needed to run a one-off command or small script in order to accomplish something...whether it was in AD, Exchange, VMWare or something else.
Running a single command is one thing, but what about making a reusable piece of code that anyone can run, or even better, schedule it? Get a report every Monday about drive space, remove old log files every month, report on logon failures...
We're going to take a command that fulfills a 'single-serving' role and turn it into something more dynamic; something that can be run over and over and be both relevant and timely!
Be ready to learn about parameters, basic functions, comment-based help, and other useful techniques - bring your laptop and code along with us!
Let's build a PowerShell tool!
Watch me present this topic via YouTube: https://youtu.be/akTypRvwr7g (video 1 of 2)
Slides from my last presentation at the Cape Town Meteor meetup, on optimising the UI, specifically for Hybrid apps and for Meteor JS hybrid apps.
The main thrust is really more about design patterns, and carefully controlling data management in your mobile app, with great examples of these patterns out in the real world.
see the mobile patterns video here : https://www.youtube.com/watch?v=e6WWX4TF3UI
AD113 Speed Up Your Applications w/ Nginx and PageSpeededm00se
My slide deck from my session, AD113: Speed Up Your Applications with Nginx + PageSpeed, at MWLUG 2015 in Atlanta, GA at the Ritz-Carlton.
For more, see:
- https://edm00se.io/self-promotion/mwlug-ad113-success
- https://github.com/edm00se/AD113-Speed-Up-Your-Apps-with-Nginx-and-PageSpeed
1,2,3 … testing : is this thing on(line)? Meet your new Microsoft Testing toolsNETUsergroupZentrals
Is your environment acting the way you intended it to be, as in do your users see what you wanted them to see?
Is your app breaking under stress or even worse going down when components are acting up (or down in this case)?
In the past people were using Azure Devops Load Testing and related. But we all know some of these services have been deprecated. In this session you will be guided though all the options you have today lining out all the testing capabilities you have in the Microsoft Coding Universe.
Let’s take a stroll through the various options for load, chaos and automated testing in all things Microsoft devops and Azure. In doing so you will get to learn which services to use to improve reliability, performance usability and resilience of the applications you are building.
Mike Martin
As a Microsoft Technical Evangelist, Mike is an Azure goto for ISV’s (independent software vendors). He’s been active in the IT industry for more than 20 years and has performed almost all types of job profiles, going from coaching and leading a team to architecting and systems design and training. Today he’s primarily into the Microsoft Cloud Platform and Application Lifecycle Management. He’s not a stranger to both dev and IT Pro topics, they even call him the perfect hybrid solution.
This presentation provides an introduction to Azure DocumentDB. Topics include elastic scale, global distribution and guaranteed low latencies (with SLAs) - all in a managed document store that you can query using SQL and Javascript. We also review common scenarios and advanced Data Sciences scenarios.
Oscon London 2016 - Docker from Development to ProductionPatrick Chanezon
Docker revolutionized how developers and operations teams build, ship, and run applications, enabling them to leverage the latest advancements in software development: the microservice architecture style, the immutable infrastructure deployment style, and the DevOps cultural model.
Existing software layers are not a great fit to leverage these trends. Infrastructure as a service is too low level; platform as a service is too high level; but containers as a service (CaaS) is just right. Container images are just the right level of abstraction for DevOps, allowing developers to specify all their dependencies at build time, building and testing an artifact that, when ready to ship, is the exact thing that will run in production. CaaS gives ops teams the tools to control how to run these workloads securely and efficiently, providing portability between different cloud providers and on-premises deployments.
Patrick Chanezon offers a detailed overview of the latest evolutions to the Docker ecosystem enabling CaaS: standards (OCI, CNCF), infrastructure (runC, containerd, Notary), platform (Docker, Swarm), and services (Docker Cloud, Docker Datacenter). Patrick ends with a demo showing how to do in-container development of a Spring Boot application on a Mac running a preconfigured IDE in a container, provision a highly available Swarm cluster using Docker Datacenter on a cloud provider, and leverage the latest Docker tools to build, ship, and run a polyglot application architected as a set of microservices—including how to set up load balancing.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
Water scarcity is the lack of fresh water resources to meet the standard water demand. There are two type of water scarcity. One is physical. The other is economic water scarcity.
Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control.
In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including
A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain
A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
28. What are nags?
•Important information which we
occasionally display at the top of
a user’s feed
•Can be a Call to Action “Confirm
your email” or an announcement
“You can now add a map to any
board!”
30. Enter: The Experience Framework
•Each time we reload the home
feed, ask the experience
framework: What should I show
in this nag?
31. Single Nag Manager that relies on the experience
framework to give it the contents to render
Nag Manager
•Experiment with nag messaging, call to actions, images
•Add any new nag dynamically, controlled from the backend.
•Cool-down management - should not see more than one nag in a set
period of time
32. What does that nag data look like?
Handling actions
"bg_img_url_2x" = "http://mobile-
assets.pinterest.com/iphone/nags/invite-
mail@2x.png"
"title_text" = "Pinspire your friends!"
"detailed_text" = "Know someone who'd like
Pinterest? Invite them along."
"button1_text" = "No, Thanks”!
"button1_uri" = ""!
"button2_text" = "Invite Friends"!
"button2_uri" = "pinterest://invite_friends"
33. Handling Actions
•All initialization and presentation of view controllers is handled through a
central “Navigation Manager.”
• Centralizes code to create and present view controllers
• Consistency to other platforms for deep links
•Allows dynamic insertion of nags from the backend without having to write
new client code and submit a new release
40. Experiment between classic user ed and NUX
•When doing experiments where we need to call the network to get the
user’s treatment group, need to make sure we’re not adding perceived
latency
• Structure view controllers in a way that you can asynchronously load in the modules
dependent on the treatment group
• If need to transition to different view controllers, set a time out in which we transition
to the control treatment
• “Be Fast or Fail Fast”
41. 1 step vs. 2 steps vs. 3 steps
Experiment with different versions of NUX
42. Experiment with different versions of NUX
•Backend controls all strings, allowing us to dynamically experiment with
different text (Messaging, titles, calls-to-action)
43. steps = (
{
"continue_button_text" = Continue;
"detailed_text" = "Tap a network to find people who share your interests.";
"follow_button_text" = "Follow selected people";
step = 1;
"title_text" = "First things first";
"total_steps" = 2;
},
{
"completion_message" = "Finding Pins for you...";
"continue_button_text" = "Tap at least {0} more to continue";
"detailed_text" = "Tap whatever you're interested in these days.";
"finish_text" = Finish;
"num_interests" = 5;
"skip_text" = "Pinterest is much more interesting when you tell us what you like.";
step = 2;
"title_text" = "Pick 5 interests";
"total_steps" = 2;
}
);
After signup, request all data for NUX
Enter: Experience Framework
44. Supports dynamic number and order of steps
NUXViewController : UINavigationController
•Maps an array of display data to an array of view controllers
•Protocol method advanceToNextStep called by each child view controller
• Checks the array it keeps for the next view controller to push
JSON dict for Intro
JSON dict for Friend Selector
JSON dict for Interests Selector
NUXIntroViewController
NUXConnectViewController
NUXInterestsViewController
45. Wins from Experience Framework
•Single place in the backend that manages all experiences for all platforms
•Dynamically trigger display of content
•Conflict resolution for educations that touch the same views
•Experiment with flows, messaging, and images
48. Pinterest is a data driven company
•Data matters
• 100+ experiments active at a given point in time
• 1500+ tracked metrics
• 200+ log types
49. We produce a lot of data
•We produce a lot of data
• PBs of data in S3, growing by Tens of TB a day
• Hundreds of production hadoop jobs, processing about half a PB of data each day
52. Kafka 101
•Distributed pub-sub service
•Designed for high throughput
Producer Producer Producer
Consumer Consumer Consumer
Kafka
cluster
53. Anatomy of a topic
•Topic is a category to which messages are published
•Partition is a ‘shard’ of a topic controlling the level of consumption parallelism
•Messages are assigned unique identifiers called offsets
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9 10 11 12
Writes
Partition 0
Partition 1
Partition 2
54. Save the day
•Kafka is optimized for local writes
•Local disk capacity is good for a few
days worth of data
•Data needs to be saved (at least) daily
to long term storage - Amazon S3
55. How soon is “eventually”
•Amazon S3 is a cloud file system
•Eventual consistency model
• No guarantees on when uploaded data will become visible to the readers
• No monotonicity - data available in the past may magically disappear
56. Secor design guidelines
•Objectives:
• Persist Kafka logs to S3
• Cause no data loss
• Work properly with eventual consistency model
•Properties:
• Horizontal scalability
• Fault tolerance
• Customizability
57. No-S3-reads principle
•Secor never reads data from S3
• Lightweight metadata is stored in strongly
consistent state repository
•Strategic choice of file names
• s3n://logs/<topic>/
<generation>_<partition>_<start_offset>
• <generation> represents software compatibility
version
• Inconsistencies introduced by consumer failures
get fixed automatically by file overwrites
58. Date clustering
•Data processing tools rely on date-
partitioned directory structure
• s3n://logs/event/dt=2014-04-04/
•Timestamps extracted from messages on
the fly
•Support for pluggable parsers for thrift,
json, etc.
63. Developing - Ideal state
•able to iterate quickly
•easy errors caught automatically
•easy-to-understand and powerful abstractions
64. Fast iteration
Developing
•Build tasks and dependencies modeled as a graph
•“cumberbatch” watches for changes of file contents
•“orchestrator” knows tasks and dependencies
•build the minimum tasks to heal damage in the graph
•maximize parallelization of tasks
•built on Grunt - access to large library of build tasks
75. Abstractions
Developing
•component framework
• styles are scoped to the component
• DOM access is scoped to the component’s DOM
• “events up, methods down”
• scaffolding script
•live component catalog - discovery of existing components
•autoprefixer, spriting - remove boilerplate
83. Integration problems are caught
Reviewing
•PRs trigger a build and tests - 3 minutes
•latest.pinterest.com is continuously deployed from “head”
•Selenium integration tests run against every deploy
87. Deploying - Ideal state
•code deploys are invisible to users
•frequent and non-disruptive to developers
•immediate rollback when there’s a problem
88. User experience
Deploying
•stickiness to a version
•flip nearly instantaneously between builds
• reduce version thrashing
• worry less about (style mismatches, JS errors due to data format mismatch with
server)
•asset versioning
99. Results
•5 engineers on web team
•all teams at Pinterest developing their own web features on our platform
•components re-used across teams
•2 scheduled deploys a day
•anomalies in key metrics surfaced immediately
•100s of simultaneous experiments