"In recent years, containers have become a key component of modern application design. Increasingly, developers are breaking their applications apart into smaller components and distributing them across a pool of compute resources. It is relatively easy to run a few containers on your laptop, but building and maintaining an entire infrastructure to run and manage distributed applications is hard and requires a lot of undifferentiated heavy lifting.
In this session, we discuss some of the core architectural principles underlying Amazon ECS, a highly scalable, high performance service to run and manage distributed applications using the Docker container engine. We walk through a number of patterns used by our customers to run their microservices platforms, to run batch jobs, and for deployments and continuous integration. We explore the advanced scheduling capabilities of Amazon ECS and dive deep into the Amazon ECS Service Scheduler, which optimizes for long-running applications by monitoring container health, restarting failed containers, and load balancing across containers."
AWS re:Invent 2016: Large-Scale, Cloud-Based Analysis of Cancer Genomes: Less...Amazon Web Services
The PanCancer Analysis of Whole Genomes (PCAWG) project is a large-scale, highly distributed research collaboration designed to identify common patterns of mutations across 2,800 cancer genomes. The use of public and private clouds were instrumental in analyzing this dataset using current best practice containerized pipelines. This session describes the technical infrastructure built for the project, how we leveraged cloud environments to perform the “core” analysis, and the lessons learned along the way.
AWS re:Invent 2016: FINRA: Building a Secure Data Science Platform on AWS (BD...Amazon Web Services
Data science is a key discipline in a data-driven organization. Through analytics, data scientists can uncover previously unknown relationships in data to help an organization make better decisions. However, data science is often performed from local machines with limited resources and multiple datasets on a variety of databases. Moving to the cloud can help organizations provide scalable compute and storage resources to data scientists, while freeing them from the burden of setting up and managing infrastructure.
In this session, FINRA, the Financial Industry Regulatory Authority, shares best practices and lessons learned when building a self-service, curated data science platform on AWS. A project that allowed us to remove the technology middleman and empower users to choose the best compute environment for their workloads. Understand the architecture and underlying data infrastructure services to provide a secure, self-service portal to data scientists, learn how we built consensus for tooling from of our data science community, hear about the benefits of increased collaboration among the scientists due to the standardized tools, and learn how you can retain the freedom to experiment with the latest technologies while retaining information security boundaries within a virtual private cloud (VPC).
AWS re:Invent 2016: Real-Time Data Exploration and Analytics with Amazon Elas...Amazon Web Services
Elasticsearch is a fully featured search engine used for real-time analytics, and Amazon Elasticsearch Service makes it easy to deploy Elasticsearch clusters on AWS. With Amazon ES, you can ingest and process billions of events per day, and explore the data using Kibana to discover patterns. In this session, we use Apache web logs as example and show you how to build an end-to-end analytics solution. First, we cover how to configure an Amazon ES cluster and ingest data into it using Amazon Kinesis Firehose. We look at best practices for choosing instance types, storage options, shard counts, and index rotations based on the throughput of incoming data. Then we demonstrate how to set up a Kibana dashboard and build custom dashboard widgets. Finally, we dive deep into the Elasticsearch query DSL and review approaches for generating custom, ad-hoc reports.
AWS re:Invent 2016: Deep Dive: Amazon EMR Best Practices & Design Patterns (B...Amazon Web Services
Amazon EMR is one of the largest Hadoop operators in the world. In this session, we introduce you to Amazon EMR design patterns such as using Amazon S3 instead of HDFS, taking advantage of both long and short-lived clusters, and other Amazon EMR architectural best practices. We talk about how to scale your cluster up or down dynamically and introduce you to ways you can fine-tune your cluster. We also share best practices to keep your Amazon EMR cluster cost-efficient. Finally, we dive into some of our recent launches to keep you current on our latest features. This session will feature Asurion, a provider of device protection and support services for over 280 million smartphones and other consumer electronics devices. Asurion will share how they architected their petabyte-scale data platform using Apache Hive, Apache Spark, and Presto on Amazon EMR.
AWS re:Invent 2016| HLC301 | Data Science and Healthcare: Running Large Scale...Amazon Web Services
Working with Amazon Web Services “AWS” and 1Strategy, an Advance AWS Consulting partner; the Cambia Health Data Sciences teams have been able to deploy HIPAA compliant and secured AWS Elastic Map Reduce (EMR) data pipelines on the cloud. In this session, we will dive deep into the architectural components of this solution and you will learn how utilizing AWS services has helped Cambia decrease processing time for analytics, increase application flexibility and accelerate speed to production. The second part of the session is going to cover machine learning and its role in reducing cost and improving quality of care. The healthcare community must rely on advanced analytics and machine learning to analyze multiple facets of healthcare data and process it at scale to gain insights on things that matter. You will learn why AWS is a well suited platform for machine learning. We will take you through the steps of building a machine learning model using Amazon ML for a real world problem of predicting patient readmissions.
AWS re:Invent 2016: Building Big Data Applications with the AWS Big Data Plat...Amazon Web Services
Building big data applications often requires integrating a broad set of technologies to store, process, and analyze the increasing variety, velocity, and volume of data being collected by many organizations. In this session, we show how you can build entire big data applications using a core set of managed services including Amazon S3, Amazon Kinesis, Amazon EMR, Amazon Elasticsearch Service, Amazon Redshift, and Amazon QuickSight.
We walk you through the steps of building and securing a big data application using the AWS Big Data Platform. We also share best practices and common use cases for AWS big data services, including tips to help you choose the best services for your specific application.
AWS re:Invent 2016: Zillow Group: Developing Classification and Recommendatio...Amazon Web Services
Customers are adopting Apache Spark ‒ an open-source distributed processing framework ‒ on Amazon EMR for large-scale machine learning workloads, especially for applications that power customer segmentation and content recommendation. By leveraging Spark ML, a set of machine learning algorithms included with Spark, customers can quickly build and execute massively parallel machine learning jobs. Additionally, Spark applications can train models in streaming or batch contexts, and can access data from Amazon S3, Amazon Kinesis, Amazon Redshift, and other services. This session explains how to quickly and easily create scalable Spark clusters with Amazon EMR, build and share models using Apache Zeppelin and Jupyter notebooks, and use the Spark ML pipelines API to manage your training workflow. In addition, Jasjeet Thind, Senior Director of Data Science and Engineering at Zillow Group, will discuss his organization's development of personalization algorithms and platforms at scale using Spark on Amazon EMR.
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...Amazon Web Services
The growing popularity and breadth of use cases for IoT are challenging the traditional thinking of how data is acquired, processed, and analyzed to quickly gain insights and act promptly. Today, the potential of this data remains largely untapped. In this session, we explore architecture patterns for building comprehensive IoT analytics solutions using AWS big data services. We walk through two production-ready implementations. First, we present an end-to-end solution using AWS IoT, Amazon Kinesis, and AWS Lambda. Next, Hello discusses their consumer IoT solution built on top of Amazon Kinesis, Amazon DynamoDB, and Amazon Redshift.
AWS re:Invent 2016: Leveraging Amazon Machine Learning, Amazon Redshift, and ...Amazon Web Services
The Howard Hughes Corporation partnered with 47Lining to develop a managed enterprise data lake based on Amazon S3. The purpose of the managed EDL is to fuse relevant on-premises and third-party data to enable Howard Hughes to answer its most valuable business questions. Their first analysis was a lead-scoring model that uses Amazon Machine Learning (Amazon ML) to predict propensity to purchase high-end real estate. The model is based on a combined set of public and private data sources, including all publicly recorded real estate transactions in the US for the past 35 years. By changing their business process for identifying and qualifying leads to use the results of data-driven analytics from their managed data lake in AWS, Howard Hughes increased the number of identified qualified leads in their pipeline by over 400% and reduced the acquisition cost per lead by more than 10 times. In this session, you will see a practical example of how to use Amazon ML to improve business results, how to architect a data lake with Amazon S3 that fuses on-premises, third-party, and public data sets, and how to train and run an Amazon ML model to attain predictive accuracy.
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)Amazon Web Services
For discovery-phase research, life sciences companies have to support infrastructure that processes millions to billions of transactions. The advent of a data lake to accomplish such a task is showing itself to be a stable and productive data platform pattern to meet the goal. We discuss how to build a data lake on AWS, using services and techniques such as AWS CloudFormation, Amazon EC2, Amazon S3, IAM, and AWS Lambda. We also review a reference architecture from Amgen that uses a data lake to aid in their Life Science Research.
AWS re:Invent 2016: Best Practices for Data Warehousing with Amazon Redshift ...Amazon Web Services
Analyzing big data quickly and efficiently requires a data warehouse optimized to handle and scale for large datasets. Amazon Redshift is a fast, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all of your data for a fraction of the cost of traditional data warehouses. In this session, we take an in-depth look at data warehousing with Amazon Redshift for big data analytics. We cover best practices to take advantage of Amazon Redshift's columnar technology and parallel processing capabilities to deliver high throughput and query performance. We also discuss how to design optimal schemas, load data efficiently, and use work load management.
Web App for Containers + MySQLでコンテナ対応したRailsアプリを作ろう!Yoichi Kawasaki
Web App for Containers は、アプリスタックのホストに Docker コンテナーを使用するため皆さんが今Linux上で利用しているOSSベースのアプリもアプリスタックごとDockerコンテナ化することでそのまま Web App for Containersで利用することができます。本ウェビナーでは簡単なMySQL + Ruby on Rails アプリ を題材に、アプリをコンテナ化し Web App for Containersにデプロイするまでの一連の流れを解説し、CIツールを使った継続的なデプロイ方法についてご紹介します。今回、AzureのフルマネージドMySQLサービスであるAzure DB for MySQLを利用して完全マネージドな環境でのアプリ実行を実現します。
セル生産方式におけるロボットの活用には様々な問題があるが,その一つとして 3 体以上の物体の組み立てが挙げられる.一般に,複数物体を同時に組み立てる際は,対象の部品をそれぞれロボットアームまたは治具でそれぞれ独立に保持することで組み立てを遂行すると考えられる.ただし,この方法ではロボットアームや治具を部品数と同じ数だけ必要とし,部品数が多いほどコスト面や設置スペースの関係で無駄が多くなる.この課題に対して音𣷓らは組み立て対象物に働く接触力等の解析により,治具等で固定されていない対象物が組み立て作業中に運動しにくい状態となる条件を求めた.すなわち,環境中の非把持対象物のロバスト性を考慮して,組み立て作業条件を検討している.本研究ではこの方策に基づいて,複数物体の組み立て作業を単腕マニピュレータで実行することを目的とする.このとき,対象物のロバスト性を考慮することで,仮組状態の複数物体を同時に扱う手法を提案する.作業対象としてパイプジョイントの組み立てを挙げ,簡易な道具を用いることで単腕マニピュレータで複数物体を同時に把持できることを示す.さらに,作業成功率の向上のために RGB-D カメラを用いた物体の位置検出に基づくロボット制御及び動作計画を実装する.
This paper discusses assembly operations using a single manipulator and a parallel gripper to simultaneously
grasp multiple objects and hold the group of temporarily assembled objects. Multiple robots and jigs generally operate
assembly tasks by constraining the target objects mechanically or geometrically to prevent them from moving. It is
necessary to analyze the physical interaction between the objects for such constraints to achieve the tasks with a single
gripper. In this paper, we focus on assembling pipe joints as an example and discuss constraining the motion of the
objects. Our demonstration shows that a simple tool can facilitate holding multiple objects with a single gripper.
【DLゼミ】XFeat: Accelerated Features for Lightweight Image Matchingharmonylab
公開URL:https://arxiv.org/pdf/2404.19174
出典:Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. ascimento: XFeat: Accelerated Features for Lightweight Image Matching, Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
概要:リソース効率に優れた特徴点マッチングのための軽量なアーキテクチャ「XFeat(Accelerated Features)」を提案します。手法は、局所的な特徴点の検出、抽出、マッチングのための畳み込みニューラルネットワークの基本的な設計を再検討します。特に、リソースが限られたデバイス向けに迅速かつ堅牢なアルゴリズムが必要とされるため、解像度を可能な限り高く保ちながら、ネットワークのチャネル数を制限します。さらに、スパース下でのマッチングを選択できる設計となっており、ナビゲーションやARなどのアプリケーションに適しています。XFeatは、高速かつ同等以上の精度を実現し、一般的なラップトップのCPU上でリアルタイムで動作します。
40. ◯☓ゲーム – 今のままだとチートが可能
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X
41. ◯☓ゲーム – 今のままだとチートが可能
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Update:
Turn : Alice
Low-Right : X
43. 修正版◯☓ゲーム
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
44. 修正版◯☓ゲーム
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
45. 修正版◯☓ゲーム
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
58. FPSのリアルタイム分析
PUT "kills" {"game_id":"e4b5","map":"Boston","killer":38,"victim":39,"coord":"274,591,48"}
PUT "kills" {"game_id":"e4b5","map":"Boston","killer":13,"victim":27,"coord":"101,206,35"}
PUT "kills" {"game_id":"e4b5","map":"Boston","killer":38,"victim":39,"coord":"165,609,17"}
PUT "kills" {"game_id":"e4b5","map":"Boston","killer":6,"victim":29,"coord":"120,422,26"}
PUT "kills" {"game_id":"30a4","map":"Los Angeles","killer":34,"victim":18,"coord":"163,677,18"}
PUT "kills" {"game_id":"30a4","map":"Los Angeles","killer":20,"victim":37,"coord":"71,473,20"}
PUT "kills" {"game_id":"30a4","map":"Los Angeles","killer":21,"victim":19,"coord":"332,381,17"}
PUT "kills" {"game_id":"30a4","map":"Los Angeles","killer":0,"victim":10,"coord":"14,108,25"}
PUT "kills" {"game_id":"6ebd","map":"Seattle","killer":32,"victim":18,"coord":"13,685,32"}
PUT "kills" {"game_id":"6ebd","map":"Seattle","killer":7,"victim":14,"coord":"16,233,16"}
PUT "kills" {"game_id":"6ebd","map":"Seattle","killer":27,"victim":19,"coord":"16,498,29"}
PUT "kills" {"game_id":"6ebd","map":"Seattle","killer":1,"victim":38,"coord":"138,732,21"}
We have two players in a round of tic-tac toe. The item storing the data for this particular game of tic tac toe is here stored in DynamoDB.
This is a match between Alice and Bob, and we’re going to have Alice go first.
We can see with this move that Bob is not very good at this game. By playing that, Alice can guarantee a win by playing in the lower-left.
Let’s say Bob realizes that he not good at this game, and wants to come up with some other way to win.
Based on the API calls we’ve sketched out, this would work and Bob would win, or crash the game.
Here all of those will be merged together. UpdateItem lets you pick specific attributes in an item to update, leaving all the rest of the attributes alone.
PutItem on the other hand replaces the whole item, so then it would have been last write wins. That opens another can of worms around being able to “undo” moves, but that’s a different issue that we’ll fix in the same way.
Apply the write only if the values in the item are still what the request expected them to be.
9-10 min
But, only one of those writes will arrive first. Writes to each item are serialized by DynamoDB.
Introduce Redshift Product
Typical data we see developers gathering about the player includes session length, telemetry data, in game data like how long to the first purchase etc, basically any information that will tell you where the game is doing well versus where it is not.
This data tends to be unstructured and so developers often deploy a NoSQL solution to store that data. They will later use a batch based sort job to cleanse the data and move it into a relational database of some sort, likely a DataWareHouse, for analysis.
Many Game Developers use AWS’s NoSQL offering DynamoDB which can handle very high volumes of read and writes and is highly durable. And as always you can install and manage your own NoSQL offering like MongoDB, Cassandra, Couchbase, etc too.
Typical data we see developers gathering about the player includes session length, telemetry data, in game data like how long to the first purchase etc, basically any information that will tell you where the game is doing well versus where it is not.
This data tends to be unstructured and so developers often deploy a NoSQL solution to store that data. They will later use a batch based sort job to cleanse the data and move it into a relational database of some sort, likely a DataWareHouse, for analysis.
Many Game Developers use AWS’s NoSQL offering DynamoDB which can handle very high volumes of read and writes and is highly durable. And as always you can install and manage your own NoSQL offering like MongoDB, Cassandra, Couchbase, etc too.
Typical data we see developers gathering about the player includes session length, telemetry data, in game data like how long to the first purchase etc, basically any information that will tell you where the game is doing well versus where it is not.
This data tends to be unstructured and so developers often deploy a NoSQL solution to store that data. They will later use a batch based sort job to cleanse the data and move it into a relational database of some sort, likely a DataWareHouse, for analysis.
Many Game Developers use AWS’s NoSQL offering DynamoDB which can handle very high volumes of read and writes and is highly durable. And as always you can install and manage your own NoSQL offering like MongoDB, Cassandra, Couchbase, etc too.
Alternative to Kinesis: Kafka
Still good solution – managing is non-trivial