The document discusses rendering visual effects and animation at scale using Amazon Web Services (AWS). It provides examples of companies using AWS for rendering including theme parks, gaming, and manufacturing. It then discusses the workflow components of VFX/animation rendering and challenges of on-premise rendering capacity. The document outlines how AWS provides scalability and faster outputs through rendering in the cloud. It discusses licensing models for cloud rendering, shared file systems, managing cloud infrastructure, and benchmarks showing improved performance of cloud rendering over on-premise. Overall, the document examines the state of cloud rendering on AWS and trends toward more workloads moving fully to the cloud.
4K Media Workflows on AWS By Usman Shakeel of Amzaon AWSETCenter
4K video is fast setting the minimum bar for content capture, be it consumer, prosumer, professional, episodic or theatrical. This in turn is leading to more demanding infrastructure requirements for media production, management and distribution. Consumers now have a variety of viewing devices and expect an optimal viewing experience for all of them, regardless of screen size or bandwidth availability. Media companies need to deliver on that consumer expectation while at the same time optimizing for costs. They need to produce content for the highest quality viewing experience, but have the ability to deliver to a wide variety of quality levels. They need an infrastructure that scales automatically and just works, whether it is a full 4K production workflow, subsequent archive and content management, or a multi-device streaming. In addition to discussing the way in which an agile cloud infrastructure can meet the demands of a 4K workflow, this session will cover cutting edge technologies like remote application visualization and server-less workflow automation that can greatly help today's media companies in the race to serve their gadget savvy consumer.
Cost Effective Rendering in the Cloud with Spot InstancesAmazon Web Services
Usman Shakeel from Amazon Web Services, explains to us how to use AWS Spot Instances to implement low cost video rendering applications and workflows.
This presentation was delivered during the AWS Toronto Media and Entertainment Symposium
Build a Cloud Render-Ready InfrastructureAvere Systems
Webinar presented September 8, 2015
Rendering applications place high-demands on both compute and storage in visual effects infrastructures. With peaks and valleys in the workflow being the norm, leading VFX creators look to the cloud to build infrastructures that provide flexibility to meet ongoing IT management challenges. In this webinar, you’ll hear from industry innovators about the advantages of cloud rendering and how VFX IT leaders are designing this on-demand solution with Avere Systems and Google Cloud Platform. Designed for CTOs, information systems directors, systems engineers and administrators, the content will discuss the initial steps and technical insights of a render-ready hybrid cloud IT architecture.
AWS September Webinar Series - Visual Effects Rendering in the AWS Cloud with...Amazon Web Services
Visual effects rendering has traditionally been a time consuming, resource intensive process. As a result, content producers are moving rendering workloads to the AWS cloud to take advantage of the scalable, on-demand compute resources that can accelerate their rendering workloads.
By attending this webinar, you will learn how to create a scalable rendering infrastructure to grow your farm for any size workload, reduce overall processing time with on-demand and reserve compute instances, and move to a project based cost structure. You will also learn how to implement hybrid rendering workloads using Thinkbox dependency manager.
Learning Objectives:
How to use AWS Cloud to rapidly scale up and down rendering infrastructure to power ThinkBox Deadline software in the cloud for visual effects rendering
Who should attend:
IT administrators, rendering and visual effects professionals
(MED304) The Future of Rendering: A Complete VFX Studio in the AWS Cloud | AW...Amazon Web Services
Today's studios and visual effects companies require massive computing power and large amounts of storage to produce high-end digital scenes and videos. Maintaining the infrastructure required for these jobs is expensive and operationally difficult, plus demand fluctuates day to day. Geographically diverse workforces adds additional complexity to data and content transfer. The low-cost, utility computing model as well as unique virtualization capabilities offered by AWS are well-suited to addressing these challenges. In this session you will learn how to build and deploy a studio-quality, scalable Arnold render farm on AWS with reusable templates. We'll also demonstrate how to run Maya and Deadline remotely with AWS AppStream, and use them to edit scenes and coordinate render jobs entirely in the cloud.
GPU Renderfarm with Integrated Asset Management & Production System (AMPS)Budianto Tandianus
Was presented in GPU Technology Conference 2014 by Dr. Chen Quan.
The presentation recording and the definitive version of the slide can be downloaded from : http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php?searchByKeyword=S4356&searchItems=session_id&submit=
4K Media Workflows on AWS By Usman Shakeel of Amzaon AWSETCenter
4K video is fast setting the minimum bar for content capture, be it consumer, prosumer, professional, episodic or theatrical. This in turn is leading to more demanding infrastructure requirements for media production, management and distribution. Consumers now have a variety of viewing devices and expect an optimal viewing experience for all of them, regardless of screen size or bandwidth availability. Media companies need to deliver on that consumer expectation while at the same time optimizing for costs. They need to produce content for the highest quality viewing experience, but have the ability to deliver to a wide variety of quality levels. They need an infrastructure that scales automatically and just works, whether it is a full 4K production workflow, subsequent archive and content management, or a multi-device streaming. In addition to discussing the way in which an agile cloud infrastructure can meet the demands of a 4K workflow, this session will cover cutting edge technologies like remote application visualization and server-less workflow automation that can greatly help today's media companies in the race to serve their gadget savvy consumer.
Cost Effective Rendering in the Cloud with Spot InstancesAmazon Web Services
Usman Shakeel from Amazon Web Services, explains to us how to use AWS Spot Instances to implement low cost video rendering applications and workflows.
This presentation was delivered during the AWS Toronto Media and Entertainment Symposium
Build a Cloud Render-Ready InfrastructureAvere Systems
Webinar presented September 8, 2015
Rendering applications place high-demands on both compute and storage in visual effects infrastructures. With peaks and valleys in the workflow being the norm, leading VFX creators look to the cloud to build infrastructures that provide flexibility to meet ongoing IT management challenges. In this webinar, you’ll hear from industry innovators about the advantages of cloud rendering and how VFX IT leaders are designing this on-demand solution with Avere Systems and Google Cloud Platform. Designed for CTOs, information systems directors, systems engineers and administrators, the content will discuss the initial steps and technical insights of a render-ready hybrid cloud IT architecture.
AWS September Webinar Series - Visual Effects Rendering in the AWS Cloud with...Amazon Web Services
Visual effects rendering has traditionally been a time consuming, resource intensive process. As a result, content producers are moving rendering workloads to the AWS cloud to take advantage of the scalable, on-demand compute resources that can accelerate their rendering workloads.
By attending this webinar, you will learn how to create a scalable rendering infrastructure to grow your farm for any size workload, reduce overall processing time with on-demand and reserve compute instances, and move to a project based cost structure. You will also learn how to implement hybrid rendering workloads using Thinkbox dependency manager.
Learning Objectives:
How to use AWS Cloud to rapidly scale up and down rendering infrastructure to power ThinkBox Deadline software in the cloud for visual effects rendering
Who should attend:
IT administrators, rendering and visual effects professionals
(MED304) The Future of Rendering: A Complete VFX Studio in the AWS Cloud | AW...Amazon Web Services
Today's studios and visual effects companies require massive computing power and large amounts of storage to produce high-end digital scenes and videos. Maintaining the infrastructure required for these jobs is expensive and operationally difficult, plus demand fluctuates day to day. Geographically diverse workforces adds additional complexity to data and content transfer. The low-cost, utility computing model as well as unique virtualization capabilities offered by AWS are well-suited to addressing these challenges. In this session you will learn how to build and deploy a studio-quality, scalable Arnold render farm on AWS with reusable templates. We'll also demonstrate how to run Maya and Deadline remotely with AWS AppStream, and use them to edit scenes and coordinate render jobs entirely in the cloud.
GPU Renderfarm with Integrated Asset Management & Production System (AMPS)Budianto Tandianus
Was presented in GPU Technology Conference 2014 by Dr. Chen Quan.
The presentation recording and the definitive version of the slide can be downloaded from : http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php?searchByKeyword=S4356&searchItems=session_id&submit=
(CMP404) Cloud Rendering at Walt Disney Animation StudiosAmazon Web Services
"Each year, the technical complexity of making the next great Walt Disney Animation Studios film increases. Animation and Visual FX studios continue to push the bounds of what is possible in computer graphics. This complexity drives rapid technological growth in both computational resources and storage to the point that it exceeds what we can physically provide with our on-premise compute cluster. As a result, we have started to adopt a hybrid approach with the cloud.
This session addresses the hurdles that animation and VFX studios face and focuses on automation of 'disposable' components (specifically infrastructure, licensing, fleet management, data and dependency management in a large-scale batch workload). We apply these general cloud techniques and utilities to an animation/VFX workload and push the limits with a very large scale cloud renderfarm deployment.
The team from Walt Disney Animation Studios walks through how they use cloud technologies to maximize render capacity. Learn how to leverage high-performance storage (like Amazon EFS), Amazon EC2 networking and the latest EC2 Spot features to provide a fully functional renderfarm at production-quality scale."
Amazon Elastic Compute Cloud (Amazon EC2) provides a broad selection of instance types to accommodate a diverse mix of workloads. In this technical session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations.
We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Speaker:
Ian Massingham, AWS Technical Evangelist
source: http://www.sfbayacm.org/?p=1394
The specifics of a cloud’s computing architecture may have an impact on application design. This is particularly important in Infrastructure as a Service (IaaS) cloud environments.
This presentation analyzes aspects of the Amazon EC2 IaaS cloud environment that differ from a traditional datacenter and introduces general best practices for ensuring data privacy, storage persistence, and reliable DBMS backup. Best practices for application robustness and scalability on demand are reviewed and are especially significant in leveraging the full potential of an IaaS cloud. The need for a cloud application management and configuration system is briefly reviewed and two alternate approaches to cloud application management are described (RightScale and Kaavo).
Automated Media Workflows in the Cloud (MED304) | AWS re:Invent 2013Amazon Web Services
Ingesting, storing, processing and delivering a large library of content involves massive complexity. This session walks through sample code that leverages AWS Services to perform all these tasks while coordinating the activities with Amazon Simple Workflow Service (SWF). Along the journey you are introduced to best practices for cost optimization, monitoring, reporting, and exception or error handling. In addition to the sample workflow, a guest speaker from Netflix takes the audience on a deep dive into their “digital supply chain” where you learn how they have automated their processes in moving data all the way from the studios to the last mile. Services covered include Amazon SWF, Amazon Simple Storage Service (S3), Amazon Glacier, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Transcoder, Amazon Mechanical Turk, and Amazon CloudFront.
VMware Cloud on AWS is now generally available and many VMware users are wondering what it can do for them. Despite its name, VMware Cloud on AWS is not much like AWS. In this webinar we will explain the architecture, technical limitations, pricing, and benefits.
Ruben Perez from RenderRocket presented at Amazon Web Services' The Start-Up Project - Los Angeles. RenderRocket provides outsourced 3D animation rendering services to studios and other businesses.
Amazon EC2 allows you to bid for and run spare EC2 capacity, known as Spot instances, in a dynamically priced market. On average, customers save 80% to 90% compared to On Demand prices by using Spot instances. Achieving these savings has historically required time and effort to find the best deals while managing compute capacity as supply and demand fluctuate.
AWS re:Invent 2016 Recap: What Happened, What It MeansRightScale
Get behind the hype and headlines from AWS re:Invent 2016 and find out what it all means to you. We’ll share what’s working for AWS users and highlight which new features and services you’ll want to look at. Whether or not you attended re:Invent, this wrap-up will help you develop your 2017 cloud to-do list.
AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)Amazon Web Services
This mid-level technical session will provide an overview of the techniques that you can use to build high-scalabilty applications on AWS. Take a journey from 1 user to 10 million users and understand how your application's architecture can evolve and which AWS services can help as you increase the number of users that you serve.
Deep learning is an implementation of machine learning that uses neural networks to solve difficult and complex problems, such as computer vision, natural language processing, and recommendations. Due to the availability of deep learning libraries and frameworks, developers have the ability to enhance the capabilities of their applications and projects. In this workshop, you learn how to build and deploy a powerful deep learning framework called MXNet on containers. The portability and resource management benefit of containers means developers can focus less on infrastructure and more on building. The labs start by demonstrating the automation capabilities of AWS CloudFormation to stand up core infrastructure; as an added bonus, you use Spot Fleet to leverage the cost benefits of using Spot Instances, especially for developer environments. Then, you walk through creating an MXNet container in Docker and deploying it with Amazon ECS. Finally, you walk through an image classification demo of MXNet to validate that everything is working as expected. Note: This workshop focuses on containerizing MXNet. The features of MXNet and capabilities of deep learning in general are vast, and there are recorded sessions from re:Invent that dive deeper on these topics. All you need to participate is a laptop and AWS account. Pizza will be provided.
Building Highly Scalable Immersive Media Solutions on AWSETCenter
Immersive media content such as 360 degree video places a unique set of demands on cloud-based infrastructure from a complete end to end solution point of view. A key goal for any solution of this nature is to keep costs low while not impacting availability, scale and compute performance. In this talk we will look at how to solve ingest, processing, storage and delivery of live and on-demand content for immersive media delivery, and present a reference design for 360 degree streaming using best-practice architectural patterns.
Speakers: Konstantin Wilms, Principal Solutions Architect M&E, AWS and Chad Schmutzer, Specialist Solutions Architect SPOT, AWS
(CMP404) Cloud Rendering at Walt Disney Animation StudiosAmazon Web Services
"Each year, the technical complexity of making the next great Walt Disney Animation Studios film increases. Animation and Visual FX studios continue to push the bounds of what is possible in computer graphics. This complexity drives rapid technological growth in both computational resources and storage to the point that it exceeds what we can physically provide with our on-premise compute cluster. As a result, we have started to adopt a hybrid approach with the cloud.
This session addresses the hurdles that animation and VFX studios face and focuses on automation of 'disposable' components (specifically infrastructure, licensing, fleet management, data and dependency management in a large-scale batch workload). We apply these general cloud techniques and utilities to an animation/VFX workload and push the limits with a very large scale cloud renderfarm deployment.
The team from Walt Disney Animation Studios walks through how they use cloud technologies to maximize render capacity. Learn how to leverage high-performance storage (like Amazon EFS), Amazon EC2 networking and the latest EC2 Spot features to provide a fully functional renderfarm at production-quality scale."
Amazon Elastic Compute Cloud (Amazon EC2) provides a broad selection of instance types to accommodate a diverse mix of workloads. In this technical session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations.
We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Speaker:
Ian Massingham, AWS Technical Evangelist
source: http://www.sfbayacm.org/?p=1394
The specifics of a cloud’s computing architecture may have an impact on application design. This is particularly important in Infrastructure as a Service (IaaS) cloud environments.
This presentation analyzes aspects of the Amazon EC2 IaaS cloud environment that differ from a traditional datacenter and introduces general best practices for ensuring data privacy, storage persistence, and reliable DBMS backup. Best practices for application robustness and scalability on demand are reviewed and are especially significant in leveraging the full potential of an IaaS cloud. The need for a cloud application management and configuration system is briefly reviewed and two alternate approaches to cloud application management are described (RightScale and Kaavo).
Automated Media Workflows in the Cloud (MED304) | AWS re:Invent 2013Amazon Web Services
Ingesting, storing, processing and delivering a large library of content involves massive complexity. This session walks through sample code that leverages AWS Services to perform all these tasks while coordinating the activities with Amazon Simple Workflow Service (SWF). Along the journey you are introduced to best practices for cost optimization, monitoring, reporting, and exception or error handling. In addition to the sample workflow, a guest speaker from Netflix takes the audience on a deep dive into their “digital supply chain” where you learn how they have automated their processes in moving data all the way from the studios to the last mile. Services covered include Amazon SWF, Amazon Simple Storage Service (S3), Amazon Glacier, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Transcoder, Amazon Mechanical Turk, and Amazon CloudFront.
VMware Cloud on AWS is now generally available and many VMware users are wondering what it can do for them. Despite its name, VMware Cloud on AWS is not much like AWS. In this webinar we will explain the architecture, technical limitations, pricing, and benefits.
Ruben Perez from RenderRocket presented at Amazon Web Services' The Start-Up Project - Los Angeles. RenderRocket provides outsourced 3D animation rendering services to studios and other businesses.
Amazon EC2 allows you to bid for and run spare EC2 capacity, known as Spot instances, in a dynamically priced market. On average, customers save 80% to 90% compared to On Demand prices by using Spot instances. Achieving these savings has historically required time and effort to find the best deals while managing compute capacity as supply and demand fluctuate.
AWS re:Invent 2016 Recap: What Happened, What It MeansRightScale
Get behind the hype and headlines from AWS re:Invent 2016 and find out what it all means to you. We’ll share what’s working for AWS users and highlight which new features and services you’ll want to look at. Whether or not you attended re:Invent, this wrap-up will help you develop your 2017 cloud to-do list.
AWS Summit London 2014 | Scaling on AWS for the First 10 Million Users (200)Amazon Web Services
This mid-level technical session will provide an overview of the techniques that you can use to build high-scalabilty applications on AWS. Take a journey from 1 user to 10 million users and understand how your application's architecture can evolve and which AWS services can help as you increase the number of users that you serve.
Deep learning is an implementation of machine learning that uses neural networks to solve difficult and complex problems, such as computer vision, natural language processing, and recommendations. Due to the availability of deep learning libraries and frameworks, developers have the ability to enhance the capabilities of their applications and projects. In this workshop, you learn how to build and deploy a powerful deep learning framework called MXNet on containers. The portability and resource management benefit of containers means developers can focus less on infrastructure and more on building. The labs start by demonstrating the automation capabilities of AWS CloudFormation to stand up core infrastructure; as an added bonus, you use Spot Fleet to leverage the cost benefits of using Spot Instances, especially for developer environments. Then, you walk through creating an MXNet container in Docker and deploying it with Amazon ECS. Finally, you walk through an image classification demo of MXNet to validate that everything is working as expected. Note: This workshop focuses on containerizing MXNet. The features of MXNet and capabilities of deep learning in general are vast, and there are recorded sessions from re:Invent that dive deeper on these topics. All you need to participate is a laptop and AWS account. Pizza will be provided.
Building Highly Scalable Immersive Media Solutions on AWSETCenter
Immersive media content such as 360 degree video places a unique set of demands on cloud-based infrastructure from a complete end to end solution point of view. A key goal for any solution of this nature is to keep costs low while not impacting availability, scale and compute performance. In this talk we will look at how to solve ingest, processing, storage and delivery of live and on-demand content for immersive media delivery, and present a reference design for 360 degree streaming using best-practice architectural patterns.
Speakers: Konstantin Wilms, Principal Solutions Architect M&E, AWS and Chad Schmutzer, Specialist Solutions Architect SPOT, AWS
Let’s get started. Join this session to continue your journey through the core AWS services with live demonstrations of how to set up and use the services.
AWS Webcast - AWS Webinar Series for Education #2 - Getting Started with AWSAmazon Web Services
This webinar will cover the basics of getting started with AWS. After a brief overview, this session will dive into core AWS services with live demonstrations of how to set up and utilize compute, storage, and other services. The focus will be on the ease of use and the ability to clone environments that largest customers are running to highlight AWS’ versatility and ease of use as a cloud platform.
AWS Architecting Cloud Apps - Best Practices and Design Patterns By Jinesh VariaAmazon Web Services
Jinesh Varia, Technology Evangelist, Discusses AWS architecture best practices and design patterns at the AWS Enterprise Tour - SF - 2010
http://jineshvaria.s3.amazonaws.com/public/cloudbestpractices-jvaria.pdf
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...Amazon Web Services
You're on the verge of a new startup and you need to build a world-class, high-scale web application on AWS so it can handle millions of users. How do you build it quickly without having to reinvent and re-implement the best-practices of large successful Internet companies? NetflixOSS is your answer. In this session, we’ll cover how an emerging startup can leverage the different open source tools that Netflix has developed and uses every day in production, ranging from baking and deploying applications (Asgard, Aminator), to hardening resiliency to failures (Hystrix, Simian Army, Zuul), making them highly distributed and load balanced (Eureka, Ribbon, Archaius) and managing your AWS resources efficiently and effectively (Edda, Ice). You’ll learn how to get started using these tools, learn best practices from engineers who actually created them, so, like Netflix, you can too unleash the power of AWS and scale your application processes as you grow.
(CMP405) Containerizing Video: The Next Gen Video Transcoding PipelineAmazon Web Services
"Media delivery requirements are continually changing, driven by accelerating mobile, tablet, smart TV, and set-top technology advances. Media services need to deliver higher-resolution content at lower bitrates to consumers, which has traditionally been a compute-intensive undertaking with slow advancements in the adoption of video codecs, containers, and related technologies.
In this session, we look at some of the existing workflow constraints, and explore a solution to process media in an agile fashion using modern, efficient codecs. We use Amazon S3 events and AWS Lambda to configure media both pre- and post-process, process content in parallel with Amazon ECS using custom containers for a high level of elastic compute density, and deliver generated media to reference protocol clients via Amazon CloudFront. We also leverage Amazon EFS for scalable, shared storage in the distributed containerized environment for video processing. Issues include: parallel processing of content using Amazon ECS, pipelining and conversion of data using AWS Lambda, building an Amazon ECS-based media transcoding cluster, and delivering next-gen media through Amazon CloudFront."
Workshop; Deploy a Deep Learning Framework on Amazon ECS and Spot InstancesAmazon Web Services
by Asif Khan, Technical Business Development Manager, AWS
Deep learning is an implementation of machine learning that uses neural networks to solve difficult and complex problems, such as computer vision, natural language processing, and recommendations. Due to the availability of deep learning libraries and frameworks, developers have the ability to enhance the capabilities of their applications and projects. In this workshop, you learn how to build and deploy a powerful deep learning framework called MXNet on containers. The portability and resource management benefit of containers means developers can focus less on infrastructure and more on building. The labs start by demonstrating the automation capabilities of AWS CloudFormation to stand up core infrastructure; as an added bonus, you use Spot Fleet to leverage the cost benefits of using Spot Instances, especially for developer environments. Then, you walk through creating an MXNet container in Docker and deploying it with Amazon ECS. Finally, you walk through an image classification demo of MXNet to validate that everything is working as expected. Note: This workshop focuses on containerizing MXNet. The features of MXNet and capabilities of deep learning in general are vast, and there are recorded sessions from re:Invent that dive deeper on these topics. All you need to participate is a laptop and AWS account. Pizza will be provided. Level 300
Azure vs AWS Best Practices: What You Need to KnowRightScale
Azure is now the clear #2 in public cloud behind AWS. While some cloud users are evaluating Azure vs. AWS, many enterprises are planning to use both cloud providers. But there are some notable differences between how the two clouds operate and the best practices for deploying workloads in each.
The Azure vs. AWS Best Practices: What You Need to Know webinar will cover:
Recent and coming enhancements for Azure.
Azure vs. AWS differences for compute, networking, and storage.
Best practices for cloud deployments in Azure and AWS.
How to use both Azure and AWS.
Building a Just-in-Time Application Stack for AnalystsAvere Systems
Slide presentation from Webinar on February 17, 2016.
People in analytical roles are demanding more and more compute and storage to get their jobs done. Instead of building out infrastructure for a few employees or a department, systems engineers and IT managers can find value in creating a compute stack in the cloud to meet the fluctuating demand of their clients.
In this 45-minute webinar, you’ll learn:
- How to identify the right analytical workloads
- How to create a scalable compute environment using the cloud for analysts in under 10 minutes
- How to best manage costs associated with the cloud compute stack
- How to create dedicated client stacks with their own scratch space as well as general access to reference data
Health systems departments, research & development departments, and business analyst groups all face silos of these challenging, compute-intensive use cases. By learning how to quickly build this flexible workflow that can be scaled up and down (or off) instantly, you can support business objectives while efficiently managing costs.
AWS Webcast - Webinar Series for State and Local Government #2: Discover the ...Amazon Web Services
This webinar will cover the basics of getting started with AWS. After a brief overview this session will dive into live demonstration of core AWS services of how to set up and utilize compute (EC2), storage (S3), and other services. The focus will be on how you get started with AWS, including creating user accounts, set up multiple EC2 virtual machine instances, set up an email alert for changes in EC2 based on usage, upload data to S3 services and make it available via the internet.
An overview on current Microsoft Technologies around Private - / Hybrid-Clouds and what's coming up with the next version aka Azure Stack from our session at e2evc Berlin.
Let’s get started. Join this session to continue your journey through the core AWS services with live demonstrations of how to set up and use the services.
Virtualizing Apache Spark and Machine Learning with Justin MurrayDatabricks
This talk explains the reasons why virtualizing Spark, in-house or elsewhere, is a requirement in today’s fast-moving and experimental world of data science and data engineering. Different teams want to spin up a Spark cluster “on the fly” to carry out some research and quickly answer business questions. They are not concerned with the availability of the server hardware – or with what any other team might be doing on it at the time. Virtualization provides the means of working within your own sandbox to try out the new query or Machine Learning algorithm. Deep performance test results will be shown that demonstrate that Spark and ML programs perform equally well on virtual machines just like native implementations do. An early introduction is given to the best practices you should adhere to when you do this.
Similar to Usman Shakeel - Cloud Rendering at Scale :: AWS Rendering Seminar (20)
사례로 알아보는 Database Migration Service : 데이터베이스 및 데이터 이관, 통합, 분리, 분석의 도구 - 발표자: ...Amazon Web Services Korea
Database Migration Service(DMS)는 RDBMS 이외에도 다양한 데이터베이스 이관을 지원합니다. 실제 고객사 사례를 통해 DMS가 데이터베이스 이관, 통합, 분리를 수행하는 데 어떻게 활용되는지 알아보고, 동시에 데이터 분석을 위한 데이터 수집(Data Ingest)에도 어떤 역할을 하는지 살펴보겠습니다.
Amazon Elasticache - Fully managed, Redis & Memcached Compatible Service (Lev...Amazon Web Services Korea
Amazon ElastiCache는 Redis 및 MemCached와 호환되는 완전관리형 서비스로서 현대적 애플리케이션의 성능을 최적의 비용으로 실시간으로 개선해 줍니다. ElastiCache의 Best Practice를 통해 최적의 성능과 서비스 최적화 방법에 대해 알아봅니다.
Internal Architecture of Amazon Aurora (Level 400) - 발표자: 정달영, APAC RDS Speci...Amazon Web Services Korea
ccAmazon Aurora 데이터베이스는 클라우드용으로 구축된 관계형 데이터베이스입니다. Aurora는 상용 데이터베이스의 성능과 가용성, 그리고 오픈소스 데이터베이스의 단순성과 비용 효율성을 모두 제공합니다. 이 세션은 Aurora의 고급 사용자들을 위한 세션으로써 Aurora의 내부 구조와 성능 최적화에 대해 알아봅니다.
[Keynote] 슬기로운 AWS 데이터베이스 선택하기 - 발표자: 강민석, Korea Database SA Manager, WWSO, A...Amazon Web Services Korea
오랫동안 관계형 데이터베이스가 가장 많이 사용되었으며 거의 모든 애플리케이션에서 널리 사용되었습니다. 따라서 애플리케이션 아키텍처에서 데이터베이스를 선택하기가 더 쉬웠지만, 구축할 수 있는 애플리케이션의 유형이 제한적이었습니다. 관계형 데이터베이스는 스위스 군용 칼과 같아서 많은 일을 할 수 있지만 특정 업무에는 완벽하게 적합하지는 않습니다. 클라우드 컴퓨팅의 등장으로 경제적인 방식으로 더욱 탄력적이고 확장 가능한 애플리케이션을 구축할 수 있게 되면서 기술적으로 가능한 일이 달라졌습니다. 이러한 변화는 전용 데이터베이스의 부상으로 이어졌습니다. 개발자는 더 이상 기본 관계형 데이터베이스를 사용할 필요가 없습니다. 개발자는 애플리케이션의 요구 사항을 신중하게 고려하고 이러한 요구 사항에 맞는 데이터베이스를 선택할 수 있습니다.
Demystify Streaming on AWS - 발표자: 이종혁, Sr Analytics Specialist, WWSO, AWS :::...Amazon Web Services Korea
실시간 분석은 AWS 고객의 사용 사례가 점점 늘어나고 있습니다. 이 세션에 참여하여 스트리밍 데이터 기술이 어떻게 데이터를 즉시 분석하고, 시스템 간에 데이터를 실시간으로 이동하고, 실행 가능한 통찰력을 더 빠르게 얻을 수 있는지 알아보십시오. 일반적인 스트리밍 데이터 사용 사례, 비즈니스에서 실시간 분석을 쉽게 활성화하는 단계, AWS가 Amazon Kinesis와 같은 AWS 스트리밍 데이터 서비스를 사용하도록 지원하는 방법을 다룹니다.
Amazon EMR - Enhancements on Cost/Performance, Serverless - 발표자: 김기영, Sr Anal...Amazon Web Services Korea
Amazon EMR은 Apache Spark, Hive, Presto, Trino, HBase 및 Flink와 같은 오픈 소스 프레임워크를 사용하여 분석 애플리케이션을 쉽게 실행할 수 있는 관리형 서비스를 제공합니다. Spark 및 Presto용 Amazon EMR 런타임에는 오픈 소스 Apache Spark 및 Presto에 비해 두 배 이상의 성능 향상을 제공하는 최적화 기능이 포함되어 있습니다. Amazon EMR Serverless는 Amazon EMR의 새로운 배포 옵션이지만 데이터 엔지니어와 분석가는 클라우드에서 페타바이트 규모의 데이터 분석을 쉽고 비용 효율적으로 실행할 수 있습니다. 이 세션에 참여하여 개념, 설계 패턴, 라이브 데모를 사용하여 Amazon EMR/EMR 서버리스를 살펴보고 Spark 및 Hive 워크로드, Amazon EMR 스튜디오 및 Amazon SageMaker Studio와의 Amazon EMR 통합을 실행하는 것이 얼마나 쉬운지 알아보십시오.
Amazon OpenSearch - Use Cases, Security/Observability, Serverless and Enhance...Amazon Web Services Korea
로그 및 지표 데이터를 쉽게 가져오고, OpenSearch 검색 API를 사용하고, OpenSearch 대시보드를 사용하여 시각화를 구축하는 등 Amazon OpenSearch의 새로운 기능과 기능에 대해 자세히 알아보십시오. 애플리케이션 문제를 디버깅할 수 있는 OpenSearch의 Observability 기능에 대해 알아보세요. Amazon OpenSearch Service를 통해 인프라 관리에 대해 걱정하지 않고 검색 또는 모니터링 문제에 집중할 수 있는 방법을 알아보십시오.
Enabling Agility with Data Governance - 발표자: 김성연, Analytics Specialist, WWSO,...Amazon Web Services Korea
데이터 거버넌스는 전체 프로세스에서 데이터를 관리하여 데이터의 정확성과 완전성을 보장하고 필요한 사람들이 데이터에 액세스할 수 있도록 하는 프로세스입니다. 이 세션에 참여하여 AWS가 어떻게 분석 서비스 전반에서 데이터 준비 및 통합부터 데이터 액세스, 데이터 품질 및 메타데이터 관리에 이르기까지 포괄적인 데이터 거버넌스를 제공하는지 알아보십시오. AWS에서의 스트리밍에 대해 자세히 알아보십시오.
Amazon Redshift Deep Dive - Serverless, Streaming, ML, Auto Copy (New feature...Amazon Web Services Korea
이 세션에 참여하여 Amazon Redshift의 새로운 기능을 자세히 살펴보십시오. Amazon Data Sharing, Amazon Redshift Serverless, Redshift Streaming, Redshift ML 및 자동 복사 등에 대한 자세한 내용과 데모를 통해 Amazon Redshift의 새로운 기능을 알고 싶은 사용자에게 적합합니다.
From Insights to Action, How to build and maintain a Data Driven Organization...Amazon Web Services Korea
데이터는 혁신과 변혁의 토대입니다. 비즈니스 혁신을 이끄는 혁신은 특정 시점의 전략이나 솔루션이 아니라 성장을 위한 반복적이고 집단적인 계획입니다. 혁신에 이러한 접근 방식을 채택하는 기업은 전략과 비즈니스 문화에서 데이터를 기반으로 하는 경우가 많습니다. 이러한 접근 방식을 개발하려면 리더가 데이터를 조직의 자산처럼 취급하고 조직이 더 나은 비즈니스 성과를 위해 데이터를 활용할 수 있도록 권한을 부여해야 합니다. AWS와 Amazon이 어떻게 데이터와 분석을 활용하여 확장 가능한 비즈니스 효율성을 창출하고 고객의 가장 복잡한 문제를 해결하는 메커니즘을 개발했는지 알아보십시오.
[Keynote] Accelerating Business Outcomes with AWS Data - 발표자: Saeed Gharadagh...Amazon Web Services Korea
데이터는 최종 소비자의 성공에 초점을 맞춘 디지털 혁신에서 중추적인 역할을 하고 있습니다. 모든 기업들은 데이터를 자산으로 사용하여 사례 제공을 추진하고 까다로운 결과를 해결하고 있습니다. AWS 클라우드 기술과 분석 솔루션의 강력한 성능을 통해 고객은 혁신 여정을 가속화할 수 있습니다. 이 세션에서는 기업 고객들이 클라우드에서 데이터의 힘을 활용하여 혁신 목표를 달성하고 필요한 결과를 제공하는 방법에 대해 다룹니다.
LG전자 - Amazon Aurora 및 RDS 블루/그린 배포를 이용한 데이터베이스 업그레이드 안정성 확보 - 발표자: 이은경 책임, L...Amazon Web Services Korea
LG ThinQ는 LG전자의 가전제품과 서비스를 아우르는 플랫폼 브랜드로서 앱 하나로 간편한 컨트롤, 똑똑한 케어, 스마트한 쇼핑까지 한번에 가능한 플랫폼입니다. ThinQ 플랫폼은 글로벌 서비스로 제공되고 있어, 작업 시간을 최소화하고, 서비스의 영향을 최소화 할 필요가 있었습니다. 따라서 DB 버전 업그레이드 작업 시 애플리케이션 배포가 필요없는 Blue/Green Deployment 방식은 최선의 선택이 되었습니다.
KB국민카드 - 클라우드 기반 분석 플랫폼 혁신 여정 - 발표자: 박창용 과장, 데이터전략본부, AI혁신부, KB카드│강병억, Soluti...Amazon Web Services Korea
온프레미스 분석 플랫폼에는 자원 증설 비용, 자원 관리 비용, 신규 자원 도입 및 환경 설정의 리드타임 등 다양한 측면에서의 한계가 존재합니다. 이에 KB국민카드에서는 기존 분석 플랫폼의 한계를 극복함과 동시에 시너지를 낼 수 있는 클라우드 기반 분석 플랫폼을 설계 및 도입하였습니다. 본 사례 소개는 KB국민카드의 데이터 혁신 여정과 노하우를 소개합니다.
SK Telecom - 망관리 프로젝트 TANGO의 오픈소스 데이터베이스 전환 여정 - 발표자 : 박승전, Project Manager, ...Amazon Web Services Korea
SK Telecom의 망관리 프로젝트인 TANGO에서는 오라클을 기반으로 시스템을 구축하여 운영해 왔습니다. 하지만 늘어나는 사용자와 데이터로 인해 유연하고 비용 효율적인 인프라가 필요하게 되었고, 이에 클라우드 도입을 검토 및 실행에 옮기게 되었습니다. TANGO 프로젝트의 클라우드 도입을 위한 검토부터 준비, 실행 및 이를 통해 얻게 된 교훈과 향후 계획에 대해 소개합니다.
코리안리 - 데이터 분석 플랫폼 구축 여정, 그 시작과 과제 - 발표자: 김석기 그룹장, 데이터비즈니스센터, 메가존클라우드 ::: AWS ...Amazon Web Services Korea
2022년 코리안리는 핵심업무시스템(기간계/정보계 시스템)을 AWS 클라우드로 전환하는 사업과 AWS 클라우드 기반에서 손익분석을 위한 어플리케이션 구축 사업을 동시에 진행하고 있었습니다. 이에 따라 클라우드 전환 이후 시스템 간 상호운용성과 호환성을갖춘 데이터 분석 플랫폼 또한 필요하게 되었습니다. 코리안리 IT 환경에 적합한 플랫폼 선정을 위하여 AWS Native Analytics Platform, 3rd Party Analytics Platform (클라우데라, 데이터브릭스)과의 PoC를 진행하고, 최종적으로 AWS Native Analytics Platform 으로 확정하였습니다. 코리안리는 메가존클라우드와 함께 2022년 10월부터 4개월(구축 3개월, 안정화 및 교육 1개월) 동안 AWS 기반 데이터 분석 플랫폼을 구축하고 활용 범위를 지속적으로 확대하고 있습니다.
LG 이노텍 - Amazon Redshift Serverless를 활용한 데이터 분석 플랫폼 혁신 과정 - 발표자: 유재상 선임, LG이노...Amazon Web Services Korea
LG 이노텍은 세계 시장을 선도하는 글로벌 소재·부품기업으로, Amazon Redshift 을 데이터 분석 플랫폼의 핵심 서비스로 활용하고 있습니다.지속적인 데이터 증가와 업무 확대에 따른 유연한 아키텍처 개선의 필요성에 대처하기 위해, 2022년에 AWS 에서 발표된 Redshift Serverless 를 활용한, 비용 최적화된 아키텍처 개선 과정의 실사례를 엿볼수 있는 기회가 됩니다.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
2. Visual Effects and
Animation
1
Who is using AWS for Rendering?
3 Theme Parks
5 Gaming
Marketing2
4 Manufacturing
6 Life Sciences
7 Engineering and Architecture
7. The challenge of making a film
On-premise capacity
Rendering in the Cloud
8. The challenge of making a film
On-premise capacity
Rendering in the Cloud
Cloud provides you the capability to
scale fast and get the outputs faster
Initial project on-boarding
artwork
10. They all ask us the same thing …
The ability to spin up thousands of cores on-demand
… without any upfront investment
… and leverage the most up-to-date configurations
A project based “disposable” infrastructure
… with a flexible licensing / utility / by the hour
11. They all tell us the same thing …
=< $0.01
per core/hour
Access to thousands of
cores whenever needed
No upfront investments in
infrastructure
Easier collaboration
Ecosystem of software
providers
Access to large memory
configs to do 6K/10K renders
Project based “disposable”
infrastructure
12. … when rubber meets the road !
Share FS Everywhere Latency Large Datasets Lots of instances
{Data/Content}
14. Rendering in the Cloud - State of the Union
Scale at a very cheap price
EC2 Spot
15. Leveraging Spot successfully today requires some
effort
Build stateless, distributed, scalable applications
Choose which instance types fit your workload the best
Ingest price feed data for AZs and regions
Make run time decisions on which Spot pools to launch in based on
price and volatility
Manage interruptions
Monitor and manage market prices across AZs and instance types
Manage the capacity footprint in the fleet
And all of this while you don’t know where the capacity is
Serve your customers
16. Spot Fleet
Instead of writing all that code to manage Spot Instances,
simply specify:
• Target Capacity – The number of EC2 instances that you want
in your fleet.
• Maximum Bid Price – The maximum bid price that you are
willing to pay.
• Launch Specifications – # of and types of instances, AMI id,
VPC, subnets or AZs, etc.
• IAM Fleet Role – The name of an IAM role. It must allow EC2 to
terminate instances on your behalf.
17. Spot Fleet Example – Instance Weighting
Say your workload needs at least 60 GB of memory
Want capacity to complete 20 units of work
Choices:
• r3.2xlarge (61.0 GB, 8 vCPUs) = 1 unit of 20
• r3.4xlarge (122.0 GB, 16 vCPUs) = 2 units of 20
• r3.8xlarge (244.0 GB, 32 vCPUs) = 4 units of 20
An option to bid for all of these instance types:
18. AWS Cloud Scale is “Large”
•10s/100s/1000s/10000s cores on-demand in the Cloud
•A “Large” renderfarm (Actual example):
55,000 cores
•At re:Invent 2015:
~40,000 cores on
EC2 Spot Market
Rendering in the Cloud - State of the Union
Scale at a very cheap price
19. • BYOL
Vray, Solid Angle, Autodesk Maya 3DsMax
• SaaS
Thinkbox Deadline Cloud Wizard
• AWS Marketplace
• Elastic Licensing models
Thinkbox Deadline Usage Based Licensing
• Render nodes pull metered licenses from Cloud-based license server
• Usage is tracked per minute
• Bulk minutes will be available via Thinkbox’s online store
• Store will eventually host 3rd party licensing (Nuke, VRay, etc)
Pixar Renderman/Tractor (demo at SIGGRAPH)
Rendering in the Cloud - State of the Union
Licensing at Cloud Scale
20. Rendering in the Cloud - State of the Union
Hydrating the Cloud Renderfarm
S3 as the source of truth for your content/data
• On AWS Marketplace/SaaS
(Aspera, Signiant, File Catalyst, Expedat)
• S3 Multi-part Upload
• AWS import/export Snowball
Direct to Shared File Systems
• EFS throughput scales linearly to the storage
• Lustre can hydrate from an S3 bucket
• Avere can be fronted to S3 or an on-premise NAS
+ Direct Connect
EFS
S3
Multipart
AWS Snowball
21. Rendering in the Cloud - State of the Union
Shared FileSystem Everywhere (some ideas)
Shared Storage
On-prem Storage
AWS Direct Connect
Storage Cache
Amazon S3
Luster on EC2
Avere on EC2
EFS
AWS Direct Connect
Hydrate workers
EC2 Spot
Shared Storage
FXT on-prem
22. Rendering in the Cloud - State of the Union
Shared FS (Content/Data Share) Everywhere (some ideas)
Elastic File System (Amazon EFS)
• Designed to support Peta-Byte scale file systems
• Throughput scales linearly to storage
• Same latency spec across each AZ
• Thousands of concurrent NFS connections
• Works great for Large I/O sizes
• Pay for only what you use not what you provision
• Managed with multi-copy durability Amazon EFS
23. Rendering in the Cloud - State of the Union
Move the Graphic Artist to the Cloud …
Rendering is going Global
• NVIDIA GPU based EC2 instances
• Nice DCV
• Teradici PCoIP
• Frame, Otoy
• Windows and Linux (VNC+VirtualGL)
3D Modeler
Modeling Dumb Client
Remote Application
running on a G2 instance
G2
24. Rendering in the Cloud - State of the Union
Managing your “disposable” infrastructure
Launch a CloudFormation stack
with all the infrastructure
resources for a specific project
Autoscale the stack as
appropriate
AMI
CloudFormation
Template
CloudFormation
Terminate
Template
25. Rendering in the Cloud - State of the Union
The Crown Jewels
• AWS alignment with the latest MPAA cloud based application
guidelines for content security – August 2015
• VPC private endpoint for S3 – enables a true private workflow
capability
• Encryption & key management capabilities
• Glacier Vault for high-value media/originals
26. Rendering in the Cloud - A Sample Architecture
(All in Cloud Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
Pipeline and License Manager
3D Modeler
Remote
App Visualization
AWS Direct Connect
Modeling Dumb Client
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
Appstream or Teradici running on a G2 instance
Pipeline Manager running on EC2
G2
EC2 SPOT
EFS
Hydrate workers
EC2 Spot
27. Render Farm
Rendering in the Cloud - A Sample Architecture
(A Hybrid Pipeline)
Shared Storage
Renderfarm
On-Prem Storage
AWS Direct Connect
Storage Cache
Amazon S3
Avere on EC2
Scalable Renderfarm on EC2
EFS
Hydrate workers
EC2 Spot
On-premise
Renderfarm
Cloud renderfarm as an
extension of on-prem renderfarm
FXT on-prem
Pipeline and License
Manager (also manage
cloud renderfarm)
28. A Customer Example
A large scale test:
• Deploying ~40K cores
• In 20 min
• Leveraging Spot Fleet
• Hybrid rendering environment
using Avere (on-premise Data)
• EFS
Faster than on-premise
environment
29. Disney Animation Renderfarm
Renderfarm
Avere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb
30. Disney Animation Renderfarm
(on the Cloud)
Renderfarm
Avere FXT
cluster
WDAS Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Burbank
Artists
Redundant 10Gb
virtual private cloud
Avere vFXT
Oregon
Spot InstancesEFS
34. Benchmarks: On Premise vs. The Cloud
0
20
40
60
80
100
120
stream triad disk read disk write
On Prem
r3.4xlarge
r3.8xlarge
m4.4xlarge
m4.10xlarge
cr1.8xlarge
Higher is better
37. Rendering in the Cloud vs. On-Premise
-
5,000
10,000
15,000
20,000
25,000
30,000
1 10 20 30 40 50 60 70 80 90
RenderTime(s)
Frame #
EC2/EFS
On Prem
Lower is better
38. Lessons Learned
• Use as many different instance types as you can. Especially older generations.
• Think about ways to modify your workload
• Use every availability-zone
• Check your limits, especially your EBS limit and VPC setup (address space)
• Resource-Oriented Bidding
• Diversified Allocation
• Benchmark your workload and set pricing accordingly
• Set ONLY realistic pricing that you will pay for
• Don’t be afraid to ask for help or pre-planning your run from AWS
39. Conclusion
Cloud Rendering on AWS
State of the Union is getting stronger …
Rendering Forecast
Partly cloudy with a chance of all in the cloud…
Technology Trends
•Containers for Rendering
•GPU based Rendering
•More vendors investing in Elastic Licensing models due to the push from their
customers
•All in-Cloud rendering with more robust application streaming solutions