Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...Shift Conference
Take a deep look into Azure Machine Learning, a cloud service that helps you build, train, deploy, and manage models. Walk through the data science process and then have some fun creating a ML recognition model based on the Simpsons cartoon with PyTorch. You'll leave this session with a better grasp of the technological components of Azure Machine Learning services.
Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...Shift Conference
Take a deep look into Azure Machine Learning, a cloud service that helps you build, train, deploy, and manage models. Walk through the data science process and then have some fun creating a ML recognition model based on the Simpsons cartoon with PyTorch. You'll leave this session with a better grasp of the technological components of Azure Machine Learning services.
Amazon EC2 Container Service is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. Part of ECS is Amazon EC2 Container Registry (ECR). Amazon ECR is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. This session will describe how you can use ECS and ECR for your applications.
Speaker: Sascha Möllering, Solutions Architect, AWS
Infrastructure Automation on AWS using a Real-World Customer ExampleAPI Talent
This technical session focuses on a customer use case and how using the AWS Cloud together with automation has enabled them to standardise and automate their systems.
This talk will describe how this is achieved with two tools, Cloud formation and Puppet. Cloud formation is a declarative templating language that enables the deployment of environments in a standardised way. Combined with a configuration management tool like Puppet allows for the automation of ongoing software deployments and maintenance in a low overhead manner. Puppet is a Configuration Management tool that installs and configures software on instances. Taken together a complete system can be built from the ground up.
Kleiner Überblick über die Möglichkeiten WordPress auf AWS zu hosten und die Vorteile der automatischen Skalierung für sich zu nutzen. Vortrag auf dem WordCamp Frankfurt 2016
Flynn Bundy - 60 micro-services in 6 months WinOps Conf
In this talk, I want to take the audience on a journey of how we (Coolblue) migrated 60 .Net micro-services to the AWS Cloud. This talk covers the high’s, low’s and everything in between when working in a multi-disciplinary Developer / Operations Cloud team. This talk will cover the evolution of our processes and toolsets to align with Chaos Engineering best practices. Most importantly, I want to highlight how we changed the way we thought about services and servers in general.
The key takeaways from this talk would be related to:
Continous Inspection (TeamCity)
Continous Deployment (Octopus Deploy)
Infrastructure as Code (Cloudformation)
Chaos Engineering (Chaos Monkey)
Monitoring and Logging (Datadog and Splunk)
.Net and .Net Core (on Windows Server 2016)
Automation in AWS Cloud
Let’s dive into the world of serverless and give you real world examples of how to get started. We will focus on Azure Functions in Java and discuss how to provision, deploy and test them in a productive environment. In my demos we will see the ease of local development leveraging from the great integration in Visual Studio Code. Finally, let’s ship our samples and scale them in Azure. If you are tired of server maintenance and want to achieve more with your java functions , don’t miss this session.
Switching SaaS Hosting From dedicated virtual machines to container-based clu...AWS Germany
Presentation "Switching SaaS Hosting From dedicated virtual machines to container-based clusters" from Dr. Sven Ehlert at the AWS E-Business Web Day for windows applications. All videos and presentations can be found here: http://amzn.to/2ds3aMX
We will walk through the exploration, training and serving of a machine learning model by leveraging Kubeflow's main components. We will use Jupyter notebooks on the cluster to train the model and then introduce Kubeflow Pipelines to chain all the steps together, to automate the entire process.
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...CodeOps Technologies LLP
Running day-1 Ops on your Kubernetes is somewhat easy, but it is quite daunting to manage day two challenges. Learn about AKS best practices for your cloud-native applications so that you can avoid blow up your workloads.
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Trivadis
Heutzutage schreibt man nicht nur Applikationen mit Code. Dank der Cloud wird die Konfiguration von Infrastruktur wie virtuellen Maschinen oder Netzwerken in Code definiert und automatisiert ausgeliefert. Man spricht von Infrastructure as Code, kurz: IAC. Für Infrastructure as Code auf Azure gibt es viele tools wie Ansible, Puppet, Chef, etc. Zwei Lösungen stechen durch Ihren unterschiedlichen Ansatz heraus - Die Azure Resource Manager Templates (ARM) als Microsoft-native Lösung, immer auf dem neusten Stand, aber an Azure gebunden. Auf der anderen Seite Terraform von HashiCorp mit einer deskriptiven Sprache als Grundlage, dafür weniger Features im Security-Bereich. Für einen Grosskunden haben wir die beiden Technologien verglichen. Die Resultate zeigen wir in dieser Session mit Livedemos auf.
This session was about Master Data Services and what it also could be used as - the client wanted an application to validate and submit warehouse inventories.
Amazon EC2 Container Service is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. Part of ECS is Amazon EC2 Container Registry (ECR). Amazon ECR is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. This session will describe how you can use ECS and ECR for your applications.
Speaker: Sascha Möllering, Solutions Architect, AWS
Infrastructure Automation on AWS using a Real-World Customer ExampleAPI Talent
This technical session focuses on a customer use case and how using the AWS Cloud together with automation has enabled them to standardise and automate their systems.
This talk will describe how this is achieved with two tools, Cloud formation and Puppet. Cloud formation is a declarative templating language that enables the deployment of environments in a standardised way. Combined with a configuration management tool like Puppet allows for the automation of ongoing software deployments and maintenance in a low overhead manner. Puppet is a Configuration Management tool that installs and configures software on instances. Taken together a complete system can be built from the ground up.
Kleiner Überblick über die Möglichkeiten WordPress auf AWS zu hosten und die Vorteile der automatischen Skalierung für sich zu nutzen. Vortrag auf dem WordCamp Frankfurt 2016
Flynn Bundy - 60 micro-services in 6 months WinOps Conf
In this talk, I want to take the audience on a journey of how we (Coolblue) migrated 60 .Net micro-services to the AWS Cloud. This talk covers the high’s, low’s and everything in between when working in a multi-disciplinary Developer / Operations Cloud team. This talk will cover the evolution of our processes and toolsets to align with Chaos Engineering best practices. Most importantly, I want to highlight how we changed the way we thought about services and servers in general.
The key takeaways from this talk would be related to:
Continous Inspection (TeamCity)
Continous Deployment (Octopus Deploy)
Infrastructure as Code (Cloudformation)
Chaos Engineering (Chaos Monkey)
Monitoring and Logging (Datadog and Splunk)
.Net and .Net Core (on Windows Server 2016)
Automation in AWS Cloud
Let’s dive into the world of serverless and give you real world examples of how to get started. We will focus on Azure Functions in Java and discuss how to provision, deploy and test them in a productive environment. In my demos we will see the ease of local development leveraging from the great integration in Visual Studio Code. Finally, let’s ship our samples and scale them in Azure. If you are tired of server maintenance and want to achieve more with your java functions , don’t miss this session.
Switching SaaS Hosting From dedicated virtual machines to container-based clu...AWS Germany
Presentation "Switching SaaS Hosting From dedicated virtual machines to container-based clusters" from Dr. Sven Ehlert at the AWS E-Business Web Day for windows applications. All videos and presentations can be found here: http://amzn.to/2ds3aMX
We will walk through the exploration, training and serving of a machine learning model by leveraging Kubeflow's main components. We will use Jupyter notebooks on the cluster to train the model and then introduce Kubeflow Pipelines to chain all the steps together, to automate the entire process.
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...CodeOps Technologies LLP
Running day-1 Ops on your Kubernetes is somewhat easy, but it is quite daunting to manage day two challenges. Learn about AKS best practices for your cloud-native applications so that you can avoid blow up your workloads.
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Trivadis
Heutzutage schreibt man nicht nur Applikationen mit Code. Dank der Cloud wird die Konfiguration von Infrastruktur wie virtuellen Maschinen oder Netzwerken in Code definiert und automatisiert ausgeliefert. Man spricht von Infrastructure as Code, kurz: IAC. Für Infrastructure as Code auf Azure gibt es viele tools wie Ansible, Puppet, Chef, etc. Zwei Lösungen stechen durch Ihren unterschiedlichen Ansatz heraus - Die Azure Resource Manager Templates (ARM) als Microsoft-native Lösung, immer auf dem neusten Stand, aber an Azure gebunden. Auf der anderen Seite Terraform von HashiCorp mit einer deskriptiven Sprache als Grundlage, dafür weniger Features im Security-Bereich. Für einen Grosskunden haben wir die beiden Technologien verglichen. Die Resultate zeigen wir in dieser Session mit Livedemos auf.
This session was about Master Data Services and what it also could be used as - the client wanted an application to validate and submit warehouse inventories.
You have some on-premise application. Perheaps you have Wordpress/PHP or Node.js/Javascript, you like them, but you don't want to handle, some ops issues, like managing FastCGI (for PHP) or Node service.
Why don't you publish them on Azure?
Web Sites already support PHP and Node. And you can store MySql DB on ClearDb as DBaaS.
And then you can scale out your app bringing your session out the server with Redis.
And, again, you can do some worker jobs with Azure Web Jobs.
And undestand how can you use Kudu features to debug and work better with websites.
Geek Sync | Deployment and Management of Complex Azure EnvironmentsIDERA Software
You can watch the replay of this Geek Sync webinar in the IDERA Resource Center: http://ow.ly/pg7N50A4svf.
Today's data management professional is finding their landscape changing. They have multiple database platforms to manage, multi-OS environments and everyone wants it now.
Join IDERA and Kellyn Pot’Vin-Gorman as she discusses the power of auto deployment in Azure when faced with complex environments and tips to increase the knowledge you need at the speed of light. Kellyn will cover scripting basics, advanced Portal features, opportunities to lessen the learning curve and how multi-platform and tier doesn't have to mean multi-cloud.
Attendees can expect to learn how to build automation scripts efficiently, even if you have little scripting experience, and how to work with Azure automation deployments. This session will allow you to begin building a repository of multi-platform development scripts to use as needed.
About Kellyn: Kellyn Pot’Vin-Gorman is a member of the Oak Table Network and an IDERA ACE and Oracle ACE Director alumnus. She is the newest Technical Solution Professional in Power BI with AI in the EdTech group at Microsoft. Kellyn is known for her extensive work with multi-database platforms, DevOps, cloud migrations, virtualization, visualizations, scripting, environment optimization tuning, automation, and architecture design. She has spoken at numerous technical conferences for Oracle, Big Data, DevOps, Testing and SQL Server. Her blog, http://dbakevlar.com and social media activity under her handle, DBAKevlar is well respected for her insight and content.
Hello All,
It is time for the second Tokyo Azure Meetup!
As a natural continuation of our first topic, we will proceed with Big Data.
Until recently you needed to learn new language or master new concepts in order get started with Big Data.
Moreover, you needed to spend a lot of time setting up infrastructure that will meet the business demands for Big Data processing.
Not any more!
If you know C# and T-SQL you are ready to become Big Data master!
Public cloud and especially Microsoft Azure are very well suited for working with Big Data.
Join us for our next event and and I can assure you that after the session you will be ready to start working with Big Data.
And maybe you are asking why this is important.
I believe that we don't have choice but build smart applications and get as much possible insights from the data we collect from various sources in order to take the best business decisions and please our customers.
Today we have so much data available publicly or coming from our customers and it is very challenging to process it and turn it into valuable business asset.
Not any more!
Join for our next meetup and you will see how Microsoft create amazing opportunity for each .Net developer to become Big Data expert and every company to start using Big Data to accelerate its growth.
I have been working closely with the product team developing U-SQL language that empower Azure Data Lake Analytics, which is one of the processing engines for Azure Data Lake and I will be very happy to share my experience with you!
See you very soon!
Kanio
PuppetConf 2017: Unlocking Azure with Puppet Enterprise- Keiran Sweet, Source...Puppet
For the last year Sourced has been assisting a large Canadian based financial organization migrate workloads to Microsoft's Azure public cloud platform. As part of this deployment, Puppet is leveraged to ensure high levels of automation and compliance across the environment. In this updated session we will walk through our approach to integrating Puppet in Azure environments to ensure that automation, security, compliance and infrastructure as code is at the forefront.
Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...Tokyo Azure Meetup
Serverless architecture is the next big shift in computing - completely abstracting the underlying infrastructure and focusing 100% on the business logic.
Today we can create applications directly in our browser and leave the decision how they are hosted and scaled to the cloud provider. Moreover, this approach give us incredible control over the granularity of our applications since most of the time we are dealing with single function at a time.
In this presentation we will cover:
• Introduce Serverless Architectures
• Talk about the advantages of Serverless Architectures
• Discuss in details in event-driven computing
• Cover common Serverless approaches
• See practical applications with Azure Functions
• Compare AWS Lambda and Azure Functions
• Talk about open source alternatives
• Explore the relation between Microservices and Serverless Architectures
Going Serverless - an Introduction to AWS GlueMichael Rainey
Going "serverless" is the latest technology trend for enterprises moving their processing to the cloud, including data integration and ETL tools. But what does that mean and when should I use serverless ETL? In this session, we'll dive into the world of Amazon's fully managed data processing service called AWS Glue. With no server to provision or resources to allocate, and an easy to populate metadata catalog, AWS Glue allows the data engineer to focus on his or her craft; building data transformations and pipelines. Gaining an understanding of the similarities and differences between traditional ETL tools, such as Oracle Data Integrator, and Glue will prepare attendees for the new world of data integration. Presented at Collaborate 18.
Video link:
https://www.youtube.com/watch?v=bv-3RgIZIZY
https://www.youtube.com/channel/UCcuMPYJ4Osax4528rgqQWrw?sub_confirmation=1
https://medium.com/@varunmanik1/aws-copilot-by-deploying-an-amazon-elastic-container-service-infrastructure-application-3854f1e5eacf
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...Amazon Web Services
Organizations around the globe are leveraging the cloud to accomplish world-changing missions. This session will address how AWS can help organizations put more money toward their mission and scale outreach and operations to achieve more with less. Hear some of AWS’s most advanced customers on how their organizations handle DevOps, continuous integration and deployment. Learn how these practices allow them to rapidly develop, iterate, test and deploy highly-scalable web applications and core operational systems on AWS. The discussion will focus on best practices, lessons learned, and the specific technologies and services they use.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
2. #CampusDays
Agenda
Elements in a BIG DATA Project on AZURE
• Walkthrough of the elements needed
HDInsight
• Deploy through Azure Portal
• Deploy with Powershell and Windows Azure SQL Database
• Multiple Storage Accounts and Configuration Values
• Deploy as part of your normal ETL
4. #CampusDays
Elements in a BIG DATA Project on AZURE
• AZURE Account
• Storage Account
• SQL Server
• SQL Databases
• Firewall rules
• HDInsight Cluster
• Data
• Hive Scripts
• Machine Learning
6. #CampusDays
Deployment via AZURE portal
Requirements
• AZURE Account
• Either a free trial
• MSDN Subscription
• Or paid subscription
• Create one here - http://azure.microsoft.com/da-dk/pricing/free-trial/
8. #CampusDays
Deployment via AZURE portal
• SQL Server
• Create either when creating a datebase
• Or alone without a database
9. #CampusDays
Deployment via AZURE portal
• SQL Databases
• Easy created only name, server and subscription needed
10. #CampusDays
Deployment via AZURE portal
• Firewall Rules
• Cluster will not be able to see metastore and cluster creation fails
11. #CampusDays
Deployment via AZURE portal
• HDInsight Cluster
• Needs a storage account
• Firewall rules must be set to allow all AZURE Services
12. #CampusDays
Deployment via AZURE portal
• Upload files to Azure
• Use Azure Explorer
• Upload files yourself
• Import job via portal
• Ship harddrive to Microsoft
• Demo
13. #CampusDays
Deployment via AZURE portal
• Many steps
• Easy to make mistakes
• This will be done over and over again
• Is there another way to make this easier?
• YES!
• Lets have a look at it
16. #CampusDays
Let’s automate it – using PowerShell
• Why Automate it?
• Reliability
• Repeatability
• Save time
• Eliminate tiresome work
• Eliminate manual work
• Manual work is bound to fail at
some point
17. #CampusDays
Let’s automate it – using PowerShell
• Configuration
• Flexible
• Create and recreate
• Upload data to Cluster
• Easy to make changes to project
• Easy to test
21. #CampusDays
Let’s automate it – using PowerShell
• After usage – clean up -> save money
• Script to cleanup cluster
• Storage
• SQL server
• SQL databases
This saves money, and we can easily
recreate the objects needed
23. #CampusDays
Let’s automate it – using PowerShell
• Firewall Rule is required
• Cluster will not be able to see metastore and cluster creation fails
• Allow All Azure Services
• On SQL Server created early
New-AzureSqlDatabaseServerFirewallRule
-ServerName Campusdays2014
-AllowAllAzureServices
-Verbose
24. #CampusDays
Let’s automate it – using PowerShell
• Remember to Add-AzureAccount to your Powershell session.
• Otherwise you’ll get an error.
26. #CampusDays
HDInsight as a part of your ETL
• Normal ETL on-prem
• Benefits of the Cloud
• Staying on-prem
27. #CampusDays
Keep the cost down and the flexibility high
• Supports Hybrid scenarios
• Run on-prem
• Create HDInsight cluster
• Do some cool stuff
• Destroy the cluster
• No need for PowerShell knowledge
28. #CampusDays
HDinsight SSIS Components
• Community driven
• More than 10 SSIS components (Incl. connections)
• First step for moving to the cloud
Title Slide – Insert session title, session code and speaker names
Project this slide while attendees are arriving.
Please do not add additional elements to this slide
Section title slide (Optional)
Vi skal idag kigge lidt på hvad indholdet I en BIG DATA løsning på AZURE kan indeholde
Der kommer en hurtig gennemgang af elementerne , hvad de indeholder og hvad de skal bruges til
Effter denne gennemgang, går vi over til den lidt mere praktiske del, hvor vi kigger på hvordan elementerne kan oprettes
Kan vi gøre det på flere forskellige made, og hvilken er den bedste? Hvis sådan en finds.
Hvad vil jeg opnå med denne løsnig,
Det er målet at stille en stor mængde data til rådighed for en Machine Learning Algoritme som vi skal kigge på senere.
Der skal oprettes et helt project på AZURE som understøtter dette. Vi skal downloade data og uploade det til vores Cluster
Section title slide (Optional)
Man skal selvfølgelig have en AZURE account
En storage account er her der oprettes sql servers, databaser, containers etc. dvs denne SKAL bruges og være på plads når der arbejdes med data på Azure.
SQL server, der skal oprettes en SQL server på azure, som skal hoste alle de databaser der skal benyttes, i dette tilfælde bliver den udelukkende brugt til at gemme META data om HDInsight Clusteret, det oprettes senere
Databaserne, det er selvklart her data gemmes, og der kan oprettes N databaser.
Firewall Rules, disse skal være på plads for at styrer adgangen til databaser og services på AZURE.
Dette er for at tillade trafik og adgang fra de enkelte services til den database der indeholde rmetadata
HDInsight Cluster, dette er AZURES Hadoop løsning, der indeholder alt hvad der skal til for at arbejde med BIG DATA i Skyen
Det er dette cluster der muligt at opload data til iform af eks. CSV filer, som kan lægges i tabeller, for senere at lave forespørgsler på disse via HOVE scripts
Data, det data der skal gemmes i Data containeren skal oploades, dette kan som sagt være alt slags data, tekst, billeder, lyd, blot der er en eller anden form for meta data der kan gøres søgbar via HIVE scripts
Hive Scripts – dette er HADOOP eller HDInsights query language, det min der meget om SQL men der er dog visse begrænsninger på dette. Der skal skrives et HIVE script som man derefter commiter på sin HDINsight – hastigheden på forespørgslerne kan virke langsommeligt, men husk at det er data uden indexes, og det vi kan kalde blandet data.
Lad os starte med at se på hvordan vi kan få deployet elementerne via AZURE portalen, det vil sige manuelt.
Det er et krav at man har en AZURE konto, dette kan enten være en gratis prøve version
En MSDN konto, eller en betalt konto hvor man har mulighed for at sætte en beløbsgrænse på
HUSK at det altid koster penge at have eks. Et HDInsight Cluster stående og være tændt, hvorimod det IKKE koster noget at bruge storage. Med andre ord det er altså CPU tid man betaler for på AZURE.
Her er link til oprettelse af en gratis prøve version
Der er behov for at have oprettet en storage account, denne skal benyttes til at oprette Containers med, det er her data bliver gemt ifbm. blobs på HDInsight Clusteret.
Denne oprettes ved at klikkepå storage, hvor man hjælpsomt får af vide hvis man endnu ikke har oprettet en, skal dette gøres klikkes selvklart på opret storage account.
Det er vigtigt at huske på at en storage accounts navn skal være unikt, da det kommer til at blive benyttet som subdomæne på *.core.windows.net – samt at navnet SKAL skrives med lowercase.
Vælg herefter den location der er tættest på den fysiske lokation der skal benytte storage accounten mest. Dvs. er der tale om en dansk løsning, så vil det kunne betale sig at vælge ”North Europe” mens var vi nu i Seattle, så ville jeg vælge North America. Dette alene pga. netforbindelser og afstande.
Vælg dernæst om din storage account skal være Georedundant, Local redundant, Zone redundant eller Read Access Geo Redundant
Opret en sql server, det er ikke muligt at oprette denne alene, hvorfor der skal gøres samtidigt med at der oprettes en database.
Angiv et databasenavn, vælg den subscription der skal benyttes
New SQL database Server
Lokation, og igen som med Storage Accounten
Vælg herefter den location der er tættest på den fysiske lokation der skal benytte storage accounten mest. Dvs. er der tale om en dansk løsning, så vil det kunne betale sig at vælge ”North Europe” mens var vi nu i Seattle, så ville jeg vælge North America. Dette alene pga. netforbindelser og afstande.
Angiv et brugernavn og password, der skal benyttes ifbm administration af serveren
Har du allerede en SQL server oprettet, kan du oprette en database på denne.
Giv den et navn, vælg din subscription, og derefter en server hvorpå den skal bo.
Det er vigtigt at oprette en Firewall regel der tillader alle AZURE services at tilgå din netop oprettede server og database – gøres dette ikke er det ikke muligt for servicen at benytte databasen
Opret herefter et Hdinsight cCluster,
Angiv et Cluster navn, hvor mange noder der skal benyttes, minimum 2 noder hvis det er produktion. Er det test eller demo kan det være rigeligt med 1 node, dog er der så ikke meget cluster over instansen.
Igen skal der huskes at angives et brugernavn og et password til administrationen af clusteret
Ship harddrives to Microsoft, these have to be encrypted with bitlocker
Section title slide (Optional)
Konfigurerbar, det er muligt at lave alle
Hvorfor automatisere arbejdet for at skabe stabilitet, gentagelser, spar tid,
Slippe for det kedelige arbejde, og skabe tid til at lave det spændende.
Opgaven med at oprette og nedlægge instanser på Azure er trivielle og det skal gå galt på et tidspunkt, da det er manuelt arbejde.
Demo placeholder (Optional)
Demo placeholder (Optional)
Demo placeholder (Optional)
Content slide – (white background)
Section title slide (Optional)
Normal ETL process on-premises, mix with jobs in the cloud.
Extrapolates. As a developer I know SSIS but not Hive or Sqoop….