Download to read offline
More Related Content
Azure automation invades your data centre
- 1. AZURE AUTOMATION INVADES YOURDATA CENTRE KIERAN JACOBSEN READIFY
- 2. WHO AMI I •TECHNICAL LEAD AT READIFY • INTERNAL AND CUSTOMER INFRASTRUCTURE • BIG FAN OF AUTOMATION
- 3. AGENDA • AZURE AUTOMATION •AZURE WORKER LIMITATIONS • HYBRID WORKERS, GROUPS, LIMITATIONS • AZURE AUTOMATION AUTHORING TOOLKIT • WEB HOOKS
- 4. AUTOMATION CAN MEANMANY THINGS • CLOUD SERVICE AUTOMATION • INFRASTRUCTURE AUTOMATION • PROCESS AUTOMATION
- 5. AZURE AUTOMATION • MANAGEDSERVICE • AZURE AND CLOUD FOCUS • BACKED BY POWERSHELL • DR, HA, PROVISIONING, MONITORING, PATCHING, BACKUPS • HIGHLY AVAILABLE
- 6. CONCEPTS • ACCOUNT • RUNBOOKS •ASSETS • JOBS • WORKERS
- 7.
- 8. AZURE WORKER LIMITATIONS •LIMITED TO SPECIFYING WHICH AZURE REGION • NO CONTROL OVER IP ADDRESS • TRACEABILITY • FIREWALLS • LIMITED CONTROL OVER MAKE UP OF AZURE WORKER
- 9. HYBRID WORKERS • RUNBOOKSRUNNING WITHIN YOUR DC • REQUIRE OPERATIONS MANAGEMENT SUITE WITH AUTOMATION SOLUTION/PLUGIN • SUPPORT SCRIPT, WORKFLOW AND GRAPHICAL RUNBOOKS • NO INBOUND FIREWALL REQUIREMENTS
- 10.
- 11. HYBRID WORKER GROUPS •COLLECTIONS OF WORKERS • RUNBOOKS ARE EXECUTED AGAINST GROUPS • IDEAL FOR PROVIDING HA • SHARE “RUN AS” PERMISSIONS
- 12. DEMO RUNNING A RUNBOOKACROSS MULTIPLE WORKERS & CONFIGURING “RUN AS”
- 13. HYBRID WORKER LIMITATIONS •MODULE DEPLOYMENT • EXECUTION CONTEXT • NO SIMPLE FILE OR EVENT TRIGGERS • NO PRIORITISATION OF WORKERS IN A GROUP • DOCUMENTATION
- 14. AZURE AUTOMATION AUTHORINGTOOLKIT • MANAGE AZURE AUTOMATION ACCOUNTS FROM ISE • CREATE, EDIT AND MODIFY RUNBOOKS AND ASSETS • AVAILABLE FROM THE POWERSHELL GALLERY HTTPS://WWW.POWERSHELLGALLERY.COM/PACKAGES/AZUREAUTOMATIONAUTHORINGTOOLKIT
- 15. DEMO USING THE AZUREAUTOMATION AUTHORING TOOLKIT
- 16. WEB HOOKS • STARTJOBS FROM HTTP REQUESTS • IDEA FOR APPLICATION AND 3RD PARTY INTEGRATION • GREAT FOR STARTING JOBS IF AZURE CMDLETS ARE NOT INSTALLED • RUNBOOKS MAY NEED MODIFICATIONS TO RUN FROM WEBHOOKS
- 17.
- 18. LINKS • BLOG: HTTP://POSHSECURITY.COM •TWITTER: @KJACOBSEN • RUNBOOKS FROM THIS PRESENTATION: HTTPS://GITHUB.COM/POSHSECURITY/POSHSECURITYAZUREAUTOMATION • HYBRID WORKERS: HTTPS://AZURE.MICROSOFT.COM/EN-US/DOCUMENTATION/ARTICLES/AUTOMATION-HYBRID- RUNBOOK-WORKER/# • WEB HOOKS: HTTP://BLOG.CORETECH.DK/JGS/AZURE-AUTOMATION-USING-WEBHOOKS-PART-1-INPUT-DATA/ • AZURE AUTOMATION AUTHORING TOOLKIT: HTTPS://WWW.POWERSHELLGALLERY.COM/PACKAGES/AZUREAUTOMATIONAUTHORINGTOOLKIT
Editor's Notes
- #2 Hi everyone, My name is Kieran Jacobsen and tonight I want to talk to you about Microsoft Azure Automation and using the new hybrid workers within your data centre.
- #3 So just a little bit about me. I work as a Technical Lead at Readify, my role is to manage and support not only Readify’s infrastructure but that of our customers as well. In terms of scale, we are looking at almost 100 Azure subscriptions, and similar numbers of Azure AD and VSTS instances. This causes some unique challenges and requires some unique solutions. I have also lived a lot of what we have preached. About a year ago, we made the call to move from co-located infrastructure to Azure IAAS, this has lead to me, as a system administrator to live a lot of the things we have often spoken and heard about. I have been working on moving all sorts of infrastructure components from a classic on premise deployment to Azure. It has been one of the most interesting infrastructure projects I have ever worked on. Automation has always been a massive thing for me. Since my first job, almost 10 years ago, to now, I have always made use of automation. Anything that can make my job easier is something I will want to do. In my first role, I was the Windows guy in a team of Unix and mainframe engineers. Automation here was all about server deployments and server maintenance tasks, with a little bit of monitoring thrown in. At my next gig, I was automating all the tiny bits and pieces the team were doing, from WSUS to Certificate authorities and the bits in between. I then ended up supporting a bank. Now that was some crazy level automation! The automation there wasn’t the traditional server deployment or user management automation, in this case it was moving files around representing millions of dollars. Precision was a must, you needed to be absolutely positive that things occurred in a certain way, every time, and if something didn’t happen correctly that people knew about it. When peoples pay checks are in the balance, you need to ensure you know what you are doing. Now at Readify, I find myself automating all sorts of things. In the past 12 months I have looked at automated user creations, deletions and just keeping user data in sync between traditional systems like HR and Payroll systems, to systems like Active Directory and then on to cloud systems like Office 365, Azure Active Directory, CRM Online and a bunch of other places. I also need to automate infrastructure management and deployment tasks, like deploying servers, setting up DNS, configuring Office 365, cleaning up log files, buying certificates, backing up and restoring files, even whitelisting TOR addresses at times.
- #4 So tonight's agenda is pretty simple, we will cover off what Azure Automation is and some basic concepts, look at the limitations of the Azure Work, then take a look at Hybrid Workers, groups and their limitations. I will then show you the Azure Automation Authoring Toolkit and web hooks. We will finish off with a nice end to end demo showing off some user creation steps.
- #5 So one of the big things about automation is that it means so many different things, different people have different ideas and goals for automation. For some, they see automation as just something that occurs between different cloud systems, they want to automate between different platforms using their publicly available API. Now the big thing for this style of automation is that the automation is typically outside of our network, we are often connecting multiple public cloud systems together. Azure Automation was originally designed with this style of automation in mind. Now for those who have more of a system administration background, they might see automation as something that happens on premise, within the corporate network. Often or not, some of the automation tasks within our environment focus on lower level infrastructure, this might be our core network switches, storage area networks, or even a mainframe. Infrastructure automation like this often requires not only a connection to the corporate network, but we might need to install third party applications. We typically are not going to open up our core switching infrastructure to the Internet, so we need our automation system to be connected to our network. Products like CA UniCenter are great examples in this space. Finally we have process automation. Process automation aims at turning existing business processes, no matter what they maybe, into repeatable execution steps. Process automation doesn’t just target the little things, like how do we deploy a server, or how do we create a user, but looks at the business process from start to finish. With process automation, we are not just looking at automating things that occur in the IT team, but the enterprise as a whole. One of the big things with process automation is that it may require access to all sorts of part of the enterprise, be it our internal network or cloud services.
- #6 In April 2014, Microsoft released a preview of Azure Automation. Microsoft’s goal with Azure Automation is to provide a managed service for scripting and automation, focusing on simplifying cloud management with process automation. It is really important to know that from its early infancy, Azure Automation was heavily designed to provide automation for Azure and third party cloud services. We will see later on, that many of the limitations it has as a product, come from this focus. One of the best thing about Azure Automation, is that it lives and breathes PowerShell. At first it only supported workflows, thankfully this changed and since last year it has not only supported standard PowerShell scripts, but it now includes a graphic runbook development method. Azure Automation really is about targeting our processes, and not just individual tasks. Processes like disaster recovery, scaling and high availability, provisioning, monitoring and patching are a big focus in Azure Automation. In fact most of Microsoft’s early examples focused on these aspects. The big thing though with Azure Automation, and the reason I am such a huge fan, is its availability. One of the big risks with the more traditional automation platforms is the loss of your primary server can often throw your entire environment, and even your organisation into a complete spin. Anything that keeps my organisation going and doesn’t require me to wake up at 3am to fix, is freaking awesome.
- #7 So there are a few concepts you should be aware of when looking at Azure Automation. At the top, we have our automation account. Your account contains everything you want to do and everything you need to make it happen. Next we have runbooks. Runbooks contain our processes or procedures that we want to execute in a repeatable fashion. Think of these as checklist almost, if you want to get something done, we will follow the steps outlined in a runbook to accomplish the required task. Assets are reusable components or items that are shared across all runbooks, they could be schedules specifying when our runbooks should be run, or they could be PowerShell modules, certificates for authentication, credentials, connections or variables. Variables store pieces of information that we might need across multiple runbooks or the execution of the same runbook. Variables can be strings, Boolean values, integers or datetime values. Jobs are an executed instance of a runbook. Jobs contain a snapshot of the runbook and required assets at the time when it was started. Jobs get executed by workers, and have a state of either new, completed, suspended, queued, running, failed or stopped. Finally we have the often workers. This is an often overlooked component of any automation system. Originally, there was only one type of worker and it ran on Azure. Whilst this was good for quite a few automation tasks, it has some serious limitations.
- #8 So let’s take a quick look around Azure Automation. So here is my automation account in the new Azure Portal. You can quickly see how I have three runbooks, 18 assets, a dsc configuration, a hybrid worker group and 2 dsc nodes. You can also see that I have had a whole bunch of jobs run, most completed, but there are some suspended, and a few failed as well. Scrolling down, you can see that my runbooks are synchronized from GitHub. If you are not using some sort of source code repository for your scripts, please start to do so. It is 2016, and we should all be using something, git, VSTS, it doesn’t matter as long as you are using something. Right now Azure Automation only support GitHub, VSTS is coming soon. Hopefully VSTS will be coming very soon. If we drill in to Runbooks, we have the runbooks you will be seeing tonight, going back and selecting assets, you can see that we haven’t got many. I have 13 modules, Azure Automation comes with 10 normally, however I have added some DSC modules in. Whilst we are here, why don’t we run a runbook. If I go to Runbooks, and select Get-MyFirstRunBook, this book simply returns a nice hello world message. Let’s hit run, for now we will specify to run it on Azure. Now we will wait for the job to complete, notice the states it is going through, queued, running and then hopefully completed. I can also view a list of all of the jobs by selecting the jobs tile under details. Here I can see jobs as they are running, as well as go back and view previous jobs. Overall the interface is pretty easy to navigate, if just a tiny bit annoying with all of the blades. Let’s talk about Azure Workers.
- #9 Microsoft designed Azure Automation as a platform where we could run automation tasks from anywhere, call it free range automation. Whilst it is fantastic that we can run our automation tasks from anywhere, we often need to know actually where these tasks are being executed. Whilst we do get to specify which Azure region our account is created, and that to an extent will control where our Azure Workers are located, that is it in terms of control. This lack of control introduces a bunch of challenges. We can’t specify what IP address Azure workers have, we can’t even specify it as a static address like we can with our virtual machines. Now some people might ask, why is this important? Well it introduces a few issues. Have you ever tried to confirm if an event log entry was caused by a worker or a malicious user? Turns out this is pretty tricky. Whilst we can go back and see some of history information for previous jobs, the IP address of the worker sadly isn’t one of them. This might seem silly to some, but this is crucial for quite a lot of enterprise environments. If you get pwned, you are going to need to make sense of those log files. Ever wanted to create firewall rules for the incoming connections from an Azure worker? That is going to be hard. Some of us need to integrate our automation with systems that have IP restrictions, things like HR and payroll systems, private APIs and payment gateways. I need to be able to tell some network guy at a partner organisation what IP address to expect a connection from. What is the make up of the worker? What operating system, the .net framework version and PowerShell version does it have? With the Azure Worker, we don’t have any control over these, we get the versions Microsoft tells us we can have. Whilst we can specify additional PowerShell modules, if it is more complex than that, we are at a dead end. Say we needed a 3rd party vendor application to accomplish our automation goals? Well, we are out of luck. We don’t even have the option of connecting a worker to an Azure virtual network. If we could do that, we would at least have corporate network connectivity. Now one solution would be to use Windows Remoting and have the worker connect to a server within our control, but this opens up issues with double hop, credssp and firewall rules. Overall, these limitations can make it a hard for Azure Automation to be adopted into enterprise environments.
- #10 Enter Hybrid Workers. Hybrid workers allow us to develop more advanced runbooks than we could previously, allowing for runbooks to access resources within your network, integrate with 3rd party frameworks, and give us finer grained control over the execution environment. They solve many of the limitations with the Azure worker. To make use of Hybrid workers, you will need to implement the Operations Management Suite. Now I haven’t tested if hybrid workers will function if you are using OMS via the SCOM connector, however I have read of this being possible. For my production environments, and even tonight’s presentation they are direct attached. You will also need to install and configure the OMS Automation solution as well. Hybrid workers support all three runbook types, and most importantly you don’t need to open any inbound firewall ports, instead the worker agents will connect out to Azure over HTTPS, and monitor for jobs that they need to perform. I have taken a peak at the internals, and all of this is achieved via Azure Service Bus. I really do wish I could hook PowerShell into custom Azure Service Bus instances as well, if anyone has any neat solutions, please let me know. Now Microsoft’s documentation here refers a lot to resources within your local data centre, however I see hybrid workers as being highly useful to IAAS situations just as they are on premise. Let’s take a look at hybrid workers.
- #11 So lets run our first job on a hybrid worker. For tonight's demonstrations, I have two Windows Server 2012R2 servers, they are domain controllers for a domain called CORE. Firstly, I am going to show you the OMS console. In the OMS console, you can see that I have the automation solution added, and it is configured to my azure automation account, poshsecurity-aa. Let’s go back to the Azure portal. Whilst I have my hybrid workers already configured and running, if you wanted to set your own up, there are two values you need, and we get both of these from the Key icon here. We need to take a note of one of the access keys, and then the URL endpoint for our azure automation account. Adding a hybrid worker is as simple as calling add-hybridrunbookworker, and specifying these two values and the name of the group to add them to. We will talk about the groups in a minute. Let’s take a look at our group, if I go into Hybrid Worker Groups, we can see a single group. Digging in to that, we can see there is two hybrid workers, DC01 and DC02. Now on to running our first hybrid job. I am going to run a job called Get-Hostname. This runbook simply outputs the hostname of the worker it is running on. If we hit start on this runbook, we will be asked once again where do we want to run this job, let’s select hybrid and then our domaincontrollers group. Now this is going to be queued up, and then executed, once it is completed, lets look at the output. As you can see, that DC01/DC02 the hostname of one of our works is displayed.
- #12 Hybrid Worker Groups are collections of workers, a little bit like a server farm, that can complete our automation activities. There is no reason why we couldn’t have multiple groups, each configured or placed in different places on our network. You might have one group setup that has access to your internal HR systems, another group might near your webserver farm to perform activities there. When a Job is created, one, and only one worker in the group that job has been assigned to, will complete it. Don’t think of groups as load balancing, whilst they will to an extent distribute the jobs, this isn’t so much designed for load balancing and more designed for high availability. Now just to note, the failover isn’t as smooth and as seamless as it could be. If a worker does fail, it make take some time for everything to work it out. The main driver for work groups is to ensure that we always have a worker available to complete our automation tasks. Workers in a group do not need to be in the same data centre, they could represent geographically dispersed systems at multiple locations for availability. Workers run jobs under the same execution also called a run as account. No matter what runbook job is sent to the group, they are all executed as the same account.
- #13 This time, why don’t we start a bunch of jobs and see what happens. I have some PowerShell code, here that will spin up 5 jobs for us, and then read the output back for us. for ($a = 0; $a -le 10;$a++ ) { "Starting Job $a" $null = Start-AzureRmAutomationRunbook -Name 'Get-Hostname' -RunOn 'DomainControllers' -ResourceGroupName 'poshsecurity-aa' -AutomationAccountName 'poshsecurity-aa' } $Jobs = Get-AzureRmAutomationJob -ResourceGroupName 'poshsecurity-aa' -AutomationAccountName 'poshsecurity-aa' | select-object -first 10 foreach ($job in $Jobs) { (Get-AzureRmAutomationJobOutput -Id $job.id -ResourceGroupName 'poshsecurity-aa' -AutomationAccountName 'poshsecurity-aa').text } We should see that some ran on DC01 and others ran on DC02. Pretty neat Eh? Now let's take a look at changing the account that these runbook jobs are running as. So I have another runbook, Get-RunningUser, will simply return as output the user account that we are running as. Let's run it and see what it returns. So let's select to run on the hybrid worker. And we can see that it returns that the runbook was running as nt authority\system. Now before we change the account jobs will be run as, we need to ensure we have a credential asset defined with the appropriate settings. If I go to assets, and then credentials, you can see I have one called AutomationAccount. These are domain credentials that we want to use to run our jobs. Now if I go back into the group settings, then select "hybrid worker group settings". Now as you can see, we have the run as selected as "default". Let's select custom, next we will be asked to select a credental, and select the AutomationAccount. I am going to save, and go back to runbooks, and run the get-runninguser. And if we look at the output, then we see that the account is core\azureautomation, which is the user it was configured for. Who here is sick of all the jumping around in the portal yet? I know I am.
- #14 Unfortunately, all this comes with some limitations. Now most of these might not be a show stopper for you, they might not even be an issue, it is still best that you are aware of them. Modules are not automatically deployed to hybrid workers. Unlike with Azure Workers, modules installed as assets will not be deployed automatically. Either script the prerequisite module install or use DSC. If you have come this far, why not sure Azure Automation DSC? Execution context, as I mentioned earlier, is tied to the worker group. Now for most people, you probably don’t care about executing one runbook as a different user account than another. Thankfully there are some easy solutions to this one. Now I for one would like to see file close triggers, and I would love if the story of trigging from event logs was much simpler. You certainly can trigger jobs from Windows events, but it is a lot of work. One thing that would be nice to see is weighting or prioritization within the worker groups. It would be nice to be able to say, run the runbooks here on this worker, unless it is dead. Each hybrid worker in a group has the same chance to perform the job as the others. Whilst this might not cause issues to most people, there are probably situations where this could be an issue. Now for the big one. Documentation. Right now, there is quite a bit in Azure that either isn’t documented, has minimal documentation or the documentation contains errors; and this goes well and truly for Automation and particularly hybrid workers. The documentation actually says that you cannot change the execution context, it also says that web hooks cannot trigger jobs on hybrid workers. These things will get fixed, but I recommend that you don’t trust the documentation, just because it says one thing, doesn’t mean that is actually the case.
- #15 Who here has heard of the Azure Automation Authoring Tool kit? It is also called the Azure Automation ISE Add-On? The toolkit makes working with Azure Automation incredibly easy, by bringing all of the elements of Automation into the ISE. We can manage automation activities, create and edit runbooks and assets locally, push changes to our Automation account and also test PowerShell workflows and scripts locally, in Azure Workers and in Hybrid Workers as well. You can even synchronize the automation account with your github repos right from the ISE. I was only put onto this about 3 or so weeks ago, and I have been amazed how useful this has been to me. It reduces the amount of time spent randomly clicking around the new Azure Portal. There are a few limitations; you can’t setup webhooks or schedules on runbooks, and you can only modify some assets, connection, credentials and variables. Let’s take a look at a quick demo of the toolkit.
- #16 So here I have the PowerShell ISE. As you can see over to the right, I have the add-on visible. On the first Tab, you can see the base path of where the add-on will store runbooks and assets, you can see that I have signed in to azure, selected a subscription and an azure automaiton account. On the next tab, you can see my runbooks in the account. From here I can download them locally, create new runbooks and delete runbooks. If I make changes to a runbook, I can upload a draft back to the automation account, test the draft, and finally publish the draft. I can also synchronize the Azure Account with the associated source control repository. On the Assets tab, I can work with items like credentials and variables. Let’s run a runbook from here. If I click on the Get-HostnameRunbook, this is the runbook we have been using earlier, and then select “Test”, I will be presented with this test screen. From here If I select Start new Job, we should see the execution <click>. It is going to ask us where we want to execute the job, lets select the domaincontrollers hybrid worker group. We can see that the job has been created, and its status is new. It will go to the running status, and we should eventually see it completed, and get the output.
- #17 Web Hooks are a surprisingly useful way to trigger off runbook execution. From a single HTTP request we cat start a configured runbooks execution. Web Hooks are suited for integrating Azure Automation into things like your build and deployment pipelines, VSTS, GitHub, Slack, SharePoint or pretty much anything else you can think of. They also provide us with an alternative to triggering runbooks when we don’t have the Azure CMDLets installed, or in situations where we don’t want to maintain large execution workflows with third party applications. I am currently looking at Web Hooks as a simpler way for team members who don’t have the Azure CMDLet stack installed to trigger off chunks of automation. The idea would be that they simply run the CMDLet, invoke-stuff, and that would simply perform the JSON call to start the process off. They don’t need to have the unstable Azure CMDLets installed, nor wear the performance hits of trying to login in to Azure, they execute a tidy bit of code, and then Azure Automation does the rest. The other advantage to this approach is my team wouldn’t need to maintain large git repos locally and ensure they keep them updated. One limitation with Web Hooks is that they currently do not integrate with the normal parameter mechanism. Runbooks triggered by web hooks will need to be configured to receive a web hook parameter, this contains things like the Web Hook name, the request headers and the request body. The Automation environment doesn’t provide any assistance with this, if you are sending information as a web hook, say via json, you will need to convert it from json in the runbook. I don’t want to scare you off web hooks, they are extremely powerful, and extremely useful in our automation life cycle.
- #18 So I am going to show you two demos on integrating with web hooks. We will start by creating our own webhook and calling it from PowerShell. Firstly we go to the Runbook that we want to run, and select webhook. We then customize the settings, entering a name, expiry and make sure you copy the URL!!! Now this runbook doesn't need parameters, but don't forget to say, run on hybrid worker. Then we hit create. Now that it is created, we go to powershell, and call invoke-restmethod, the url we specified, and the method post. When this returns, it will return the Job ID for the job we just kicked off. And If I switch back to the azure portal, we can see a job has been queued up, and has been executed successfully. Now let’s look at a more interesting example. Let’s take a look at it in the ISE. New-ADUser is a runbook that accepts data via a webhook. Specifically firstname and lastname. It will then create an active directory user based upon that information. After it creates the account, it is going to send me a message in Slack with the accounts password. I have also created a little function to kick the whole process off. So Let’s paste that into a PowerShell window, execute it, and then switch over to slack. And there we have the password, and if I look in AD, we can see the account has been created.
- #19 So that is everything I have for tonight. I want to thank you all for coming and listening to me. I will be posting up the slide deck on my blog, PoshSecurity.com. I write heavily about automation and security, and often security automation. You can follow me on Twitter at @kjacobsen. The runbooks for tonight are in GitHub, you can find them at that address. I have included some links, I recommend the link on web hooks, it covers quite a bit of the basics. Finally, I recommend that you look at the Azure Automation Authoring Toolkit. Once again, thank you. Does anyone have any questions?