Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

VMs All the Way Down (BSides Delaware 2016)


Published on

Just getting started in InfoSec and need some guidance on virtualization? Used virtual machines before, but want to expand to a more complex, dedicated virtual lab? This talk will cover the numerous hardware and software options you should consider, and will discuss both simple and complex configurations. The focus will be on setting up a lab that is home friendly, inexpensive, and as flexible as possible. Offense and defense setups will be discussed, as well as recommendations for virtualization software, server hardware, and networking gear. You will leave with a list of VMs to use, an understanding of the benefits of hosted vs. bare metal hypervisors, different virtualization packages, and how to build an inexpensive lab that emulates a multi-tiered corporate environment.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

VMs All the Way Down (BSides Delaware 2016)

  2. 2. Who Am I? John Hubbard Lead Analyst for GlaxoSmithKline’s US SOC Community SANS Instructor ◦ GMON, GPEN, GREM Guy who has set up LOTS of labs Twitter: @JHub908 Blog: 2
  3. 3. Topics Why? Types of virtualization – Type 1 vs. Type 2 Software options for virtualization Virtual switches, VLANs, and routers Suggested hardware Suggested virtual machines Balancing requirements, price, and complexity Suggestions & sources for cheap or free hardware/software 3
  4. 4. Why? Learning! Attack ◦ One on one type attacks ◦ Pivoting through environment Prevention, detection & response ◦ Firewalling & OS hardening ◦ NSM (IDS, SIEM) & CSM (Log Collection, Vulnerability Scanning/Analysis) ◦ Malware reverse engineering and forensics System Administration | Engineering | Design ◦ Virtualization, networking, secure architecture ◦ Operating Systems Emulate a company infrastructure in one computer! 4
  5. 5. Where We’re Going 5
  6. 6. Where We’re Going 6
  7. 7. Consider Your Goals Infosec ◦ Attack centric ◦ Defense centric ◦ Secure architecture Learn how to use production Hypervisors? ◦ ESXi, XenServer, Hyper-V Stationary or portable all-in-one lab? In-line, or lab as host on home network? Do you care about power/noise? 7
  8. 8. The Focus Of This Talk Get as close to a “real” network as possible… While minimizing cost for ◦ Software ◦ Hardware ◦ Power Maximize ◦ Flexibility ◦ Efficiency / silence ◦ Significant other acceptance factor 8
  9. 9. How Are We Going To Do It? Software ◦ Use FOSS to save on software cost ◦ Leverage free “home” licenses when possible ◦ Leverage trials intelligently when no free option exists Hardware ◦ Use virtualization to cut hardware / power cost ◦ Virtualize endpoints ◦ Virtual network infrastructure ◦ Virtualize security & Monitoring infrastructure ◦ Minimize hardware purchase ◦ Alton Brown theory for hardware – NO to uni-taskers 9
  10. 10. Hypervisors TYPE 1 VS. TYPE 2 10
  11. 11. Virtualization Options 11
  12. 12. Type 2 Options VMware Workstation (not player) ◦ Great choice, not free Oracle VirtualBox ◦ Great choice, free QEMU/KVM ◦ Free, supports other architectures MS Windows Client Hyper-V ◦ Works, but not recommended Parallels ◦ Works, but not recommended 12
  13. 13. VMware Workstation / Fusion Player ◦ No simultaneous VMs or snapshots, won’t work for us Workstation (PC), Fusion (Mac) ◦ Great choice, “industry standard” ◦ Workstation $250/$99 upgrades, Fusion $80/$50 upgrades ◦ Integrates with ESXi – Use VMs over network like local ◦ 64bit host CPU required, VT-X required for 64bit guest Recommended if: ◦ You’re willing to pay for it ◦ Want to control local and remote ESXi VMs ◦ Need compatibility with almost any prepacked VM 13
  14. 14. Oracle VirtualBox Free! PC, Linux, macOS The other “standard” Hardware virtualization not required for 32bit Can run headless ◦ VRDP over network for VMware like experience Recommended if: ◦ You like free things that work very well ◦ You want to use old hardware for lab 14
  15. 15. Other Type 2 Options Windows Client Hyper-V ◦ Free with Windows 8/10 Professional ◦ Don’t see any benefits over VirtualBox ◦ You might like the OS integration QEMU/KVM (Linux only) ◦ Generic FOSS virtualization solution ◦ BYOGUI – Virt-manager makes like VMware / VirtualBox ◦ Run different architectures (ARM, PowerPC, MIPS) ◦ Recommended if: You like virtual Raspberry Pi Parallels (macOS only) ◦ Should work fine ◦ Same price as VMware and less compatible 15
  16. 16. Type 1 Options ESXi aka vSphere Hypervisor ◦ Business “standard”, free for home use, limited features, HW compatibility issues, Windows required* XenServer ◦ Business “standard”, free, Windows required for mgmt., HW Compatibility Issues Proxmox ◦ Free, supports most HW, no feature restrictions, web management Hyper-V Server ◦ Free, supports most HW, Windows required, wants Active Directory, painful 16
  17. 17. vSphere Hypervisor (ESXi) “Industry standard” solution for Type 1 Picky about hardware ◦ Can build a whitebox, use custom network drivers Windows thick client management* ◦ New web front-end available, still slightly buggy Free version has feature restrictions – shouldn’t matter Recommended if: ◦ Your hardware is compatible (Server HW or whitebox, check HCL) ◦ Want most user friendly experience ◦ Want to learn an industry standard 17
  18. 18. Proxmox VE Free, can buy support - think “VirtualBox of type 1 hypervisors” Debian based, uses KVM No restrictions – VM migration, clustering, unlimited cores Compatible with most hardware Web front-end Recommended If: ◦ Want totally open and free solution ◦ Hardware isn’t compatible with ESXi ◦ You are comfortable with some CLI and Google ◦ Clustering / centralized management wanted ◦ Don’t want to manage your lab with a Windows PC 18
  19. 19. XenServer The other “industry standard” (AWS, Linode, Rackspace) Picky about hardware Free No restrictions Managed by XenCenter Windows thick client  Recommended if: ◦ ESXi doesn’t work, still want “professional solution” ◦ Hardware is compatible ◦ Don’t mind using Windows to manage it 19
  20. 20. Microsoft Hyper-V Server Might use it at work Free Frustrating if not on a domain Obviously – Windows based management Recommended if: ◦ You like pain ◦ You have a good reason ◦ You have a specific need for this ◦ Running active directory at home 20
  21. 21. Type 1 vs. Type 2 Considerations Extra computer? Travel? ◦ Use Type 2 if you don’t have an extra computer, need it to travel Is your hardware compatible? ◦ You probably can’t run ESXi / Xen on a laptop, lucky if desktop works Networking Gear ◦ Do you have a ”real” Router/firewall/access point, can you make one? What VMs are you running? ◦ QEMU enables non-x86 VMs How do you want to manage it? Windows? Cost? Recommendation: VMware all around, or VirtualBox / Proxmox 21
  22. 22. Still Not Sure? Try them all...with nested virtualization! Use type 2 to run type 1! Install VMs in that! Test your test lab, move VM’s when ready! 5 minutes of clicking “next” to Install them all ◦ Enable VT-X for VMs (in processor settings) ◦ Add 2 virtual NICs (Required by most, 1 for mgmt., 1 for VMs) ◦ Ensure enough RAM, might not boot without it 22
  23. 23. Lab Hardware WHAT DO I NEED? 23
  24. 24. Planning Your Lab - Hardware RAM – MOST important, 1st limiting factor HDD – 2nd limiting factor, speed is nice, size most important CPU ◦ VT-X – Consider this a requirement (some super cheap old servers lack it) ◦ VT-D – Can pass PCI devices through to VM, might want ◦ AES-NI – Efficient drive encryption Package ◦ If it needs to travel – Laptop/NUC ◦ If you want quiet, expandable - Desktop ◦ Turn down for what?! – Rack Mount! Minimum specs: 16GB RAM, 500GB HDD, i5+ from last few years Ideal: 32GB+ RAM, 1TB+ SSD, quad core i7+ 24
  25. 25. My Favorite Hardware Whatever you already have + Proxmox Laptop: Refurbished ThinkPad from Newegg ◦ X220+ (small), T420+ (mid-size, extra HDD) ◦ $185-$400 + RAM upgrade Tower: Lenovo TS140 / TS150 (new version) ◦ Super Quiet ◦ $289 for i3 version + more RAM / HDD ◦ $389 for Xeon (preferred) + more RAM / HDD ◦ Need NIC for ESXi – read Lenovo notes Rack Mount: ◦ Consult wiki ◦ Many considerations 25
  26. 26. Lab Network 26
  27. 27. Planning Your Network Goal: Take fewest pieces of hardware – emulate any network Pieces you need: ◦ Firewall / Router – Virtual, or multiple interfaces with VLAN support ◦ “Smart” switch – Capable of VLANs (802.1q) & traffic mirroring ◦ Wi-Fi access point – VLAN / multi-SSID capable ◦ Server Depending on what you want, the first three might be one item 27
  28. 28. Decision Time Do I have … ◦ An extra machine and want it to be IN-LINE in my network? ◦ Win: Almost everything is Virtual, least hardware ◦ Issue: “The internet doesn’t work, what do I do?” ◦ “Just log in to ESXi go to console and restart our router VM, obviously!” ◦ An extra machine, want it to be another host on my home network ◦ Pro: Won’t ruin your tubes ◦ Con: Might have to buy stuff ◦ Have a dedicated laptop, lab can travel ◦ Whole lab on my primary computer This will drive your network setup 28
  29. 29. My Physical Network Setup 29
  30. 30. Why This? Splits core components into pieces for flexibility VLANs allow multiple layer 3 networks without tons of NICs Wi-Fi access to each VLAN with different SSID Physical access through switch ports assigned to VLAN Hypervisor allows per VM settings of VLAN Switch mirror port sits at key location to collect ALL traffic All inter-VLAN traffic goes through firewall Additional networks can be virtualized Can emulate almost anything Talk assumes this setup when discussing VLANs 30
  31. 31. How This Looks To A Packet 31
  32. 32. Favorite Networking Gear “Smart” switches – VLANs, port mirroring ◦ $30+ TPLink “easy smart” series* – Windows required before V2 ◦ $78 Cisco SG200-08 – Works for me Router/Firewall: ◦ DIY with PfSense - Free & unrestricted, can run snort too ◦ Sophos XG FW (VM) - Free, polished, and tons of security features, 50 IP limit ◦ Ubiquiti EdgeRouter X - $50, integrated FW, VLANs, VPN, DHCP, DNS, etc. Wi-Fi: ◦ Need a pure AP only ◦ Free - Use your current one in AP mode, bonus switch, DD-WRT? ◦ $90-$150 - Ubiquiti UniFi AC Series – ”enterprise grade”, with VLAN support 32
  33. 33. So I Have To Buy All That? No, you COULD do it all with 1 server! ◦ PfSense/Sophos VM = Firewall & Router ◦ Virtual switches for all zones ◦ Virtual switch port mirroring ◦ Challenge mode: HostAPD for Wi-Fi access point Details coming… 33
  34. 34. Virtual Networking Concepts Need to understand virtual networking concepts Note: Assume “NIC” == real/virtual card with 1 interface Our lab server will have ◦ Virtual machines, with multiple virtual NICs, that connect to… ◦ Multiple virtual switches, that connect to… ◦ Multiple physical NICs, that might connect to… ◦ A virtual router VM Type 2 names connection modes – you’ve likely seen this ◦ Bridged ◦ Nat ◦ Host-Only ◦ Internal (host-only, minus host connection) 34
  35. 35. Type 2 Virtual Networking 35
  36. 36. Type 2 Virtual Networking 36
  37. 37. Type 1 Virtual Networking Same idea - manual implementation without these names For type 1 hypervisor setups, usual mode is bridged Can use other types by not connecting virtual switch to phys. NIC General Process ◦ Define VLANs/segments (ex: DMZ, Desktops, Internal Servers) ◦ Create a group/switch for each VLAN ◦ Map virtual switches to physical NICs ◦ Create VMs and connect virtual adapters to correct VLAN switch/group Idea: Traffic from each VM gets tagged by virtual switch, exits onto actual network with VLAN tag that router acts on ◦ VLANs not needed if lab is your router, just use more NICs / vSwitches 37
  38. 38. VLANs & Trunking 38
  39. 39. ESXi – How To Port Groups – One for each “zone”, VLAN tags apply here Virtual Switches – One / physical NIC (vSwitch0, etc.) ◦ Note: To tap virtual switch - set Promiscuous mode to “accept” Physical NICs – Your actual hardware (vmnic0, etc.) Vmkernel NICs – Where ESXi management page is served at (vmk0) 39
  40. 40. ESXI – How To Create port groups for each zone Assign port groups to correct switch Ensure switch is connected to correct physical NIC Create VMs and assign to groups 40
  41. 41. Proxmox – How To Note: ”Linux Bridge” == virtual switch, I’ll use this term vSwitches assigned to physical NICs IP CAN be assigned to vSwitch, not needed ◦ Note: You can manage Proxmox from all vSwitch IPs – be careful! Check “VLAN Aware” box for each vSwitch To tap virtual traffic – # brctl setageing vmbr0 0 ◦ Makes vSwitch a hub – VMs can see all traffic 41
  42. 42. Proxmox Steps Create VMs, create as many virtual NICs as needed Connect virtual NICS to vSwitches Enter VLAN tags for each virtual NIC 42
  43. 43. XenServer – How To ”Network [x]” is auto-made for each physical NIC Create new virtual switch for each VLAN Assign a VLAN tag & assign new switch to correct physical NIC Google ovs-vsctl command for port mirroring instructions 43
  44. 44. XenServer – How To Create VMs and virtual NICs Assign virtual NICs to VLAN enabled switches 44
  45. 45. VMware Workstation – How To 45
  46. 46. VMware Workstation – How To 46
  47. 47. VirtualBox – How To 47
  48. 48. Virtual Machines 48
  49. 49. What To Install We’ve got hardware, hypervisors, and network. Now? Define capabilities and pick VMs accordingly Connect to network as needed SNAPSHOT! Which VMs to use? Everything! –Windows Desktop/Server, Linux, apps, BSD… Where do you get it? Isn’t that complicated & expensive? 49
  50. 50. Free Virtual Machine Sources Windows ◦ Student? Many free options – collect every server license you can ◦ DigitalRiver, Dreamspark, OnTheHub ◦ site – Free Windows VMs, XP-10! (expire after 90 days) ◦ Download, snapshot BEFORE use, re-arm, revert ◦ TechNet Evaluation Center – 180 day server licenses ◦ Bottom of your laptop?, IT Friends, Craigslist, eBay – 2008R2 = $90 Linux – prebuilt apps ready, without install & setup pain ◦ ◦ 50
  51. 51. Offense HACK ALL THE THINGS 51
  52. 52. Offense Emulate corporate infrastructure ◦ Multi-segment network – DMZ, Desktops, Servers, Guest, etc. Pick a distro - Kali, Pentoo, BlackArch, Backbox Set up network, install OS’s and services Set up virtualized defense – IDS, AV, Firewall, etc. Snapshot! Then… 52
  53. 53. Offense Attack from outside (internet based attack) Attack from Inside (unauthorized device on network) Attack from DMZ, VPN, Wi-Fi, anywhere else Try to pivot around, stop yourself, get around it Bring physical devices into mix - IoT, printers, slow cooker Did your defense pick it up? Script to revert whole environment!! ◦ VMware: Vmrun / vim-cmd ◦ VirtualBox: VBoxManage 53
  54. 54. Offensive Setup Lab is a corp. network Attack machine can be VM You can attack from any point by changing VLANs 54
  55. 55. Defense 55
  56. 56. Network Security Monitoring What is NSM? Network based, data-in-motion focused analysis Security Onion is the king of NSM distros Full packet capture - Netsniff-NG Snort / Suricata / Bro IDS Sguil / Squert IDS front-end ELSA – Log collection and searching (SIEM) Xplico, NetworkMiner, etc. for PCAP forensics EASY install 56
  57. 57. Security Onion – How To Make sure you have resources ◦ 3GB+ RAM required ◦ CPU needs based on traffic ◦ Enough space to save it all ◦ Check current router for bandwidth usage / month Plan what you want to monitor ◦ Whole network? Tap at physical switch with everything behind it ◦ Just your server? Use virtual tap from vSwitches ◦ Just a couple VMs in type 2 setup? Connect to same vSwitch 57
  58. 58. Security Onion / NSM Setup Add server for Security Onion Copy ALL traffic from network to 2nd NIC NIC only connects to Sec. Onion VM 58
  59. 59. Log Management VMs Splunk Free ◦ Collect logs from your environment ◦ 500MB / day Windows Log Collection Server ◦ Not often done, but can consolidate logs in windows for free ◦ NSA Guide: “Spotting the Adversary with Windows Event Log Monitoring” OSSIM ◦ Free SIEM from AlienVault ELK ◦ ElasticSearch, Logstash, Kibana ◦ FOSS stack for log analytics 59
  60. 60. Malware Analysis Want to run malware in contained environment Internal mode or host-only (isolated from internet) mode Multiple hosts options is ideal ◦ Malware may do different things based on OS version / domain or not REMnux is perfect distro for analysis – Think Kali for malware RE ◦ Created by Lenny Zeltser – SANS FOR610 Author ◦ Tools built in and auto-update Built for static and dynamic analysis Can easily intercept traffic, pretend to be network services ◦ Fakedns, inetsim 60
  61. 61. REMnux – How To Use an isolated vSwitch with host-only / internal networking Connect REMnux VM Install victim VMs - Linux, Win XP, Win 7/10, Win Server, etc. Set all VMs to use REMnux VM IP as gateway Install tools for analysis Snapshot everything - multiple times along tool install path Begin traffic interception Infect, analyze hosts and traffic “outbound” Revert snapshots, rinse and repeat 61
  62. 62. Malware Analysis Setup All VMs have REMnux IP as gateway No internet connection to any VM Host PC connection still active for VM control 62
  63. 63. All-in-one Lab Lab box is your home router, firewall, lab switches, and all VMs 2-3 physical NICs required ◦ To internet ◦ To switch (for normal network devices) or Wi-Fi AP if all wireless ◦ Cheap win - switch / AP could be your old router with DD-WRT in AP mode ◦ VLAN support unlikely, use 3rd NIC to plug directly in to VLANs Inline with your network – beware down time! ◦ Mitigate with simple home Wi-Fi router, ready to go as backup 63
  64. 64. Kitchen Sink Mode 64
  65. 65. Other VM Ideas WebGoat / Security Shepherd / SamauriWTF – Web app attack training SamauriSTFU – SCADA, smart meter, other energy sector Cybatiworks – ICS with physical kit. Would make interesting demo Vulnhub, Metasploitable Forensics – SIFT / DEFT Huge list: 65
  66. 66. Taking It Further Containers ◦ Built in to Proxmox, even MORE efficiency Automation ◦ Vagrant – building your VM ◦ Scripts to bring up and down whole environment at once Cloning ◦ Make a bunch of “users”, pivot Virtualize your real infrastructure & test that Honeypot VMs Reference: Wiki 66
  67. 67. Further Reference “Setting up a Test Lab with VMware” – Nicholas Chapel (BSides MSP) ◦ ◦ Focused on walkthrough of installing ESXi and setting up a VM “EC2 or Bust - How to Build Your Own Pen Testing Lab in Amazon EC2” – Grecs (BSides LV) ◦ ◦ Focused on cloud lab setup “Building a Cyber Range” – Kevin Cardwell (ShowMeCon) ◦ ◦ Focused on pen testing and ideas for making your lab emulate customer environments ◦ Book: “Building Virtual Pentesting Labs for Advanced Penetration Testing” “Proxmox Cookbook” – Wasim Ahmed 67
  68. 68. TL;DR – Free Full Type 1 or 2 Lab Extra computer or refurb laptop - i5+, 500GB HDD, 16GB RAM Proxmox or VirtualBox with Linux OS Define network segments, make vSwitches Install PfSense with multiple virtual NICS, one for each segment Get VMs and connect vNIC’s where needed ◦ / TechNet Eval center free windows VMs ◦ Bitnami / Turnkeylinux easy install app VMs Install defense / offense VMs (Kali / Security Onion) Tap virtual traffic with virtual tap or ”smart” switch Hack the planet! 68