VMware Log Insight
•Download as PPTX, PDF•
1 like•2,580 views
VMworld 2016 presentation. Sharing of vRealize Log Insight
Recommended
More Related Content
What's hot
What's hot (20)
Similar to VMware Log Insight
Similar to VMware Log Insight (20)
Recently uploaded
Recently uploaded (20)
VMware Log Insight
- 1. Insight into the World of Logs with VMware vRealize Log Insight Iwan Rahabok, VMware Karl Fultz, VMware Manny Sidhu MGT7685 #MGT7685
- 2. • This presentation may contain product features that are currently under development. • This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. • Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Technical feasibility and market demand will affect final delivery. • Pricing and packaging for any new technologies or features discussed or presented have not been determined. Disclaimer CONFIDENTIAL 2
- 3. Insight into the World of Logs With Log Insight
- 4. Keep in Touch! Iwan ‘e1’ Rahabok virtual-red-dot.info @e1_ang Linkedin.com/in/e1ang Karl Fultz www.virtuallyanything.us @kwfultz linkedin.com/in/kfultz Manny Sidhu virtual10.com @mannySidhu2 Linkedin.com/in/mannysidhu10
- 5. Hybrid Cloud (Private / Public)Physical Infrastructure SOFTWARE-DEFINED DATA CENTER Compute Network Storage End-User Computing Extensibility Applications Cloud Management Platform (CMP) Virtualized Infrastructure Introduction: Environment Landscape VMware Logs OS and App Logs 200 ESXi Host + VMs = 200GB or 2B log events per day Physical Infrastructure Logs
- 6. Primary Use Cases Troubleshooting and Root Cause Analysis • Follow the trail from vRealize Operations Manager to logs to get to root cause to an observed problem • Identify the needle in the haystack in real time when troubleshooting a problem Monitoring •Monitor metrics and events (performance & change) that are visible only in logs •Identify problems proactively, ensure SLAs and comply to IT policies Unstructured Data Warehouse • Collect all the data in one place without the need for custom parsing, transformation of data • Get full visibility across all your IT environment from a single place 6
- 7. Log Insight Technical Overview Cloud / Data Center Log Management OS Logs VC Logs App Logs System Stats Security Logs API Syslog Analyze • Can analyze any unstructured time-series data, configuration etc. • Automatically identifies structures in the data then uses machine learning to group data Scale • Central, scale-out store (no-SQL) for all collected logs • Configurable retention and archiving • Maintenance free Best for SDDC • Queries, alerts, fields, charts in the vSphere Content Pack
- 8. Contents • Use Cases – Audit & Compliance & Configuration – Performance and Capacity • Customer Sharing • Log Management Platform CONFIDENTIAL 8
- 9. Audit, Compliance, Configuration • Auditor related queries – Who modified what and when – Who snapshot which VM and when – Who changed VM power status (on/off) and when • License compliance (e.g. Oracle) vCenter Tasks vCenter Events CONFIDENTIAL 9
- 10. CONFIDENTIAL 10
- 11. CONFIDENTIAL 11
- 12. CONFIDENTIAL 12
- 13. CONFIDENTIAL 13
- 14. CONFIDENTIAL 14
- 15. CONFIDENTIAL 15
- 16. CONFIDENTIAL 16
- 17. CONFIDENTIAL 17
- 18. CONFIDENTIAL 18 vCenter Tasks vCenter Events
- 19. CONFIDENTIAL 19
- 20. CONFIDENTIAL 20
- 21. CONFIDENTIAL 21
- 22. CONFIDENTIAL 22
- 23. CONFIDENTIAL 23
- 24. Who Snapshot What VM? • vCenter tracks the data 26CONFIDENTIAL
- 25. 27 OOTB dashboard. Grouped by VM Name and Snapshot Operations Type CONFIDENTIAL
- 26. You can know what time the snapshot was created or consolidated CONFIDENTIAL 28
- 27. Who did the snapshot? 29 A Jedi did CONFIDENTIAL
- 28. Example from Production Environment CONFIDENTIAL 30
- 29. http://virtual-red-dot.info/monitoring-changes-to-vmware-vsphere-template/ vCenter Template • Who changed what template and when? • Who converted which VM into template? Vice versa? CONFIDENTIAL 31
- 30. 32 Where have what VMs run on? CONFIDENTIAL
- 31. 33 Tracking that Oracle VM. We which ESXi Host it was running on in any day CONFIDENTIAL
- 32. Performance and Capacity • Which VM hit high CPU Usage and when? • Detailed storage latency from vmkernel • Is vMotion impacting performance? • When does what VSAN event happen? • Network device monitoring CONFIDENTIAL 34
- 33. Which VM Hit High CPU Usage and When? CONFIDENTIAL 35
- 34. Which VM Needs More CPU? • This is the badly sized environment • A lot of VM hit high CPU Usage in just a period of 1 week CONFIDENTIAL 36
- 36. Detailed Storage Latency CONFIDENTIAL 38 Zooming into May 17 – 23. We also exclude all the Magnetic Disk. Device ID naa.55* is SSD, while naa.5000* is magnetic. SSD latency is high.
- 37. CONFIDENTIAL 39 Checkpoint Firewall: CPU Temperature
- 39. Business Requirements • Auditing for Privileged User Access Management (PUAM) • Auditing for Change Compliance • Ability to search and export Logs entries (even after vCenter has rolled over historical logs) 41 Technical Requirements • Need to simplify troubleshooting for a large vSphere environment. (This was a major requirement for the Operations Teams).
- 41. Heavily loaded regional Datacenter Log Insight instance 43
- 43. Business Outcomes / Value Achieved so far… • Auditors now have visibility into Privileged User Access Management (PUAM) changes. (Tracks any changes or spikes in activity made by PUAM users) • Auditors now have visibility into Change Compliance events taking place within the vSphere environment • Ability to export logs as csv files has fulfilled one of the specific audit requirements • Reduced work effort required by audit team to sift through logs 45
- 44. Technical Outcomes / Value achieved so far • BEST PART – makes troubleshooting lots easier – Got a Problem host? Log in to Log Insight console, plug in hostname, filter by name (HA event etc), adjust time interval, fix problem - DONE – No need to generate log bundles. VMware Support remote in, take a look at the Log Insight console and away they go! – Content packs (Vblock, SRM, SQL, etc..) – Email alerts – Centralized logging (including potential do to SNMP trap forwarding and integration with vR Ops) • Super slick, very responsive interface
- 45. Lessons Learnt • Best to create a cluster for larger deployments (think HA and load distribution) • Ensure QoS over remote links or keep tabs on utilization somehow • Size them right! • Should have deployed this sooner! CONFIDENTIAL 47 “Adjusted” to present situation
- 47. Log Management Platform (LMP) • A platform for logs and events from all sources – vSphere and beyond • Why? – Your environment is healthy. Sure. What do the logs say? • Benefits – No need to upload log – Assisted Analysis via content pack – Help in mastering the products you’re managing – Long term archival – Portal for specific roles – All the use cases we’ve covered. 49CONFIDENTIAL
- 48. Properties of Enterprise LMP • Remote office handling • Easy to create dashboard and use • Scalability • Rich OOTB content pack • HA • DR – Active/Active instances at the application layer. This saves DR testing as it becomes irrelevant. • Predictable cost – Logs can go out of hand, as apps can generate excessive logs when hitting errors/bugs CONFIDENTIAL 50
- 49. CONFIDENTIAL Active Active LMP • Distributed • Scalable – Log beyond vSphere • Special users – Auditors – Security Team 51
- 50. CONFIDENTIAL 52
- 51. Auditor Log Insight • The disk consumption is very low • This is showing that it only has 4m events in the past 22 days • We can keep the data on-line for years CONFIDENTIAL 53
- 52. Distributed or Centralized • Not something we change half way. Decide carefully. • They are the opposite. The benefits of Distributed is the disadvantage of Centralized. Vice versa. • Benefit of Distributed – A lot more bandwidth friendly – Smaller VM • Less performance impact on other VMs in the same cluster. Without scale-out architecture, the VM can be 8-16 vCPU VM • Easier to back up and restore – Less risky. A failure in 1 instance does not render all unavailable. • Disadvantage of Distributed – More Log Insight instance to manage (deploy, secure, update, upgrade) – More storage and resource required CONFIDENTIAL 54
- 53. Remote (ROBO) Office • Syslog is chewing up WAN link • Log is lost if WAN Link is down • Syslog is not encrypted • Need to tag logs so we know which DC it comes from CONFIDENTIAL 55
- 54. Operations: Support (GSS) • Speed up resolution of SR – VMware BCS/MCS can simply webex and analyse the logs at Log Insight – Analysis is sped up as Log Insight makes query, charting, etc. easier • Higher chance that the log is not rotated • Encourage self-service – Since the webex session is done together with customer engineer, this encourages knowledge transfer during the joint analysis. This speeds up future analysis and troubleshooting as customer engineer can setup alert. • Caveat – GSS still requires logs to be uploaded, especially for non MCS/BCS customer CONFIDENTIAL 56
- 55. Learn More Try the Hands-on Lab. Nothing to download! 57 Visit the website for resources, 60-day free trial, evaluation guide, and purchasing information. @VMLogInsight www.vmware.com/products/vrealize-log-insight vmware.com/go/vRealize-Ops-Insight-HOL loginsight.vmware.com/ Website: Hands-on Lab 1701, 1710 : Log Insight Community:
- 58. Insight into the World of Logs with VMware vRealize Log Insight Iwan Rahabok, VMware Karl Fultz, VMware MGT7685 #MGT7685
Editor's Notes
- More materials and steps by steps guides are available at our blogs.
- It can digest any type of log data. The users don’t need to think about it, they can just send their data to Log Insight. Log Insight automatically identified structures in the data and create a high performance index for performing analytics Unlike databases there is no need to engage db admins to Extract Transform and Load (ETL) the data. Just send it over It scales up and down very well. It can ingest TBs of data per node per day. It is much faster and much more efficient than the competition It has a configurable retention policy, i.e. you attach 500GB storage and if it runs out of space it rotates the old data out. Optionally they can be written to an archive, making it virtually maintenance free It ships with out of the box knowledge of vSphere logs in the form of a Content Pack, a collection of queries, alerts, dashboards and fields. We are also working with partners to include more Content Packs (e.g. for storage, applications, network devices) as well as more Vmware producst (Nicira/Networking, View, etc.)
- A regional Bank performed an internal audit on its VMware environment. One area of improvement is the speed to produce report on “Who does what action to what object when”. Object here can mean any vSphere objects, such as VM, ESXi Host, Cluster and vCenter. The action here includes commands issued at the ESXi Console via SSH session. Audit data to be stored for >1 year. Reason is compliance with regulation. A regional bank engineering evaluated VMware Log Insight 2.5. While there are 1-2 minor limitations, the product helps the bank in audit compliance. Tips: do not get bogged down on the minor limitations. 80/20 can be good enough
- We can go back in time for months. This is showing almost 2 months of data. To see all the vCenter Tasks, all we need is to select a built-in variable called VC Task Type. A log entry that has this field exist will appear. We are showing the data by day. You will see later we can change this to second when we zoom. Other than the chart, you can also see it in table format.
- As there are a lot of results, let’s see if there are 1-2 tasks that are dominating the result. We can group the result by Task Type. To make it easier to see, I’ve removed the time line. As we can see here, there is 1 tasks that happens >20000x! That’s worth investigating on why it’s happening so often. As the topic today is for audit, let’s remove the top 3 tasks as they are not relevant.
- We’ve removed the top 3 tasks. We simply say the task does not contain these 3. As you can see, the bar chart is easier to see now. The scale has changed from 20K to 200. Since the use case is audit, let’s drill down to the task “Reconfigure Virtual Machine”. We want to know who makes the changes.
- Drilling down to that single task. As you can see, it’s all just that 1 task. Since it’s only 1 task, we are showing the time line again so we can see when the changes are made. Let’s now group the event by VM, so we can see the VM names.
- Same chart, but showing the break down of the VMs. We can see the list of VMs in the legend. We can see that 1 VM has a lot of changes. This is a regular changes, expected by VDPA product, so I’m gonna exclude it. The changes were all made by its service account. So let’s exclude that account.
- Excluding the service account. All we have to do is say VC User Name not matching that user name. There is a lot less data now. The bar chart now showing maximum 12 changes on any given day, less than half of the 30 we saw earlier. The bar chart is also easier to see. Let’s drill down further to a single VM. I’ll take one of these VRNI VM.
- Drilling down to a single VM. We can see it’s all done by my brother Hicham Mourad, architect from our technical marketing team. We know what date he changes. Let’s see what time.
- Drilling down the time. The scale is now down to 1 hour. We can see he made 3 changes only 9 July at 3 different hours, but 3 changes within the hour on 13 July.
- We are interested in events impacting our consumer (VMs), so let’s filter it out.
- What does this screen tell us?
- Now… can you guess the snapshot name? It’s in the log above. Hints: I was singing an old songby The Beatles. Ok, it wasn’t technically singing, it was a bad attempt at singing
- http://virtual-red-dot.info/what-are-the-esxi-hosts-that-a-vm-has-ever-run-on/
- Storage latency at vmkernel level by Device There is an outlier at 6000 ms on a magnetic disk.
- In this example, we should check the latency at the Magnetic Disk on 22 - 23 May. However, due to time limit in this presentation, we will move on.
- This is our present architecture. It has evolved throughout the year.
- What I faced back in May 2015 may not be applicable. Certain products limitations have been addressed. So I am sharing the relevant ones. I’m also adding things that I think you might encounter, that luckily I did not have to.
- Most VMware admin do not even look at the logs. Most do not even understand the various logs well.
- Issue: Sending syslog over the WAN can consume a large amount of bandwidth. A bug in product could trigger excessive logs. If the link is down or busy, logs are lost. There is no buffering at the sender site (e.g. ESXi does not buffer) Possible solution A Log Insight server (or cluster thereof) can receive logs and then forward those logs on to another Log Insight server (or cluster thereof) via CFAPI. The chart below compares native syslog vs Log Insight Ingestion API. It shows around 30 Mb vs 1 Mb due to compression Arcsight is using the syslog protocol Reference: http://sflanders.net/2015/03/04/12-reasons-why-you-should-use-the-log-insight-forwarder Ingestion API (default protocol) should be used if since the remote destination is another Log Insight instance Forwarder adds complexity Need to be setup & managed. No license required. Forwarders using the ingestion API support sending events via encrypted channel (SSL). Forwarders minimize the attack vector of the primary Log Insight instance / remote syslog destination. In short, if the only devices allowed to send events to the primary Log Insight are the Forwarders then security becomes easier. It means simpler inter-segment routing/firewalling. You aggregate logs within local LI forwarder & allow it to connect to the main LI server via single connection and single firewall rule (else think about 100s of devices connecting to the remote LI server) This is a compelling reason to deploy forwarders in every datacenter. In a regional bank, this is also a reason why we choose Forwarder. The architecture diagram distinguish between secured and unsecured channel Forwarders are complete Log Insight instances and offer the same options as regular Log Insight instances. vSphere integration can be configured on forwarders keeping traffic local to vSphere / forwarder. Forwarders using the ingestion API can add metadata to events add additional information to an event to make it easier to query for or correlate over. While many syslog agents provide a solution to this problem (see this post for more information), they do so by manipulating the actual event instead of providing supplemental or metadata information. The Log Insight forwarder offers a tags option and supports JSON key/value pairs for specifying tags. This makes it possible to add information such as which datacenter the event came from. Forwarders support filtering of events: Filtering makes it possible to ensure “noise” is not ingested by the central Log Insight instance. However, need to define them manually one by one. Forwarders using the ingestion API can respond to throttle requests (i.e. back pressure) and can ensure no message loss during intermittent connection issues back pressure is handled by sending a 503 HTTP status code while connection issues can be detected and prevented against through REST calls. The Log Insight forwarder provides a configurable disk-backed queue used to save events until the server has acknowledged ingestion. This disk-backed queue prevents drops of forwarded events during intermittent connection issues where syslog events already sent, but not acknowledged by the server, can be lost after the TCP ACK window expires. Free versions of syslog agents like Rsyslog and Syslog-NG do not provide back pressure functionality nor do they provide a disk-based queue (some do provide a configurable in-memory queue). In addition, these agents rely only on the retry abilities of the TCP protocol (e.g. TCP ACK window) if the TCP protocol is being used to send the traffic–none of this is available for UDP. Forwarders can server as a backup: events that are forwarded are also stored on the forwarder. In addition, forwarders are complete Log Insight instances and offer the same UI as regular Log Insight instances. This means if the central Log Insight instance becomes temporarily unavailable the events still exist on the forwarder, users can be given direct access to forwarders, alerts can be configured on forwarders, etc. The forwarder local capacity can be configured to store events for as long as desired. We can keep the data small, as we are not going to query the Forwarder. 1 day of data is sufficient. This keeps them light Forwarders make it easier to support disaster recovery (DR): The forwarder is a Log Insight instance; it contains the events it forwarded and supports querying. In addition, forwarders can send events to up to 10 different destinations. This make it easy to forward events to 2 different remote destinations (e.g. Log Insight instances) to handle complete DR. Forwarders make it possible to support variable retention periods: Since forwarded events can be filtered, it is possible to send security events to one Log Insight instance where events are kept for 90 days on the remote Log Insight instance and all other events to a separate Log Insight instance where events are kept for 30 days. In ank, we will take advantage of this. 1 central instance for Security purpose (longer retention), and 2 other instance for general purpose (60 days)