0
Practical Tips for Novell ®  Cluster Services Mark Robinson CTO Linux, NDS8 [email_address] Tim Heywood CTO, NDS8 [email_a...
Agenda <ul><li>Introduction
Cluster Services in OES2
Our Environment
AutoYaST
Cluster Build Methodology
Creating Resources
Cluster Management
Troubleshooting </li></ul>
Introduction
Introduction <ul><li>Mark Robinson </li><ul><li>Linux Geek
Working with SUSE ®  since 1998
Working with OES since OES1 Beta 5
CLP, CLE, NCE ES, CNI, etc.
Ex-SysOp </li></ul><li>Tim Heywood </li><ul><li>Working with Novell ®  since ????
Working with OES since OES1 Beta 5
CNE, MCNE, CNI(ish)
Novell Knowledge Partner (SysOp) </li></ul></ul>
Introduction <ul><li>NDS8 Network Design and Support Ltd. </li><ul><li>Platinum Consulting Partner
Based in Edinburgh, work worldwide
Specialities: </li><ul><li>Linux
Workgroup
SRM </li></ul></ul></ul>
Cluster Services in OES2
Cluster Services in OES2 <ul><li>New features are Linux only
New from OES2 FCS on: </li><ul><li>Resource monitoring
XEN virtualization support
x86_64 platform support </li><ul><li>Including mixed 32/64 bit node support </li></ul><li>Dynamic Storage Technology </li>...
What's new in SP1/2? <ul><li>Major rewrite of cluster code for SP2 </li><ul><li>Removed NetWare ®  translation layer
Much faster
Much lower system load
Typical load average of 0.2! </li></ul><li>New/improved clustering for: </li><ul><li>iFolder 3
AFP
…
… </li></ul><li>NCP virtual server for POSIX filesystem resources </li></ul>
Types of Clusters <ul><li>Traditional cluster </li><ul><li>Servers (nodes)
Resources </li><ul><li>NSS
GroupWise ®
iPrint </li></ul></ul><li>XEN cluster </li><ul><li>Dom0 hosts (nodes)
XEN guests (DomU) resources
Each resource is a server in its own right
Live migration with para-virtualised DomU </li></ul></ul>
XEN Cluster Architecture OCFS2 LUN DomU Files Cluster Node Xen Dom0 Cluster Node Xen Dom0 Cluster Node Xen Dom0 Resource D...
Our Environment
Our Environment <ul><li>VMware Workstation based
VMware shared disk as an alternative to iSCSI
Virtual Machines </li><ul><li>Resource Server
Node 1 (built, in the cluster)
Node 2 (to be joined to cluster)
Node 3 (to be built) </li></ul><li>SUSE ®  Linux Enterprise Server 10 SP3/OES2 SP2 combined iso </li></ul>
VMware Setup <ul><li>Create disks standalone
Add config to node vmx files </li></ul>disk.locking = &quot;false&quot; diskLib.dataCacheMaxSize = &quot;0&quot; scsi1.pre...
Upcoming SlideShare
Loading in...5
×

Practical Tips for Novell Cluster Services

6,217

Published on

This session will use Novell Open Enterprise Server 2 SP2 to demonstrate how to cluster critical services—from NSS and Novell iPrint to Novell GroupWise, AFP and beyond. We'll cover the new features of Novell Cluster Services in the latest release of Novell Open Enterprise Server, and we'll show you how you can ensure consistency by using AutoYaST to build your nodes. This will be a practical session, so be prepared for a few thrills and spills along the way!

Speakers:
Tim Heywood CTO NDS 8
Mark Robinson CTO Linux NDS8

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,217
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
260
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Explain about building combined iso
  • Discuss flags for vmware-vdiskmanager – especially -t Disk types: 0 : single growable virtual disk 1 : growable virtual disk split in 2GB files 2 : preallocated virtual disk 3 : preallocated virtual disk split in 2GB files 4 : preallocated ESX-type virtual disk 5 : compressed disk optimized for streaming Discuss path to SAN virtual disks Mention different VMware versions
  • Tell audience that the autoyast build may take a while – we&apos;ll concentrate on existing two nodes for most of the demo.
  • Offer copy of autoyast profiles used in demos.
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Talk about creating the two pools and why. DEMO: Create pool1/vol1 Create pool1_shd/vol1_shd
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Script logs are rolling logs now – show all operations on this node for a particular resource
  • Transcript of "Practical Tips for Novell Cluster Services"

    1. 1. Practical Tips for Novell ® Cluster Services Mark Robinson CTO Linux, NDS8 [email_address] Tim Heywood CTO, NDS8 [email_address]
    2. 2. Agenda <ul><li>Introduction
    3. 3. Cluster Services in OES2
    4. 4. Our Environment
    5. 5. AutoYaST
    6. 6. Cluster Build Methodology
    7. 7. Creating Resources
    8. 8. Cluster Management
    9. 9. Troubleshooting </li></ul>
    10. 10. Introduction
    11. 11. Introduction <ul><li>Mark Robinson </li><ul><li>Linux Geek
    12. 12. Working with SUSE ® since 1998
    13. 13. Working with OES since OES1 Beta 5
    14. 14. CLP, CLE, NCE ES, CNI, etc.
    15. 15. Ex-SysOp </li></ul><li>Tim Heywood </li><ul><li>Working with Novell ® since ????
    16. 16. Working with OES since OES1 Beta 5
    17. 17. CNE, MCNE, CNI(ish)
    18. 18. Novell Knowledge Partner (SysOp) </li></ul></ul>
    19. 19. Introduction <ul><li>NDS8 Network Design and Support Ltd. </li><ul><li>Platinum Consulting Partner
    20. 20. Based in Edinburgh, work worldwide
    21. 21. Specialities: </li><ul><li>Linux
    22. 22. Workgroup
    23. 23. SRM </li></ul></ul></ul>
    24. 24. Cluster Services in OES2
    25. 25. Cluster Services in OES2 <ul><li>New features are Linux only
    26. 26. New from OES2 FCS on: </li><ul><li>Resource monitoring
    27. 27. XEN virtualization support
    28. 28. x86_64 platform support </li><ul><li>Including mixed 32/64 bit node support </li></ul><li>Dynamic Storage Technology </li></ul></ul>
    29. 29. What's new in SP1/2? <ul><li>Major rewrite of cluster code for SP2 </li><ul><li>Removed NetWare ® translation layer
    30. 30. Much faster
    31. 31. Much lower system load
    32. 32. Typical load average of 0.2! </li></ul><li>New/improved clustering for: </li><ul><li>iFolder 3
    33. 33. AFP
    34. 34.
    35. 35. … </li></ul><li>NCP virtual server for POSIX filesystem resources </li></ul>
    36. 36. Types of Clusters <ul><li>Traditional cluster </li><ul><li>Servers (nodes)
    37. 37. Resources </li><ul><li>NSS
    38. 38. GroupWise ®
    39. 39. iPrint </li></ul></ul><li>XEN cluster </li><ul><li>Dom0 hosts (nodes)
    40. 40. XEN guests (DomU) resources
    41. 41. Each resource is a server in its own right
    42. 42. Live migration with para-virtualised DomU </li></ul></ul>
    43. 43. XEN Cluster Architecture OCFS2 LUN DomU Files Cluster Node Xen Dom0 Cluster Node Xen Dom0 Cluster Node Xen Dom0 Resource DomU Linux iPrint Resource DomU Linux iPrint Resource DomU Linux iFolder Resource DomU Linux GroupWise Resource DomU NetWare pCounter Live Migrate Live Migrate
    44. 44. Our Environment
    45. 45. Our Environment <ul><li>VMware Workstation based
    46. 46. VMware shared disk as an alternative to iSCSI
    47. 47. Virtual Machines </li><ul><li>Resource Server
    48. 48. Node 1 (built, in the cluster)
    49. 49. Node 2 (to be joined to cluster)
    50. 50. Node 3 (to be built) </li></ul><li>SUSE ® Linux Enterprise Server 10 SP3/OES2 SP2 combined iso </li></ul>
    51. 51. VMware Setup <ul><li>Create disks standalone
    52. 52. Add config to node vmx files </li></ul>disk.locking = &quot;false&quot; diskLib.dataCacheMaxSize = &quot;0&quot; scsi1.present = &quot;TRUE&quot; scsi1.sharedBus = &quot;none&quot; scsi1.virtualDev = &quot;lsilogic&quot; scsi1.pciSlotNumber = &quot;35&quot; scsi1:0.present = &quot;TRUE&quot; scsi1:0.fileName = &quot;cluster-lun0.vmdk&quot; scsi1:0.mode = &quot;independent-persistent&quot; scsi1:0.redo = &quot;&quot; scsi1:1.present = &quot;TRUE&quot; scsi1:1.fileName = &quot;cluster-lun1.vmdk&quot; scsi1:1.mode = &quot;independent-persistent&quot; scsi1:1.redo = &quot;&quot; vmware-vdiskmanager -c -s 100mb -a lsilogic -t 2 cluster-lun0.vmdk vmware-vdiskmanager -c -s 1gb -a lsilogic -t 2 cluster-lun1.vmdk
    53. 53. Our Environment Resource Server (Tree master, iManager, Installation services, AutoYaST, SMT) Storage VMware Shared Disks OES2 SP2 Nodes
    54. 54. Our Environment AutoYaST Third Node Resource Server (Tree master, iManager, Installation services, AutoYaST, SMT) Storage VMware Shared Disks OES2 SP2 Nodes
    55. 55. AutoYaST
    56. 56. Why AutoYaST? <ul><li>Repeatable (exactly) </li><ul><li>No “human element” </li></ul><li>XML forms part of Documentation
    57. 57. Drink coffee (or suitable non-caffeinated beverage) while server builds itself!
    58. 58. Multiple simultaneous builds </li><ul><li>Stagger by at least 15 minutes </li></ul><li>Easy to expand cluster with new nodes
    59. 59. Helps with DR </li></ul>
    60. 60. Why AutoYaST? <ul><li>What will AutoYaST do? </li><ul><li>Disk partitioning
    61. 61. Software patterns
    62. 62. Network configuration (including VLAN, bonding etc)
    63. 63. OES services </li><ul><li>eDirectory ™ – new or existing tree
    64. 64. NSS
    65. 65. NCS </li></ul><li>Security lockdown
    66. 66. Scripts/Complete config files to do the rest
    67. 67. At the end of the install we will migrate an NSS resource to the new node with no additional configuration! </li></ul></ul>
    68. 68. AutoYaST – New Cluster <ncs> <admin_context> cn=admin.o=novell </admin_context> <admin_password> novell </admin_password> <cluster_dn> cn=cluster,ou=resources,o=novell </cluster_dn> <cluster_ip> 10.0.0.100 </cluster_ip> <config_type> New Cluster </config_type> <ldap_ip_address> node IP,LDAP server IP </ldap_ip_address> <ldap_secure_port config:type=&quot;integer&quot;> 636 </ldap_secure_port> <server_name> nodename </server_name> <start> Later </start> <sbd_dev> sdx </sbd_dev> <sbd_dev2> sdy </sbd_dev2> </ncs>
    69. 69. AutoYaST – Existing Cluster <ncs> <admin_context> cn=admin.o=novell </admin_context> <admin_password> novell </admin_password> <cluster_dn> cn=cluster,ou=resources,o=novell </cluster_dn> <cluster_ip></cluster_ip> <config_type> Existing Cluster </config_type> <ldap_ip_address> node IP,LDAP server IP </ldap_ip_address> <ldap_secure_port config:type=&quot;integer&quot;> 636 </ldap_secure_port> <server_name> nodename </server_name> <start> Later </start> <sbd_dev></sbd_dev> <sbd_dev2></sbd_dev2> </ncs>
    70. 70. Demo AutoYaST
    71. 71. Cluster Build Methodology
    72. 72. Cluster Build Methodology <ul><li>Start with a Resource Server </li><ul><li>iManager
    73. 73. Network Installation Server – HTTP or NFS
    74. 74. AutoYaST repository (can be password protected on HTTP)
    75. 75. SMT for patching
    76. 76. Magic PiXiEs server </li></ul></ul>
    77. 77. Cluster Build Methodology <ul><li>Create a “template node” </li><ul><li>NodeZ
    78. 78. Use it to </li><ul><li>Create AutoYaST template using “Clone this system...”
    79. 79. Test the shared disk
    80. 80. Create the cluster
    81. 81. Create the SBD
    82. 82. Creating resources </li></ul><li>This build will NOT be part of final production cluster </li></ul></ul>
    83. 83. Cluster Build Methodology <ul><li>Copy this XML for additional nodes </li><ul><li>Modify </li><ul><li>server name
    84. 84. IP address(es) </li></ul><li>Use diffuse to compare XML files </li></ul><li>Build the other nodes </li><ul><li>Use the XML created above
    85. 85. If not 100% right, whack it, modify XML and start build again
    86. 86. Remember these are now commodity items </li></ul><li>Whack NodeZ and rebuild to complete the system </li></ul>
    87. 87. Cluster Build Methodology <ul><li>Implement NIC bonding </li><ul><li>NIC driver independent
    88. 88. 7 different methods – some require switch support
    89. 89. Link state vs arp monitoring – blades often cannot lose local link!
    90. 90. Configurable with AutoYaST </li></ul><li>Implement Multipath (MPIO) </li><ul><li>Very simple to configure – mainly autodetect
    91. 91. Wide range of SAN support
    92. 92. Friendly LUN naming
    93. 93. Configuration file can be used (put in place with autoYaST) </li></ul></ul>
    94. 94. Creating Resources
    95. 95. File Sharing Resources <ul><li>An NSS pool </li><ul><li>Use iManager
    96. 96. Will end up as Primary for DST pair </li></ul><li>Another NSS pool </li><ul><li>Use NSSMU (just because we can)
    97. 97. Will end up as Shadow for DST pair </li></ul><li>Combine them into one resource </li><ul><li>Delete resource for shadow
    98. 98. Modify load script for primary </li></ul></ul>
    99. 99. File Sharing Resources <ul><li>POSIX filesystem based resource with NCP </li><ul><li>Easier than Samba to access files
    100. 100. Can be used for iPrint, DHCP etc
    101. 101. Use evmsgui to create and format the volume
    102. 102. Create the resource in iManager
    103. 103. Script to create NCP virtual server </li></ul></ul>
    104. 104. File Sharing Resources <ul><li>Add resource monitoring
    105. 105. Add NFS access </li><ul><li>LUM enablement of target users
    106. 106. NSS/POSIX rights
    107. 107. exportfs in load script rather then /etc/exports on nodes
    108. 108. Use fsid=x for NSS </li></ul></ul>
    109. 109. NFS access SHARED1 Virtual Server SHARED1 Volume NFSaccess Iface UID: 1012 Mis-dweeb UID: 1004 LUM NSS Rights Dweeb Gromit Wallace NFS FPC1 FPC2 FPC3 FPC4 FPC5 fpc.server.novell Mis UID: 1010 Oracle UID: 60003 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012 Mis UID: 1010 Oracle UID: 60003 Iface UID: 1012
    110. 110. iPrint <ul><li>Create iPrint on NSS
    111. 111. Run iprint_nss_relocate on each node with volume in place
    112. 112. NB: only one iPrint resource may run on a node
    113. 113. Need to accept certificates in iManager for each node </li></ul>
    114. 114. iFolder <ul><li>Create iFolder on POSIX </li><ul><li>/mnt/cluster/ifolder </li></ul><li>Run /opt/novell/ifolder3/bin/ifolder_cluster_setup on each node </li><ul><li>Copy /etc/sysconfig/novell/ifldr3_2_sp2 to nodes first </li></ul><li>NB: Only one iFolder resource may run on a node </li></ul>
    115. 115. DNS <ul><li>DNS must be on NSS as NCP server required for eDirectory ™ integration
    116. 116. Check NCP:NCPServer objects
    117. 117. LUM user required for NSS rights </li></ul>
    118. 118. DHCP <ul><li>Create DHCP on NSS
    119. 119. Leases file on NSS volume
    120. 120. Log file on NSS volume </li><ul><li>Syslog-ng configuration
    121. 121. Logrotate configuration
    122. 122. Default AppArmor configuration will not allow logging to here! </li></ul></ul>
    123. 123. GroupWise <ul><li>Create PO on NSS
    124. 124. Set namespace in load script </li><ul><li>/opt=ns=long </li></ul><li>Disable atime/diratime on volume </li><ul><li>Open nsscon
    125. 125. Run /noatime=volname </li></ul></ul>
    126. 126. OCFS2 Shared Storage <ul><li>Shared disk! Multi-mount, read/write with distributed lock management
    127. 127. /etc/ocfs2/cluster.conf automagically created by NCS
    128. 128. Fstab mounting uses /etc/init.d/ocfs2 service </li></ul>
    129. 129. Cluster Management
    130. 130. Cluster Management <ul><li>iManager </li><ul><li>The ONLY way to create/delete/edit resources
    131. 131. View event log </li></ul><li>Cluster command </li><ul><li>Same as NetWare ®
    132. 132. No cvb rebuild or device scan. These are not required on Linux as EVMS does it for you </li></ul><li>Console One </li></ul>
    133. 133. The cluster Command <ul><li>The usual suspects </li><ul><li>cluster online/offline/migrate
    134. 134. cluster join/leave
    135. 135. cluster status/resources/view/info </li></ul><li>More interesting </li><ul><li>cluster stats display – check heartbeat/SBD ticks
    136. 136. cluster pools – check NSS pools and location
    137. 137. cluster set – modify heartbeat etc
    138. 138. cluster exec – potentially very dangerous </li></ul><li>Lots of BCC commands </li></ul>
    139. 139. Troubleshooting
    140. 140. Useful Linux Tools <ul><li>ip command – manage TCP/IP on Linux </li><ul><li>ip addess show/add/del
    141. 141. ip route show </li></ul><li>ethtool – NIC settings
    142. 142. cat /proc/net/bonding/bondX
    143. 143. netstat – network communication status </li><ul><li>Check which ports services are listening on
    144. 144. Check IP based connections to node </li></ul><li>nmap – network/port scanner
    145. 145. multipath -ll </li></ul>
    146. 146. LUN Identification <ul><li>Which LUN is which?
    147. 147. lsscsi – shows LUN ID numbers
    148. 148. ls -l /dev/disk/by-id </li><ul><li>scsi-360a98000 4334616f6b5a55572d625550
    149. 149. Need to find ID on SAN. (Netapp uses ASCII!) </li></ul><li>Multipathing will show the ID as the multipath name </li><ul><li>Use friendly naming </li></ul></ul>
    150. 150. Useful Tools <ul><li>sbdutil – create/check/modify the SBD </li><ul><li>sbdutil -f to find the SBD
    151. 151. sbdutil -v to view the current state of the SBD </li></ul><li>/opt/novell/ncs/bin/ncs-configd.py </li><ul><li>-init option to pull down load scripts, fix node names etc </li></ul><li>cifsPool.py to fix CIFS attributes (TID #7005192)
    152. 152. OES2 NCS Master Reference TID, FAQ and Troubleshooting – TID #7001433
    153. 153. NSA – Novell ® Support Advisor </li><ul><li>Many patterns for NCS </li></ul></ul>
    154. 154. File Locations <ul><li>Cluster configuration file </li><ul><li>/etc/opt/novell/ncs/clstrlib.conf </li></ul><li>Load/Unload scripts </li><ul><li>/var/run/ncs (run from here)
    155. 155. /var/opt/novell/ncs </li></ul><li>Load script output logs </li><ul><li>/var/opt/novell/log/ncs </li></ul><li>System Log </li><ul><li>/var/log/messages </li></ul></ul>
    156. 156. File Locations <ul><li>Admin filesystem – virtual filesystem for NCS management </li><ul><li>/admin/Novell/Cluster </li></ul><li>Proc filesystem – virtual filesystem for Linux/NCS management </li><ul><li>/proc/ncs </li></ul><li>Cluster event log </li><ul><li>iManager
    157. 157. /admin/Novell/Cluster/EventLog.xml </li></ul></ul>
    158. 158. /proc/ncs Magic <ul><li>Enable serious debugging! </li><ul><li>echo -n &quot;TRACE ON&quot; > /proc/ncs/vll
    159. 159. echo -n &quot;TRACE SBD ON&quot; > /proc/ncs/vll
    160. 160. echo -n &quot;TRACE GIPC ON&quot; > /proc/ncs/vll
    161. 161. echo -n &quot;TRACE MCAST ON&quot; > /proc/ncs/vll
    162. 162. echo -n &quot;TRACE CVB ON&quot; > /proc/ncs/cluster
    163. 163.
    164. 164. Can be made permanent by editing /opt/novell/ncs/bin/ldncs </li></ul><li>Find the SBD </li><ul><li>cat /proc/ncs/sbdlib </li></ul></ul>
    165. 165. AdminFS Magic <ul><li>Two type of file in /admin/Novell/Cluster </li><ul><li>*.xml – contain cluster/state information
    166. 166. *.cmd – “write then read” files for issuing cluster commands </li></ul></ul>
    167. 167. Known issues <ul><li>EVMS issue with no NSS </li><ul><li>Unpatched SP2 nodes without NSS cannot load SBD kernel moduled </li></ul><li>iFolder shutdown script </li><ul><li>Doesn't shut down components if names have been changed during configuration </li></ul><li>IP address problem </li><ul><li>Unpatched nodes can allow duplicate IP addresses on network </li></ul><li>Resources in NDS sync state </li><ul><li>Check replica rings/referrals
    168. 168. Check case of cluster DN in clstrlib.conf </li></ul></ul>
    169. 169. www.nds8.co.uk
    170. 171. Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×