Updated version of my talk about Hadoop 3.0 with the newest community updates.
Talk given at the codecentric Meetup Berlin on 31.08.2017 and on Data2Day Meetup on 28.09.2017 in Heidelberg.
Hadoop World 2011: HDFS Federation - Suresh Srinivas, HortonworksCloudera, Inc.
Scalability of the NameNode has been a key issue for HDFS clusters. Because the entire file system metadata is stored in memory on a single NameNode, and all metadata operations are processed on this single system, the NameNode both limits the growth in size of the cluster and makes the NameService a bottleneck for the MapReduce framework as demand increases. This presentation will describe the features and implementation of HDFS Federation scheduled for release with Hadoop-0.23.
Updated version of my talk about Hadoop 3.0 with the newest community updates.
Talk given at the codecentric Meetup Berlin on 31.08.2017 and on Data2Day Meetup on 28.09.2017 in Heidelberg.
Hadoop World 2011: HDFS Federation - Suresh Srinivas, HortonworksCloudera, Inc.
Scalability of the NameNode has been a key issue for HDFS clusters. Because the entire file system metadata is stored in memory on a single NameNode, and all metadata operations are processed on this single system, the NameNode both limits the growth in size of the cluster and makes the NameService a bottleneck for the MapReduce framework as demand increases. This presentation will describe the features and implementation of HDFS Federation scheduled for release with Hadoop-0.23.
This exam measures your ability to accomplish the technical tasks listed below. The percentages indicate the relative weight of each major topic area on the exam. http://www.allpass4sure.com/microsoft-pdf-70-410.html
NT Domain Restructuring and Exchange Resource Forests
1. NT Domain Restructuring and Exchange Resource Forests Presented By; John Daugherty August 3, 2005
2. NT Domain Restructuring and Exchange Resource Forests About the Speaker John Daugherty Senior Consultant, PCMS Datafit – IT Advisor Group NT4, 2000, and 2003 MCSE / MCSA / CCA 12 Years in IT, dedicated to networking Performed dozens of NT to AD migrations/restructures PCMS Datafit – IT Advisor Group Microsoft Central Region VAR Partner of the Year 12 senior networking consultants Microsoft infrastructure solutions – AD, SMS, MOM, SharePoint Cisco, Symantec, and Citrix Partner Microsoft Gold Partner
3. NT Domain Restructuring and Exchange Resource Forests Topics Restructure versus Upgrade Why Restructure? 10-Steps to Restructure, Resource Forest, and Relaxation Summary Questions?
4. NT Domain Restructuring and Exchange Resource Forests Restructure Versus Upgrade Upgrade retains network structure Upgrade retains domain name Upgraded domain members need little attention
5. NT Domain Restructuring and Exchange Resource Forests Upgrade Versus Restructure Restructure is starting over from scratch Restructure can mean combining multiple NT 4 Domains into single AD Domain Restructure can mean moving a single NT 4 Domain into multiple AD Domains Restructuring is typically more complex during migration Restructuring is typically less complex, once migrated
6. NT Domain Restructuring and Exchange Resource Forests Why Restructure? Have too many Domains today Less administration in a single Domain/Forest vs. multiple NT4 Domain has become unreliable Bolt-on acquisitions – already have AD Already have AD and NT4 Domain for whatever reason
7.
8. NT Domain Restructuring and Exchange Resource Forests Step 1 – Plan, Plan, and oh yeah… PLAN! Plan migration steps – cookbook Test each step of the plan Use VMWARE or MS Virtual PC Create new BDC’s in current NT4 Domain, move to lab, promote to PDC Involve all parties in planning Don’t forget home-grown apps
10. NT Domain Restructuring and Exchange Resource Forests Step 2 – Create AD Forest structure New or existing forest Windows 2000 or 2003 domain native mode Create OU structure Create GPOs/migrate system policies (don’t forget Citrix) Create name resolution and DHC
11. NT Domain Restructuring and Exchange Resource Forests Step 2 – Create AD Forest structure Create site structure Cost = 1024/log(unused bandwidth in Kbps) Monitor AD health Microsoft, Microsoft Operations Manager Monitor WAN health Packeteer, PacketSeeker SolarWinds, Orion Test name resolution intra- and inter-forest
12. NT Domain Restructuring and Exchange Resource Forests Step 2 – Create AD Forest structure Implement Disaster Recovery Microsoft, NTBackup Veritas, Backup Exec Quest, Recovery Manager for AD Implement Directory Provisioning and Management Microsoft, AD Users and Computers (mmc) SystemTools, Hyena (mmc) Quest, Active Roles Server (web and mmc) Implement change management
13. NT Domain Restructuring and Exchange Resource Forests Step 2 – Create AD Forest structure Create/copy login scripts Consider GPOs Login scripts subdirectories (multiple domains) Create PKI Don’t forget NTP FSMO roles moved Root placeholders a good thing?
15. NT Domain Restructuring and Exchange Resource Forests Step 3 – Create Trust Relationships Mirror trusts from Domain migrated Microsoft, ADMT Quest, Domain Migration Wizard Create two-way external trust between source and target Add Domain Admin account from target to source Administrators Group Verify trusts Turn off SID Filtering
16. NT Domain Restructuring and Exchange Resource Forests SID Filtering Security hole in inter-forest trusts Can add Domain or Enterprise Admin sID to sIDHistory Impersonating an elevated user Nothing you can do in a single forest Must have at least Windows 2000 SP4 on DCs to enable Cannot disable SID Filtering for new W2k SP4 and later trusts Disable using NETDOM.exe /quarantine:No for Pre W2k SP4
17. NT Domain Restructuring and Exchange Resource Forests sIDs, ACLs, and ACEs NT4 Users and Groups = sID sIDs attached as ACEs ACEs are entries in ACLs reACLing – rewriting NT4 sID to AD GUID ACLs point to NT4 sID Many programs do not use sIDs (SQL, SMS)
18. NT Domain Restructuring and Exchange Resource Forests sIDHistory Restructure means new SID for user Window 2000 Native Mode or above is MS-supported Allows migrated accounts access to resources Multi-valued - Security token can hold up to 1023 sIDS Some applications recognize sIDs, but not sIDHistory Some applications recognize sIDHistory, but not multi-valued sIDHistory Some applications recognize multi-valued sIDHistory, but not past 5 or so values
20. NT Domain Restructuring and Exchange Resource Forests Typical Uses of sIDHistory Users migrated, but servers not reACLd Users migrated, but their workstation not migrated – allows user to continue to use their old profile with new permissions (Quest changeprofile) Some domains migrated, NT4 permissions on other domains Unknown applications set up in NT4 Domain
23. NT Domain Restructuring and Exchange Resource Forests Step 4 – Prepare for Restructure Gather information about source and target directory objects SystemTools, Hyena (small and single domain) Microsoft, ADMT (small – large and single domains) Quest, DMW (large and multiple domains) Rename users and groups to not conflict with target users or groups, unless merging Demote those BDC’s UTools, UPromote Quest, DCDemote
24. NT Domain Restructuring and Exchange Resource Forests Step 4 – Prepare for the Restructure Fully back up source and target Resolve Events Delete unused accounts Watch out for VPN users Watch out for service accounts Delete expired accounts Ignore computer objects? Perhaps
25. NT Domain Restructuring and Exchange Resource Forests Step 4 – Prepare for the Restructure Move or establish DNS to AD DNS servers for workstations and servers One last sanity check
26. NT Domain Restructuring and Exchange Resource Forests Step 5 – Migrate Directory Objects Copies NT objects into AD Issue a freeze on the source Merge appropriate groups and users Disable target users Copy passwords from source to target Migrate sIDHistory
27. NT Domain Restructuring and Exchange Resource Forests Step 5 – Migrate Directory Objects Migrate Groups first, given the choice Pick the RID Master FSMO in target if over 500 users Microsoft, ADMT v3 will ( http:// beta.microsoft.com – admt3beta) Quest, DMW can Move along quickly to allowing users to log in Password copies Administrator changes Don’t update user rights if you don’t have to!
29. NT Domain Restructuring and Exchange Resource Forests Step 6 – Migrate Workstations Migration can continue through workstation attrition Least resistance, complexity, and control Trade time and complexity for cost You will keep sIDHistory for quite some time Assumes no workstation domain-credential services Proven on dozens of domain restructures
30. NT Domain Restructuring and Exchange Resource Forests Step 6 – Migrate Workstations Users now exist in source and target with same sID Enable groups of users to log into their workstation Login script runs: UPHCLEAN installed Netdom – moves workstation to new domain Workstation reboots Quest, Changeprofile moves user profile or ADMT (TemplateScript.vbs) sIDHistory gives user access to all applications! User has experienced only one reboot
31. NT Domain Restructuring and Exchange Resource Forests Step 6 – Migrate Workstations Congratulations, your users are on the new domain! Lastly, reACL workstations (can be done later) Microsoft, ADMT Quest, DMW Many other tools can do the job Do not use “Add Mode” if using ADMT – GPO software deployment issues when users are targeted *** This is one of many ways to migrate workstations ***
33. NT Domain Restructuring and Exchange Resource Forests Step 7 – Migrate Servers Move servers to target domain using migration tools Verify users are logging in with target account Can use “Add Mode” until all domains are migrated, then reACL using “Replace Mode.” SIDHistory fine, too. DHCP servers will need to be authorized Don’t move Exchange – MS does not support 5.5 to 2003 upgrade reACL servers last – not Exchange
34. NT Domain Restructuring and Exchange Resource Forests Step 7 – Migrate Servers Move Terminal Server licenses for Windows 2000 or Windows 2003
36. NT Domain Restructuring and Exchange Resource Forests Step 8 – Migrate Exchange Clean up duplicate mailboxes (multiple orgs) Clean up resource mailboxes (conference room) Verify no two mailboxes are owned by same account LDAP Queries using header.exe or VBscript Quest, DMW reACL Information Store, prepare Exchange Account for resource ownership ADC, Set Attribute to NTDSNOMATCH Quest, EMW is automatic – with .dll
37. NT Domain Restructuring and Exchange Resource Forests Step 8 – Migrate Exchange Implement Identity Management – We’ll talk about this in a minute Microsoft, MIIS – Complex, highly scalable CPS Systems, SimpleSync – Simple, highly scalable Greenfield Approach (MS, Migration Wizard)– Choice 1 Uses ADC - Creates disabled mail-enabled users Uses MS, Mailbox Migration Wizard to export mailbox Must use pfmigrate No Inbox rules migrated Need to remove Exchange 5.5 mailbox manually No delegations copied No Calendar to/from migrated mailboxes Can’t reply to old messages from new server Custom recipients need to be recreated DL’s need to be recreated
38. NT Domain Restructuring and Exchange Resource Forests Step 8 – Migrate Exchange Quest Approach – Choice 2 Uses Quest, Exchange Migration Wizard Creates, disables, delegates mailbox-enabled target users Uses agents to synchronize source and target Synchronizes Public Folders All rules and permissions migrated 5.5 mailbox decommissioned, not deleted Calendars available in source and target Both Approaches set msExchMasterAccountSID LDAP Attribute (Associated External Account in ADUC)
40. NT Domain Restructuring and Exchange Resource Forests Step 9 – Administer Forests Identity Management Explained Synchronization of identity information Provisioning and de-provisioning of Exchange mailboxes
41. NT Domain Restructuring and Exchange Resource Forests Identity Management – Linking the objects Account Forest = Objectsid Exchange Forest = msExchMasterAccountSID Step 9 – Administer Forests
42. NT Domain Restructuring and Exchange Resource Forests You can change any attribute you want! Step 9 – Administer Forests
43. NT Domain Restructuring and Exchange Resource Forests Step 9 – Administer Forests Identity Management – Updating the objects Choose source and target objects in Identity Management app Schedule Identity Management app to run Changes from source copy to target Based on LDAP attributes Changes should be one-way – source to target Changes in target shouldn’t map to source
44. NT Domain Restructuring and Exchange Resource Forests Step 9 – Administer Forests Identity Management – Updating the objects When msExchMasterAccountSID changes, link is broken LOCK DOWN TARGET LDAP ATTRIBUTES Administer via ADUC in source and ESM/ADUC in target Copy sAMAccountName – easier to find objects in target Groups should not be copied to target Contacts should not be copied to target Don’t copy Exchange attributes to target
45. NT Domain Restructuring and Exchange Resource Forests Step 9 – Administer Forests Identity Management – Provisioning and de-Provisioning Works on a trigger One size does not fit all Delay deletes in target when source accounts are deleted Administration Tools Account / Mailbox Management Microsoft, WebAdmin Microsoft, ADMT SystemTools, Hyena Quest, Active Roles Server
46. NT Domain Restructuring and Exchange Resource Forests Step 10 - Relax Tryout for Reality Television Game Show Watch Emeril, Dazzle Loved Ones with Gourmet PB&J Spend Time Contemplating Meaning of Life Learn Japanese, Watch Jackie Chan Movies Take up Running; Hyperventilate; Give up Running Spend time with loved ones… at Argosy
47. NT Domain Restructuring and Exchange Resource Forests Summary Many Reasons to Restructure Plan, Plan, and … oh yeah… PLAN! Create migration cookbook Build AD Forests, then migrate – don’t build during migration reACL Last Migrate all Domains Before Exchange Choose the Right Tools for the Task – Free isn’t Always Better
48. NT Domain Restructuring and Exchange Resource Forests Recommended Reading Domain Migration Cookbook http://www.microsoft.com/technet/prodtechnol/windows2000serv/deploy/cookbook/cookchp1.mspx Microsoft Windows Server 2003 Unleashed http://www.samspublishing.com/title/0672321548 Deployment Options for Exchange 2003 http://wm.quest.com/Reg/Marketing/Promos/whitepapers/kmccory/welcome.asp SimpleSync with Active Directory and Exchange 2000/2003 http://cps-systems.com/simplesync/whitepapers/SimpleSync%20with%20AD-Exchange%202000.pdf
49. NT Domain Restructuring and Exchange Resource Forests Questions? [email_address] www.ITAdvisorGroup.com