Your SlideShare is downloading. ×

UseNUMA做了什么?(2012-03-14)

2,567

Published on

UseNUMA在JDK6/Linux上的实现

UseNUMA在JDK6/Linux上的实现

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,567
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
108
Comments
0
Likes
11
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Kris Mok, Software Engineer, Taobao@rednaxelafx莫枢 /“撒迦”
  • 2. UseNUMA做了什么?JDK6/Linux中的HotSpot VM实现 阿里巴巴集团-技术共享平台- 核心系统研发-专用计算组 莫枢(撒迦) 2012-03-14
  • 3. UseNUMA的有效范围• JDK 6 / Linux上: – 硬件是NUMA架构的;BIOS打开了NUMA选项 – Xen等虚拟层的NUMA选项打开(如有) – 操作系统的NUMA选项打开 • 请检查 /sys/devices/system/node/ • 可通过numastat的输出判断JVM以下各层是否都设置正确 – 只要有设置不正确的就只能看到1个node – libnuma.so.1存在并且可加载 • 无法加载到该库会使UseNUMA = false – -XX:+UseParallelGC / -XX:+UseParallelOldGC • JDK6的HotSpot VM的其它GC未实现NUMA支持 – -XX:+ForceNUMA • 强制打开UseNUMA,但并不代表就会变快
  • 4. UseNUMA的有效范围(续)$ numastat node0 node1numa_hit 619315854 14453914507numa_miss 75245098 9049098780numa_foreign 9049098780 75245098interleave_hit 657266 752980local_node 618174068 14452713656other_node 76386884 9050299631
  • 5. UseNUMA的有效范围(续)• 确认UseNUMA的开启: – jinfo –flag UseNUMA <pid> • 正确开启时显示: -XX:+UseNUMA • 未开启时显示: -XX:-UseNUMA
  • 6. UseNUMA的有效范围(续)• Bug 7060836: – RHEL 5.5 and 5.6 dont support UseNUMA because the locality group id implementation is done via a call to sched_getcpu(), which was introduced in only in recent versions of libc. In the case of RHEL 5.5 and 5.6, Hotspot cannot find sched_getcpu and UseNUMA is disabled. – RHEL 5.4同样受影响 – 在JDK7u2有修复;淘宝JDK6u32有移植该修复
  • 7. UseNUMA的实现• JVM的GC heap初始化时 – eden空间的布局会适应NUMA拓扑 • 由MutableNUMASpace实现 • 根据locality group将地址空间划分为若干块 • 每个locality group有一块独立的分配空间 – 称为LGRPSpace • 每个LGRPSpace所包含的page在指定的locality group上分配物 理内存 – MutableNUMASpace::bias_region() – => numa_tonode_memory() – 其它空间不受NUMA拓扑影响 • survivor spaces、old generation、permanent generation默认 使用interleaved方式分配内存,不偏向任何locality group 参考资料 实验例子
  • 8. UseNUMA的实现(续) 普通的ParallelScavengeHeap布局Heap PSYoungGen total 150208K, used 59114K[0x0000000758600000, 0x0000000762da0000, 0x0000000800000000) eden space 128768K, 38% used[0x0000000758600000,0x000000075b6ac8d8,0x00000007603c0000) from space 21440K, 43% used[0x00000007603c0000,0x0000000760cce0d0,0x00000007618b0000) to space 21440K, 0% used[0x00000007618b0000,0x00000007618b0000,0x0000000762da0000) ParOldGen total 343296K, used 0K[0x0000000609200000, 0x000000061e140000, 0x0000000758600000) object space 343296K, 0% used[0x0000000609200000,0x0000000609200000,0x000000061e140000) PSPermGen total 21248K, used 16956K[0x0000000604000000, 0x00000006054c0000, 0x0000000609200000) object space 21248K, 79% used[0x0000000604000000,0x000000060508f390,0x00000006054c0000)
  • 9. UseNUMA的实现(续)Permanent Generation (commited) 604000000 default anon=5312 dirty=5312 N1=5312 6054c0000 default Permanent Generation (reserved) Old Generation (commited) 609200000 default anon=85824 dirty=85824 N1=85824 61e140000 default Old Generation (reserved) Young Generation (commited) 758600000 default anon=42912 dirty=42912 N1=42912 762da0000 default Young Generation (reserved) (日志来自/proc/<pid>/numa_maps)
  • 10. UseNUMA的实现(续) 使用-XX:+UseNUMA的Heap ParallelScavengeHeap布局 PSYoungGen total 150208K, used 24082K[0x0000000758600000, 0x0000000762da0000, 0x0000000800000000) eden space 128768K, 11% used[0x0000000758600000,0x000000075d122cc8,0x00000007603c0000) lgrp 0 space 64384K, 4% used[0x0000000758600000,0x0000000758883d98,0x000000075c4e0000) local/remote/unbiased/uncommitted: 0K/0K/0K/0K, large/small pages: 0/0 lgrp 1 space 64384K, 19% used[0x000000075c4e0000,0x000000075d122cc8,0x00000007603c0000) local/remote/unbiased/uncommitted: 0K/0K/0K/0K, large/small pages: 0/0 from space 21440K, 41% used[0x00000007603c0000,0x0000000760c7e0b0,0x00000007618b0000) to space 21440K, 0% used[0x00000007618b0000,0x00000007618b0000,0x0000000762da0000) ParOldGen total 343296K, used 0K[0x0000000609200000, 0x000000061e140000, 0x0000000758600000) object space 343296K, 0% used[0x0000000609200000,0x0000000609200000,0x000000061e140000) PSPermGen total 21248K, used 16790K[0x0000000604000000, 0x00000006054c0000, 0x0000000609200000) object space 21248K, 79% used[0x0000000604000000,0x00000006050659a0,0x00000006054c0000)
  • 11. UseNUMA的实现(续)Permanent Generation (commited) 604000000 interleave=0-1 anon=4185 dirty=4185 N0=2093 N1=2092 6054c0000 default Permanent Generation (reserved) Old Generation (commited) 使用-XX:+UseNUMA的 609200000 interleave=0-1 ParallelScavengeHeap布局 61e140000 default Old Generation (reserved) LGRPSpace (lgrp_id=0) 758600000 bind=0 anon=16096 dirty=16096 N0=16096 LGRPSpace (lgrp_id=1) 75c4e0000 bind=1 anon=16096 dirty=16096 N1=16096 7603c0000 interleave=0-1 anon=2239 dirty=2239 active=2215 N0=1120 N1=1119 Survivor Space 0 (commited) 7618b0000 interleave=0-1 Survivor Space 1 (commited) 762da0000 default Young Generation (reserved)
  • 12. UseNUMA的实现(续)• Java线程启动时 – 记录线程所在的CPU对应的locality group id• 外部线程attach到JVM时 – 同上
  • 13. UseNUMA的实现(续)• Java程序正常运行时 – 每个locality group的线程在各自的LGRPSpace 中分配空间 • 每个线程在自己的TLAB里分配空间给Java对象 • 每个线程的TLAB都只在所属的locality group的 LGRPSpace里分配 – MutableNUMASpace::allocate() – 每次分配TLAB会重新记录当前线程的lgrp_id,以便适应 线程在不同locality group之间的迁移 • 没有“shared eden” – 每当有LGRPSpace填满了就触发一次minor GC
  • 14. UseNUMA的实现(续)• (Linux上不支持)NUMA拓扑发生变化时 – 所有Java线程的locality group id被清除 • MutableNUMASpace::update_layout() – 每个线程在之后第一次尝试分配空间时重新记 录local group id • MutableNUMASpace::allocate() • MutableNUMASpace::cas_allocate()
  • 15. QUESTIONS?
  • 16. Kris Mok, Software Engineer, Taobao@rednaxelafx莫枢 /“撒迦”

×