SlideShare a Scribd company logo
1 of 30
Download to read offline
Python⾼级内存管理
- xiaorui.cc
Object-specific allocators
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | <----- Python memory (under PyMem manager's control) ------> | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | <------ Virtual memory allocated for the python process -------> |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
__________________________________ __________________________________
[ ] [ ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
* Request in bytes Size of allocated block Size class idx
* ----------------------------------------------------------------
* 1-8 8 0
* 9-16 16 1
* 17-24 24 2
* 25-32 32 3
* 33-40 40 4
* 41-48 48 5
* 49-56 56 6
* 57-64 64 7
* ... ... ...
* 497-504 504 62
* 505-512 512 63
*
*
*/
名词解释
process heap
Arenas
Pool
UsedPools
FreePools
method
posix malloc
python memory pool
object buffer pool
Arena
FeeePool
Pool
Pool
Pool
Pool
Headers
No BLock
Arena
malloc heap &
pool
Process
stack
heap
bss
init data
text
UserPool
1-8
… …
249 - 256
Pool
Free Block
Free Block
Use Block
userdpool design
UserdPools
1-8
9-16
17-24
…
…
249-256
Pool
Header
Free Block
Free Block
Header
分配
回收
…
Pool
Free Block
Free Block
Use Block
同⼀个Pool下Block⼀样长
单Pool为4kb
Block及Pool都为单链表
free pool desgin
FeeePool
Pool
Pool
Pool
…
Pool
Headers
No BLock
Pool
Headers
No BLock
Pool为4kb⼤小
Pool清理Headers
where store variable ?
run-time Stack
list
dict
int
heap
[1 ,2, 3]
{“n”: “1”}
1
why ?
In [1]: a = 123
In [2]: b = 123
In [3]: a is b
Out[3]: True
In [4]: a = 1000
In [5]: b = 1000
In [6]: a is b
Out[6]: False
In [7]: a = 'n'
In [8]: b = 'n'
In [9]: a is b
Out[9]: True
In [10]: a = "python"
In [11]: b = "python"
In [12]: a is b
Out[12]: True
why ?
In [1]: def go(var):
...: print id(var)
…:
In [2]: id(a)
Out[2]: 4401335072
In [3]: go(a)
4401335072
In [10]: a = b = 'nima'
In [11]: b = a
In [12]: a is b
Out[12]: True
In [13]: b = 'hehe'
In [14]: a is b
Out[14]: False
只有引用 ?
python objects stored in memory?
names
names
object
Python Has Names, Not Variables ! ! !
整数对象池
-5 -4 … 0 … 256 257
小整数 ⼤整数
… … -5 -4 … … 257 … …
var_1 var_2
… … -5 -4 … … 257 … …
var_3 var_4
the same addr !
not the same addr !
28 bytes
解释器初始化
整数对象池
Block List
Free List
PyIntBlock PyIntBlock
PyIntBlock PyIntBlock
不会归还给 Arena和os ! ! !
字符对象池
a b c d … … …
var_1 var_2
the same addr !
单个字符38 bytes
由解释器初始化
字符串对象池
aa en cao oh woyao buyao kuai feile
var_1
0 1 2 3 …
var_2
hash存储变量
共用地址
记录引用计数
ref
ref count
x = 300
y = x
z = [x, y]
X
ref += 1
300
y
Z
ref += 1
ref += 2
References -> 4 !
What does del do?
x = 300
y = x
del x
X
ref -= 1
300
y
References -> 1!The del statement doesn’t delete objects.
• removes that name as a reference to that object
• reduces the ref count by 1
ref count case
def go():
w = 300
go()
a = “fuc . ”
del a
b = “en, a”
b = None
ref count +1
w is out of scope; ref count -1
del a; ref count -1
重新赋值; ref count -1
cyclical refclass Node:
def __init__(self, va):
self.va = va
def next(self, next):
self.next = next
mid = Node(‘root’)
left = Node(‘left’)
right = Node(‘right’)
mid(left)
left.next(right)
right.next(left)
Mid
rightleft
if del mid node:
how ?
mark & sweep
gc root
b
a
w
c
K G
R
分代回收
node node node node node node node
node node node node node node
node node node node node node
可变 vs 不可变 (obj)
string
int
tuple
list
dict
container objects
a = [10, 10, 11]
b = a
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
11
copy.copy
a = [10, 10, [10, 11] ]
b = copy.copy(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
10
ref
PyListObject
10
11
copy.deepcopy
a = [10, [ 10, 11 ] ]
b = copy.deep(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
ref
PyListObject
10
11
PyListObject
10
11
diy gc
import gc
import sys
gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK)
a=[]
b=[]
a.append(b)
print 'a refcount:',sys.getrefcount(a) # 2
print 'b refcount:',sys.getrefcount(b) # 3
del a
del b
print gc.collect() # 0
Garbage Collector Optimize
memory bound
可以降低threshold来时间换空间
cpu bound
提⾼threshold来空间换时间
暂停gc, 引⼊master worker设计
引用计数 跟 gil 的影响 ?
gc 是否是原⼦ ?
gc的 stop the world现象 ?
…
Q & A
“ END ”
– xiaorui.cc

More Related Content

What's hot

Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
MongoDB
 

What's hot (20)

Groovy collection api
Groovy collection apiGroovy collection api
Groovy collection api
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica Sets
 
Damn Fine CoffeeScript
Damn Fine CoffeeScriptDamn Fine CoffeeScript
Damn Fine CoffeeScript
 
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : NotesCUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
 
Python For Data Science Cheat Sheet
Python For Data Science Cheat SheetPython For Data Science Cheat Sheet
Python For Data Science Cheat Sheet
 
Advanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part IIAdvanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part II
 
Android Guava
Android GuavaAndroid Guava
Android Guava
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
 
Exploring slides
Exploring slidesExploring slides
Exploring slides
 
Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語
 
Py3k
Py3kPy3k
Py3k
 
Elm: give it a try
Elm: give it a tryElm: give it a try
Elm: give it a try
 
집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링
 
밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AI밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AI
 
Python postgre sql a wonderful wedding
Python postgre sql   a wonderful weddingPython postgre sql   a wonderful wedding
Python postgre sql a wonderful wedding
 
Intoduction to dynamic memory allocation
Intoduction to dynamic memory allocationIntoduction to dynamic memory allocation
Intoduction to dynamic memory allocation
 
Frsa
FrsaFrsa
Frsa
 
Pybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in PythonPybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in Python
 
Python seaborn cheat_sheet
Python seaborn cheat_sheetPython seaborn cheat_sheet
Python seaborn cheat_sheet
 
Introduzione a C#
Introduzione a C#Introduzione a C#
Introduzione a C#
 

Viewers also liked

Viewers also liked (6)

cdn的那些事儿
cdn的那些事儿cdn的那些事儿
cdn的那些事儿
 
Development with Qt for Windows CE
Development with Qt for Windows CEDevelopment with Qt for Windows CE
Development with Qt for Windows CE
 
领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发
 
Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...
 
Micro service
Micro serviceMicro service
Micro service
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 

Similar to python高级内存管理

Profiling and optimization
Profiling and optimizationProfiling and optimization
Profiling and optimization
g3_nittala
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка Twisted
Maxim Kulsha
 
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
MongoSF
 
Replication and Replica Sets
Replication and Replica SetsReplication and Replica Sets
Replication and Replica Sets
MongoDB
 
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
James Clause
 

Similar to python高级内存管理 (20)

An overview of Python 2.7
An overview of Python 2.7An overview of Python 2.7
An overview of Python 2.7
 
A tour of Python
A tour of PythonA tour of Python
A tour of Python
 
A Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsA Few of My Favorite (Python) Things
A Few of My Favorite (Python) Things
 
python-cheatsheets.pdf
python-cheatsheets.pdfpython-cheatsheets.pdf
python-cheatsheets.pdf
 
python-cheatsheets that will be for coders
python-cheatsheets that will be for coderspython-cheatsheets that will be for coders
python-cheatsheets that will be for coders
 
Profiling and optimization
Profiling and optimizationProfiling and optimization
Profiling and optimization
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка Twisted
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка Twisted
 
Functional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonFunctional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with Python
 
Introduction to Python
Introduction to PythonIntroduction to Python
Introduction to Python
 
PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)
 
PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)
 
Python 101 language features and functional programming
Python 101 language features and functional programmingPython 101 language features and functional programming
Python 101 language features and functional programming
 
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
 
Python fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanPython fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuan
 
Intro to Python
Intro to PythonIntro to Python
Intro to Python
 
Ggplot2 v3
Ggplot2 v3Ggplot2 v3
Ggplot2 v3
 
Replication and Replica Sets
Replication and Replica SetsReplication and Replica Sets
Replication and Replica Sets
 
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
 
[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2
 

More from rfyiamcool (12)

Redis cluster那些事儿
Redis cluster那些事儿Redis cluster那些事儿
Redis cluster那些事儿
 
Golang advance
Golang advanceGolang advance
Golang advance
 
Golang 高性能实战
Golang 高性能实战Golang 高性能实战
Golang 高性能实战
 
Mysql fast share
Mysql fast shareMysql fast share
Mysql fast share
 
分析mysql acid 设计实现
分析mysql acid 设计实现分析mysql acid 设计实现
分析mysql acid 设计实现
 
Raft
Raft Raft
Raft
 
大话redis设计实现
大话redis设计实现大话redis设计实现
大话redis设计实现
 
python gil
python gilpython gil
python gil
 
async io frame
async io frameasync io frame
async io frame
 
异步io框架的实现
异步io框架的实现异步io框架的实现
异步io框架的实现
 
美妙的多进程管理
美妙的多进程管理美妙的多进程管理
美妙的多进程管理
 
聊聊我接触的集群管理
聊聊我接触的集群管理聊聊我接触的集群管理
聊聊我接触的集群管理
 

Recently uploaded

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
masabamasaba
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
masabamasaba
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
masabamasaba
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
 

Recently uploaded (20)

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
WSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - Keynote
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
 
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
 
%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
tonesoftg
tonesoftgtonesoftg
tonesoftg
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 

python高级内存管理

  • 2. Object-specific allocators _____ ______ ______ ________ [ int ] [ dict ] [ list ] ... [ string ] Python core | +3 | <----- Object-specific memory -----> | <-- Non-object memory --> | _______________________________ | | [ Python's object allocator ] | | +2 | ####### Object memory ####### | <------ Internal buffers ------> | ______________________________________________________________ | [ Python's raw memory allocator (PyMem_ API) ] | +1 | <----- Python memory (under PyMem manager's control) ------> | | __________________________________________________________________ [ Underlying general-purpose allocator (ex: C library malloc) ] 0 | <------ Virtual memory allocated for the python process -------> | ========================================================================= _______________________________________________________________________ [ OS-specific Virtual Memory Manager (VMM) ] -1 | <--- Kernel dynamic storage allocation & management (page-based) ---> | __________________________________ __________________________________ [ ] [ ] -2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
  • 3. * Request in bytes Size of allocated block Size class idx * ---------------------------------------------------------------- * 1-8 8 0 * 9-16 16 1 * 17-24 24 2 * 25-32 32 3 * 33-40 40 4 * 41-48 48 5 * 49-56 56 6 * 57-64 64 7 * ... ... ... * 497-504 504 62 * 505-512 512 63 * * */
  • 5. method posix malloc python memory pool object buffer pool
  • 6. Arena FeeePool Pool Pool Pool Pool Headers No BLock Arena malloc heap & pool Process stack heap bss init data text UserPool 1-8 … … 249 - 256 Pool Free Block Free Block Use Block
  • 7. userdpool design UserdPools 1-8 9-16 17-24 … … 249-256 Pool Header Free Block Free Block Header 分配 回收 … Pool Free Block Free Block Use Block 同⼀个Pool下Block⼀样长 单Pool为4kb Block及Pool都为单链表
  • 8. free pool desgin FeeePool Pool Pool Pool … Pool Headers No BLock Pool Headers No BLock Pool为4kb⼤小 Pool清理Headers
  • 9. where store variable ? run-time Stack list dict int heap [1 ,2, 3] {“n”: “1”} 1
  • 10. why ? In [1]: a = 123 In [2]: b = 123 In [3]: a is b Out[3]: True In [4]: a = 1000 In [5]: b = 1000 In [6]: a is b Out[6]: False In [7]: a = 'n' In [8]: b = 'n' In [9]: a is b Out[9]: True In [10]: a = "python" In [11]: b = "python" In [12]: a is b Out[12]: True
  • 11. why ? In [1]: def go(var): ...: print id(var) …: In [2]: id(a) Out[2]: 4401335072 In [3]: go(a) 4401335072 In [10]: a = b = 'nima' In [11]: b = a In [12]: a is b Out[12]: True In [13]: b = 'hehe' In [14]: a is b Out[14]: False 只有引用 ?
  • 12. python objects stored in memory? names names object Python Has Names, Not Variables ! ! !
  • 13. 整数对象池 -5 -4 … 0 … 256 257 小整数 ⼤整数 … … -5 -4 … … 257 … … var_1 var_2 … … -5 -4 … … 257 … … var_3 var_4 the same addr ! not the same addr ! 28 bytes 解释器初始化
  • 14. 整数对象池 Block List Free List PyIntBlock PyIntBlock PyIntBlock PyIntBlock 不会归还给 Arena和os ! ! !
  • 15. 字符对象池 a b c d … … … var_1 var_2 the same addr ! 单个字符38 bytes 由解释器初始化
  • 16. 字符串对象池 aa en cao oh woyao buyao kuai feile var_1 0 1 2 3 … var_2 hash存储变量 共用地址 记录引用计数 ref
  • 17. ref count x = 300 y = x z = [x, y] X ref += 1 300 y Z ref += 1 ref += 2 References -> 4 !
  • 18. What does del do? x = 300 y = x del x X ref -= 1 300 y References -> 1!The del statement doesn’t delete objects. • removes that name as a reference to that object • reduces the ref count by 1
  • 19. ref count case def go(): w = 300 go() a = “fuc . ” del a b = “en, a” b = None ref count +1 w is out of scope; ref count -1 del a; ref count -1 重新赋值; ref count -1
  • 20. cyclical refclass Node: def __init__(self, va): self.va = va def next(self, next): self.next = next mid = Node(‘root’) left = Node(‘left’) right = Node(‘right’) mid(left) left.next(right) right.next(left) Mid rightleft if del mid node: how ?
  • 21. mark & sweep gc root b a w c K G R
  • 22. 分代回收 node node node node node node node node node node node node node node node node node node node
  • 23. 可变 vs 不可变 (obj) string int tuple list dict
  • 24. container objects a = [10, 10, 11] b = a PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 11
  • 25. copy.copy a = [10, 10, [10, 11] ] b = copy.copy(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 ref PyListObject Type list rc 1 items size … … 10 10 ref PyListObject 10 11
  • 26. copy.deepcopy a = [10, [ 10, 11 ] ] b = copy.deep(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 ref PyListObject Type list rc 1 items size … … 10 ref PyListObject 10 11 PyListObject 10 11
  • 27. diy gc import gc import sys gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK) a=[] b=[] a.append(b) print 'a refcount:',sys.getrefcount(a) # 2 print 'b refcount:',sys.getrefcount(b) # 3 del a del b print gc.collect() # 0
  • 28. Garbage Collector Optimize memory bound 可以降低threshold来时间换空间 cpu bound 提⾼threshold来空间换时间 暂停gc, 引⼊master worker设计
  • 29. 引用计数 跟 gil 的影响 ? gc 是否是原⼦ ? gc的 stop the world现象 ? … Q & A
  • 30. “ END ” – xiaorui.cc