SlideShare a Scribd company logo
Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
What I work on ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object]
Overview ,[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object]
Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
In the beginning ,[object Object]
In the beginning ,[object Object],[object Object]
In the beginning ,[object Object],[object Object],[object Object]
A world without failure
A world without failure ,[object Object]
A world without failure ,[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
A world without failure ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data Consistency ,[object Object]
User Expectations ,[object Object]
User Expectations ,[object Object],[object Object]
User Expectations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases
Databases ,[object Object]
Databases ,[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Databases ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What databases are good at ,[object Object]
What databases are good at ,[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object]
What databases are good at ,[object Object],[object Object],[object Object],[object Object],[object Object]
Easy solution to data consistency ,[object Object],[object Object],[object Object]
Revelation #1 ,[object Object]
Revelation #2 ,[object Object]
Revelation #3 ,[object Object]
Revelation #3 ,[object Object],[object Object]
Eat my data ,[object Object]
Where data can be ,[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data can be ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Data flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Simple Application: Save==on disk ,[object Object],[object Object],[object Object]
Saving a simple document ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #1 ,[object Object],[object Object]
Word Processor Saving -1 Bug ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Bug #2, 3 and 4 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object]
File System Integrity ,[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data journaling ,[object Object]
Atomic write(2)
Atomic write(2) ,[object Object]
Atomic write(2) ,[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object]
Atomic write(2) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Eat My Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CRASH
Write to Temp file, rename ,[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object]
Write to Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Temp file, rename ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Now all is good with the world...
Now all is good with the world... ,[object Object]
Now all is good with the world... ,[object Object],[object Object]
[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object]
Now all is good with the world... ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object]
File System Integrity ,[object Object],[object Object],[object Object],[object Object],[object Object]
data=ordered ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
other systems ,[object Object],[object Object],[object Object],[object Object],[object Object]
flush and sync ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Flush the buffers! Sync to disk  before  rename
A tale of libxml2 ,[object Object]
A tale of libxml2 ,[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object]
A tale of libxml2 ,[object Object],[object Object],[object Object],[object Object],[object Object]
so, replace ,[object Object]
gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE  *fp; char  *xmlbuf; int  fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
Nearing Nirvana ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
Except if you want to be portable... ,[object Object],[object Object],[object Object]
on fsync, POSIX Says... ,[object Object]
on fsync, POSIX Says... ,[object Object]
POSIX compliant fsync ,[object Object]
POSIX compliant fsync ,[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object]
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],gcc
POSIX compliant fsync ,[object Object],[object Object],[object Object],[object Object],pushl  %ebp movl  %esp, %ebp movl  $0, %eax popl  %ebp ret gcc
Tale of a really fast database server ,[object Object]
Tale of a really fast database server ,[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object]
Tale of a really fast database server ,[object Object],[object Object],[object Object],[object Object]
fsync() doesn't have to sync ,[object Object]
fsync() doesn't have to sync ,[object Object],[object Object]
Standards are great ,[object Object]
Standards are great ,[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object]
Standards are great ,[object Object],[object Object],[object Object],[object Object]
#ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier.  Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
#if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ #  define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 #  error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /*  fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
Yes, some OS Vendors hate you ,[object Object]
Big Files
Big Files ,[object Object]
Big Files ,[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Big Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object]
Large directories ,[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object]
Large directories ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object]
Where data is after being written ,[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
Where data is after being written ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object]
sqlite ,[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
sqlite ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object]
Performance of Large files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
inode Infos Direct blocks Indirect Blocks Double indirect blocks
Extent ,[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object]
Parallel writers ,[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Parallel writers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preallocation ,[object Object],[object Object],[object Object],[object Object],[object Object]
Tablespace allocation in NDB ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Improvements in mysql-test-run ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object]
Library Developers ,[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object]
Library Developers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
There is hope ,[object Object],[object Object],[object Object]
Good Luck!
Good Luck! ,[object Object]

More Related Content

What's hot

Kafka Tutorial: Kafka Security
Kafka Tutorial: Kafka SecurityKafka Tutorial: Kafka Security
Kafka Tutorial: Kafka Security
Jean-Paul Azar
 
Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
Treasure Data, Inc.
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理
Sadayuki Furuhashi
 
Data platformdesign
Data platformdesignData platformdesign
Data platformdesign
Ryoma Nagata
 
実践!DBベンチマークツールの使い方
実践!DBベンチマークツールの使い方実践!DBベンチマークツールの使い方
実践!DBベンチマークツールの使い方
Fujishiro Takuya
 
Kubernetes: A Short Introduction (2019)
Kubernetes: A Short Introduction (2019)Kubernetes: A Short Introduction (2019)
Kubernetes: A Short Introduction (2019)
Megan O'Keefe
 
DockerとPodmanの比較
DockerとPodmanの比較DockerとPodmanの比較
DockerとPodmanの比較
Akihiro Suda
 
MySQL・PostgreSQLだけで作る高速あいまい全文検索システム
MySQL・PostgreSQLだけで作る高速あいまい全文検索システムMySQL・PostgreSQLだけで作る高速あいまい全文検索システム
MySQL・PostgreSQLだけで作る高速あいまい全文検索システム
Kouhei Sutou
 
Kubernetes for Beginners: An Introductory Guide
Kubernetes for Beginners: An Introductory GuideKubernetes for Beginners: An Introductory Guide
Kubernetes for Beginners: An Introductory Guide
Bytemark
 
Interceptors: Into the Core of Pedestal
Interceptors: Into the Core of PedestalInterceptors: Into the Core of Pedestal
Interceptors: Into the Core of Pedestal
Kent Ohashi
 
[2D4]Python에서의 동시성_병렬성
[2D4]Python에서의 동시성_병렬성[2D4]Python에서의 동시성_병렬성
[2D4]Python에서의 동시성_병렬성
NAVER D2
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorApache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Databricks
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservices
pflueras
 
Scala の関数型プログラミングを支える技術
Scala の関数型プログラミングを支える技術Scala の関数型プログラミングを支える技術
Scala の関数型プログラミングを支える技術
Naoki Aoyama
 
FIWARE Big Data Ecosystem : Cygnus and STH Comet
FIWARE Big Data Ecosystem : Cygnus and STH CometFIWARE Big Data Ecosystem : Cygnus and STH Comet
FIWARE Big Data Ecosystem : Cygnus and STH Comet
fisuda
 
Designing Salesforce Platform Events
Designing Salesforce Platform EventsDesigning Salesforce Platform Events
Designing Salesforce Platform Events
CodeScience
 
[넥슨] kubernetes 소개 (2018)
[넥슨] kubernetes 소개 (2018)[넥슨] kubernetes 소개 (2018)
[넥슨] kubernetes 소개 (2018)
용호 최
 
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
 
쿠버네티스의 이해 #1
쿠버네티스의 이해 #1쿠버네티스의 이해 #1
쿠버네티스의 이해 #1
상욱 송
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspective
Jonas Bonér
 

What's hot (20)

Kafka Tutorial: Kafka Security
Kafka Tutorial: Kafka SecurityKafka Tutorial: Kafka Security
Kafka Tutorial: Kafka Security
 
Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理
 
Data platformdesign
Data platformdesignData platformdesign
Data platformdesign
 
実践!DBベンチマークツールの使い方
実践!DBベンチマークツールの使い方実践!DBベンチマークツールの使い方
実践!DBベンチマークツールの使い方
 
Kubernetes: A Short Introduction (2019)
Kubernetes: A Short Introduction (2019)Kubernetes: A Short Introduction (2019)
Kubernetes: A Short Introduction (2019)
 
DockerとPodmanの比較
DockerとPodmanの比較DockerとPodmanの比較
DockerとPodmanの比較
 
MySQL・PostgreSQLだけで作る高速あいまい全文検索システム
MySQL・PostgreSQLだけで作る高速あいまい全文検索システムMySQL・PostgreSQLだけで作る高速あいまい全文検索システム
MySQL・PostgreSQLだけで作る高速あいまい全文検索システム
 
Kubernetes for Beginners: An Introductory Guide
Kubernetes for Beginners: An Introductory GuideKubernetes for Beginners: An Introductory Guide
Kubernetes for Beginners: An Introductory Guide
 
Interceptors: Into the Core of Pedestal
Interceptors: Into the Core of PedestalInterceptors: Into the Core of Pedestal
Interceptors: Into the Core of Pedestal
 
[2D4]Python에서의 동시성_병렬성
[2D4]Python에서의 동시성_병렬성[2D4]Python에서의 동시성_병렬성
[2D4]Python에서의 동시성_병렬성
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorApache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservices
 
Scala の関数型プログラミングを支える技術
Scala の関数型プログラミングを支える技術Scala の関数型プログラミングを支える技術
Scala の関数型プログラミングを支える技術
 
FIWARE Big Data Ecosystem : Cygnus and STH Comet
FIWARE Big Data Ecosystem : Cygnus and STH CometFIWARE Big Data Ecosystem : Cygnus and STH Comet
FIWARE Big Data Ecosystem : Cygnus and STH Comet
 
Designing Salesforce Platform Events
Designing Salesforce Platform EventsDesigning Salesforce Platform Events
Designing Salesforce Platform Events
 
[넥슨] kubernetes 소개 (2018)
[넥슨] kubernetes 소개 (2018)[넥슨] kubernetes 소개 (2018)
[넥슨] kubernetes 소개 (2018)
 
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
 
쿠버네티스의 이해 #1
쿠버네티스의 이해 #1쿠버네티스의 이해 #1
쿠버네티스의 이해 #1
 
Event Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspectiveEvent Driven-Architecture from a Scalability perspective
Event Driven-Architecture from a Scalability perspective
 

Similar to Eat my data

Edubooktraining
EdubooktrainingEdubooktraining
Edubooktraining
norhloudspeaker
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
Michael Fong
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .net
Michael Pavlovsky
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5
sabtolinux
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy Steps
Tim Serong
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
Dave Stokes
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008
guestd9065
 
Measuring Firebird Disk I/O
Measuring Firebird Disk I/OMeasuring Firebird Disk I/O
Measuring Firebird Disk I/O
Mind The Firebird
 
Ch23 system administration
Ch23 system administration Ch23 system administration
Ch23 system administration
Raja Waseem Akhtar
 
file_c.pdf
file_c.pdffile_c.pdf
file_c.pdf
Osmania University
 
Filehandlinging cp2
Filehandlinging cp2Filehandlinging cp2
Filehandlinging cp2
Tanmay Baranwal
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01
MongoDB
 
File Handling In C++
File Handling In C++File Handling In C++
Ext 0523
Ext 0523Ext 0523
Ext 0523
littlebtc
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
IXIASOFT
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))
Papu Kumar
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
guest8212a5
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnostics
guest8212a5
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-oper
Sabbir Alam
 
Ungooglable
UngooglableUngooglable
Ungooglable
Elizabeth Leddy
 

Similar to Eat my data (20)

Edubooktraining
EdubooktrainingEdubooktraining
Edubooktraining
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
 
Sequential file programming patterns and performance with .net
Sequential  file programming patterns and performance with .netSequential  file programming patterns and performance with .net
Sequential file programming patterns and performance with .net
 
Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5Latihan8 comp-forensic-bab5
Latihan8 comp-forensic-bab5
 
High Availability in 37 Easy Steps
High Availability in 37 Easy StepsHigh Availability in 37 Easy Steps
High Availability in 37 Easy Steps
 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
 
Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008Purdue CS354 Operating Systems 2008
Purdue CS354 Operating Systems 2008
 
Measuring Firebird Disk I/O
Measuring Firebird Disk I/OMeasuring Firebird Disk I/O
Measuring Firebird Disk I/O
 
Ch23 system administration
Ch23 system administration Ch23 system administration
Ch23 system administration
 
file_c.pdf
file_c.pdffile_c.pdf
file_c.pdf
 
Filehandlinging cp2
Filehandlinging cp2Filehandlinging cp2
Filehandlinging cp2
 
Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01Keeping data-safe-webinar-2010-11-01
Keeping data-safe-webinar-2010-11-01
 
File Handling In C++
File Handling In C++File Handling In C++
File Handling In C++
 
Ext 0523
Ext 0523Ext 0523
Ext 0523
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
 
File Handling In C++(OOPs))
File Handling In C++(OOPs))File Handling In C++(OOPs))
File Handling In C++(OOPs))
 
Mastering InnoDB Diagnostics
Mastering InnoDB DiagnosticsMastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
 
Harrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnosticsHarrison fisk masteringinnodb-diagnostics
Harrison fisk masteringinnodb-diagnostics
 
Computer basics--basic comp-oper
Computer basics--basic comp-operComputer basics--basic comp-oper
Computer basics--basic comp-oper
 
Ungooglable
UngooglableUngooglable
Ungooglable
 

Recently uploaded

Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Vadym Kazulkin
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 

Recently uploaded (20)

Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 

Eat my data

  • 1. Eat My Data: (now with 20% more rant!) How everybody gets file I/O wrong Stewart Smith [email_address] Senior Software Engineer, MySQL Cluster MySQL AB
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11. A world without failure
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 64.
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
  • 75.
  • 76. Now all is good with the world...
  • 77.
  • 78.
  • 79.
  • 80.
  • 81.
  • 82.
  • 83.
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94. gint common_save_xml(xmlDocPtr doc, gchar *filename) { FILE *fp; char *xmlbuf; int fd, n; fp = g_fopen(filename, &quot;w&quot;); if(NULL == fp) return -1; xmlDocDumpFormatMemory(doc, (xmlChar **)&xmlbuf, &n, TRUE); if(n <= 0) { errno = ENOMEM; return -1; } if(fwrite(xmlbuf, sizeof (xmlChar), n, fp) < n) { xmlFree (xmlbuf); return -1; } xmlFree (xmlbuf); /* flush user-space buffers */ if (fflush (fp) != 0) return -1; if ((fd = fileno (fp)) == -1) return -1; #ifdef HAVE_FSYNC /* sync kernel-space buffers to disk */ if (fsync (fd) == -1) return -1; #endif fclose(fp); return 0; }
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 101.
  • 102.
  • 103.
  • 104.
  • 105.
  • 106.
  • 107.
  • 108.
  • 109.
  • 110.
  • 111.
  • 112.
  • 113.
  • 114.
  • 115.
  • 116.
  • 117. #ifdef HAVE_DARWIN_THREADS # ifdef F_FULLFSYNC /* This executable has been compiled on Mac OS X 10.3 or later. Assume that F_FULLFSYNC is available at run-time. */ srv_have_fullfsync = TRUE; # else /* F_FULLFSYNC */ /* This executable has been compiled on Mac OS X 10.2 or earlier. Determine if the executable is running on Mac OS X 10.3 or later. */ struct utsname utsname; if (uname(&utsname)) { fputs(&quot;InnoDB: cannot determine Mac OS X version!&quot;, stderr); } else { srv_have_fullfsync = strcmp(utsname.release, &quot;7.&quot;) >= 0; } if (!srv_have_fullfsync) { fputs(&quot;InnoDB: On Mac OS X, fsync() may be&quot; &quot; broken on internal drives,&quot; &quot;InnoDB: making transactions unsafe!&quot;, stderr); } # endif /* F_FULLFSYNC */ #endif /* HAVE_DARWIN_THREADS */
  • 118. #if defined(HAVE_DARWIN_THREADS) # ifndef F_FULLFSYNC /* The following definition is from the Mac OS X 10.3 <sys/fcntl.h> */ # define F_FULLFSYNC 51 /* fsync + ask the drive to flush to the media */ # elif F_FULLFSYNC != 51 # error &quot;F_FULLFSYNC != 51: ABI incompatibility with Mac OS X 10.3&quot; # endif /* Apple has disabled fsync() for internal disk drives in OS X. That caused corruption for a user when he tested a power outage. Let us in OS X use a nonstandard flush method recommended by an Apple engineer. */ if (!srv_have_fullfsync) { /* If we are not on an operating system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } else { ret = fcntl(file, F_FULLFSYNC, NULL); if (ret) { /* If we are not on a file system that supports this, then fall back to a plain fsync. */ ret = fsync(file); } } #elif HAVE_FDATASYNC ret = fdatasync(file); #else /* fprintf(stderr, &quot;Flushing to file %p&quot;, file); */ ret = fsync(file); #endif
  • 119.
  • 121.
  • 122.
  • 123.
  • 124.
  • 125.
  • 126.
  • 127.
  • 128.
  • 129.
  • 130.
  • 131.
  • 132.
  • 133.
  • 134.
  • 135.
  • 136.
  • 137.
  • 138.
  • 139.
  • 140.
  • 141.
  • 142.
  • 143.
  • 144.
  • 145.
  • 146.
  • 147.
  • 148.
  • 149. inode Infos Direct blocks Indirect Blocks Double indirect blocks
  • 150.
  • 151.
  • 152.
  • 153.
  • 154.
  • 155.
  • 156.
  • 157.
  • 158.
  • 159.
  • 160.
  • 161.
  • 162.
  • 163.
  • 164.
  • 165.
  • 166.
  • 168.