Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQLの限界に挑戦する

3,716 views

Published on

MySQLのSQLクエリ最大長に朝鮮

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

MySQLの限界に挑戦する

  1. 1. 1 MySQLの限界に挑戦する(?!) MySQL Casual Talks vol.10 2017-02-01 19:10 LT @meijik
  2. 2. 免責事項 • 本プレゼンテーションにおいて示されている 見解は、私自身の見解であって、私の所属す る会社・団体の見解を必ずしも反映したもの ではありません。ご了承ください。 2
  3. 3. 3 MySQLの限界 • よいこのみんなはこれを見てるよね! • 限界までMySQLを使い尽くす!!(漢のコンピュータ道) – http://nippondanji.blogspot.jp/2009/05/mysql.html • 今回のLTはそれを順に見ていこうとしました が、結果は。。。。
  4. 4. 4 SQL文の最大長 • 限界までMySQLを使い尽くす!!(漢のコン ピュータ道)によると。。。。 – MySQLサーバーが実行出来るSQL文の最大長 は、max_allowed_packetシステム変数で表され る。max_allowed_packetの最大値は1GBであ る。max_allowed_packetの値はセッションごとに も設定可能なので、デフォルトではそこそこの値 (16MBなど)に設定しておいて、必要に応じて大 きな対を使うと良いだろう。 – デフォルト値は5.6より前は1MB, 5.6以降4MB
  5. 5. 5 無意味なクエリでトライ! UNION ALLで一連の値を返すクエリを作成。 ↓こんな感じ select repeat(' union all select "b"',4000) into @a; set @a= concat('select "a" ', @a); prepare stmt1 from @a; execute stmt1; deallocate prepare stmt1;
  6. 6. 6 できたクエリ • select "a" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union allselect "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b"
  7. 7. 7 できたクエリ(一部拡大) • select "a" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union allselect "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" union all select "b" select “a” union all select “b” union all select “b” …
  8. 8. 8 さて実行! mysql> select repeat(' union all select "b"',4000) into @a; Query OK, 1 row affected (0.00 sec) mysql> set @a= concat('select "a" ', @a); Query OK, 0 rows affected (0.00 sec) mysql> prepare stmt1 from @a; ERROR 1436 (HY000): Thread stack overrun: 246184 bytes used of a 262144 byte stack, and 16000 bytes needed. Use 'mysqld -- thread_stack=#' to specify a bigger stack.
  9. 9. 9 スタック不足 • thread_stack – https://dev.mysql.com/doc/refman/5.6/ja/serve r-system-variables.html#sysvar_thread_stack • 各スレッドのスタックサイズ。デフォルトの 256KBは、通常の操作では十分な大きさです。 スレッドスタックサイズが小さすぎるとサーバ で処理できる SQL ステートメントの複雑さ、 ストアドプロシージャーの再帰の深さなど、メ モリーを大量に消費する処理が制限されます。
  10. 10. 10 じゃ2倍の512KBで • 成功! • じゃreapeatも二倍の8000で! mysql> prepare stmt1 from @a; ERROR 1064 (42000): memory exhausted near '"b" union all select "b"' at line 1 →失敗orz
  11. 11. 11 ERROR 1064 (42000) • 通常クエリが文法的に間違っているときによ くでる。 • perror 1064 – MySQL error code 1064 (ER_PARSE_ERROR): %s near ‘%-.80s’ at line %d • ただ今回は文法的にはあっている(はず)で キーワード(%s) が “memory exhausted “
  12. 12. 12 ソースをfindstr (windowsのgrepライクなコマンド) • mysql-5.7.17¥sql>findstr /nsi /c:"memory exhausted" * • sql_hints.yy.cc:1362: yyoverflow (YY_("memory exhausted"), • sql_hints.yy.cc:2099: yyerror (thd, scanner, ret, YY_("memory exhausted")); • sql_yacc.cc:19899: yyoverflow (YY_("memory exhausted"), • sql_yacc.cc:41019: yyerror (&yylloc, YYTHD, YY_("memory exhausted"));
  13. 13. 13 構文解析と字句解析 • クエリの解析 – 字句解析 – 構文解析 • 構文的には正しい(無意味な)複雑なクエリを 構文解析していて、内部的に用意しておいた メモリを使い果たしたらしい。。。 • 通常のプログラムでは構文解析にyacc, 字句 解析にlexを利用する。 – 拡張子.yyはyaccのソースファイル
  14. 14. 14 bisonとflex • クエリの解析 – 字句解析: lex -> GNU版がflex – 構文解析: yacc -> GNU版がbison
  15. 15. 15 MySQLの場合 • クエリの解析 – 字句解析: 自前で行っている – 構文解析: BNFで記述したソースをGNU bisonに てコンパイルして行っている。 • BNF(バッカス・ナウア記法) • バッカス・ナウア記法(英: Backus-Naur form)とは、文脈自由文法を定義するのに用 いられるメタ言語のことで、一般にBNFやBN 記法と略される。(wikipediaより引用)
  16. 16. 16 SQL標準をBNFで • 初期のSQLは書籍などでBNF(バッカス・ナウ ア記法)の記述があった。 • 現在奇特な方がSQL-92, 99, 2003のBNFを 公開してなさる。(多分読みきれん) – https://github.com/ronsavage/SQL • これを使えば多分クエリの整形ツールをつ くったり(多分できる)、RDBMSを作ったりする ことができると思われます。(多分無理)
  17. 17. 17 まとめ • MySQLのクエリの最大長(max_allowed_packet) • スタックに引っかかる場合→スタックサイズを 大きくして対応 • 構文解析でメモリを使い果たした場合→簡単 な対応なし(まぁほとんど起こらない) • MySQLのビルドになぜbisonが必要なのか、 と@kumagiのツイートの意味が分かった。
  18. 18. 宣伝: 書籍情報 • おうちで学べるデータベースのきほん – DBの初心者はこちらをどうぞ。 – なぜか韓国語版が発売 • 入手方法 – じゃんけんで勝つ。 – もしくは、Amazonでポチる w。 18
  19. 19. 19 THANK YOU • ご清聴ありがとうございました。 • [fyi] yacc/lexの参考書籍 – 日本語 • lex&yaccプログラミング (オライリージャパン) • Cコンパイラ設計―yacc/lexの応用 • yacc/lex―プログラムジェネレータon UNIX – 英語 • flex&bison (O‘reilly) – 比較的新しい(2009/08)本(@meijikは未見) – SQLの例がある(らしい)
  20. 20. 20 MySQLの限界にチョット挑戦した。 MySQL Casual Talks vol.10 2017-02-01 LT @meijik

×