Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

R-3.0.0でGLM

5,369 views

Published on

第30回Tokyo.Rでの発表資料です

Published in: Education
  • ACCESS that WEBSITE Over for All Ebooks (Unlimited) ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... DOWNLOAD FULL EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • ACCESS that WEBSITE Over for All Ebooks (Unlimited) ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... DOWNLOAD FULL EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

R-3.0.0でGLM

  1. 1. R-3.0.0を使ってみてわかった1つのこと2013/04/20Tokyo.R#30和田 計也サイバー系
  2. 2. 2013/04/20はじめに※この発表は個人の見解であり、所属する組織の公式見解ではありません。サイバー系2
  3. 3. 2013/04/20u和田 計也(@wdkz)u 静岡県袋井市出身uサイバー系企業でデータマイニングエンジニア職u 最近は奥華子ばっか聴いてますu前職はバイオベンチャーu バイオマーカ探索してたu 学生時代は枯草菌の研究自己紹介サイバー系3
  4. 4. 2013/04/20いきなりですがサイバー系u祝!!Tokyo.R 30回達成u祝!!R-3.0.0リリース4
  5. 5. 2013/04/20R-3.0.0の特徴サイバー系u 一番のハイライトは2^31-1以上の長さのベクトル対応5http://blog.revolutionanalytics.com/2012/07/big-vectors-coming-to-r.html
  6. 6. 2013/04/20kohskeさんのまとめがめっちゃわかりやすい!サイバー系6http://qiita.com/items/2d21344ba11d030c3cf0挙動が微妙って...
  7. 7. 2013/04/20早速トライR-3.0.0サイバー系u 2^31-1の呪縛からの脱却っぷりを試してみよう!u 使用データはもちろんairline(からカラム選択した)データ・約1.2億行 x 6列・使用マシンは244Gbyte メモリのやつ・OSは64bit Ubuntu12.047
  8. 8. 2013/04/20(参考)怪物マシンはサイバー系u AWS利用した8スポットインスタンスなら$0.35/hrぐらい
  9. 9. 2013/04/20UbuntuマシンにR-3.0.0をセットアップサイバー系u cranミラーとか利用するとapt-getで3.0.0が入らないよ9設定ファイルをちょいと変更$ sudo vi /etc/apt/sources.list deb http://ppa.launchpad.net/marutter/rdev/ubuntu precise maindeb-src http://ppa.launchpad.net/marutter/rdev/ubuntu precise main#(12.10の場合はquantalね)$sudo add-apt-repository ppa:gwibber-daily/ppa$ sudo apt-get update$ sudo apt-get install r-base↑これでR-3.0.0がインストールできるhttp://www.personal.psu.edu/mar36/blogs/the_ubuntu_r_blog/2013/03/moving-to-r-300-on-ubuntu.html(2013/04/19現在)
  10. 10. 2013/04/20R-3.0.0起動サイバー系10OSのメモリは240Gbyte弱
  11. 11. 2013/04/20airline.csv(カラム選択済み)読み込みサイバー系111.2億行程度なら2^31-1(≒21.4億)まで達してないからR-2系でもread.csv等でデータセットを読み込むところまでは可能だったそもそもこのairlineどうやって作ってるよ?って疑問を持った人は→の資料を参照
  12. 12. 2013/04/20(参考)R-2.15.3だともちろん爆死サイバー系12Rのバージョンは2.15.3読み込んだairlineは3Gbyte弱読み込んだairlineは1.2億行 x 6列はい爆死OSのメモリは240Gbyte弱
  13. 13. 2013/04/20待望のR-3.0.0だと...サイバー系Rのバージョンは3.0.0!?あとは一緒13
  14. 14. 2013/04/20待望のR-3.0.0だと...サイバー系14u tracebackを見ると、glm関数内で呼び出してるmodel.matrix関数でエラってるっぽい!?確かにmodel.matrix関数でエラーが再現した
  15. 15. 2013/04/20戦略サイバー系u R-3.0.0は2^31-1以上の要素数を持つmatrixを扱うことができるようになったが、model.matrix関数でデザイン行列を作成しようとすると(デカいmatrixの場合に)エラーになるu 仕方ないので以下のようにやってみたi) Year毎にmatrixを分割してii) それぞれでmodel.matrixを適用させデザイン行列を  作成してiii) それぞれcsvファイルに出力してiv) 全csvファイルを結合してv) ↑この結合したcsvファイルをread.csvで読み込む15model.matrix後の要素数が2^31以上
  16. 16. 2013/04/20仕方ない...サイバー系16#formulaobject <- Cancelled ~ Year + Month + DayOfWeek + CRSElapsedTime + Distance#NA除外しとかないとmodel.matrixの返り値とnrowが異なっちゃうairline <- airline[-which(is.na(airline$CRSElapsedTime) |is.na(airline$Distance)),]chunk.level <- levels(as.factor(data[,chunk]))for(i in 1:length(chunk.level)){assign(paste("airline", chunk.level[i], sep=""),subset(airline, Year==chunk.level[i]))  assign(paste("mm", chunk.level[i], sep=""),model.matrix(object,data=get(paste("airline", chunk.level[i], sep=""))))  write.table(get(paste("mm", chunk.level[i], sep="")),file=paste("mm", chunk.level[i], ".csv", sep=""), sep=",",col.names=ifelse(i==1,T,F), row.names=F, quote=F)}csv_files <- dir()[grep("^mmd{4}.csv$", dir())]cmd_str <- paste("cat ", paste(csv_files, collapse=" "), " > mm.csv", sep="")system(cmd_str)airline.mm <- read.table("mm.csv", sep=",", skip=1, header=F)clname <- as.character(read.table("mm.csv", sep=",", nrows=1,stringsAsFactors=F))colnames(airline.mm) <- clname read.csv使うと(メモリ量が充分でない場合に)コケる場合があった。その場合はカラム名なしでread.table後にカラム名を付けるとOKだった
  17. 17. 2013/04/20デザイン行列のcsvファイルを作るまでサイバー系17
  18. 18. 2013/04/20デザイン行列を読み込むサイバー系18
  19. 19. 2013/04/20そしてGLMへサイバー系19
  20. 20. 2013/04/20GLMの結果が出た!!サイバー系2048G byte !!
  21. 21. 2013/04/20R-3.0.0の時代がやってきた。サイバー系21u airlineのような1.2億行程度のミドルデータなら、メモリにデータを載せてRの標準関数で分析できる時代が到来した!!ただし元データがR上で3Gbyte弱だったとしても、glmするのに200G byte弱のメモリが必要...
  22. 22. 2013/04/20最後にサイバー系R-3.0.0を使ってみてわかった、ただ1つのこと22相変わらず memory notmapped エラーは出ます!!

×