Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

高速・省メモリにlibsvm形式で ダンプする方法を研究してみた

3,475 views

Published on

JapanR 2016でのLTのときの資料です。
xgboost::xgb.DMatrixを作るときのメモリ消費量を抑えるために、一旦libsvm形式のファイルを出力してから読み込む方法を使うようにしました。このとき高速にlibsvm形式でダンプするために、data.table::fwriteを魔改造したと言うお話です。

Published in: Engineering
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

高速・省メモリにlibsvm形式で ダンプする方法を研究してみた

  1. 1. libsvm 2016/11/27 hskksk @ JapanR 2016
  2. 2. • : • : R, Python, C++ • : • :
  3. 3. xgboost kaggler
  4. 4. Bosch Production Line Performance 15
  5. 5. xgboost
  6. 6. xgb.DMatrix
  7. 7. # feature label = readRDS("label.rds") feature_set_A = readRDS("feature_set_A.rds") feature_set_B = readRDS("feature_set_B.rds") # feature cbind mat = cbind( feature_set_A, feature_set_B ) ↑ # DMatrix dmat = xgb.DMatrix(mat, label=label)
  8. 8. cbind
  9. 9. ※cbind rm(vars); gc()
  10. 10. xgb.DMatrix
  11. 11. Python
  12. 12. libsvm ※R
  13. 13. 1. cbind libsvm 2. DMatrix
  14. 14. cbind libsvm data.table::fwrite_libsvm(list_of_matrices, file)
  15. 15. data.table fork fwrite
  16. 16. # feature label = readRDS("label.rds") feature_set_A = fread("feature_set_A.csv") feature_set_B = fread("feature_set_B.csv") # feature list # 1 label matrices = list(label, feature_set_A, feature_set_B) # libsvm fwrite_libsvm(matrices, "libsvm.txt") # DMatrix dmat = xgb.DMatrix("libsvm.txt")
  17. 17. fwrite OpenMP 8.5GB/120sec @ Xeon 2.5GHz ✕ 8
  18. 18. data.table PR
  19. 19. https://github.com/hskksk/data.table kaggler !!
  20. 20. Enjoy Kaggling with R !!

×