Recent IT Development and Women: Big Data and The Power of Women in Goryeo

Uploaded on

Intro to Big Data and the power of women in Goryeo dynasty with the Mongol Empire in Korea

Intro to Big Data and the power of women in Goryeo dynasty with the Mongol Empire in Korea

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. HiPIC Recent IT Development and Women: Big Data and The Power of Women in Goryeo KWiSE Annual Meeting Chapman University, CA Oct 20th 2012 Jongwook Woo (PhD) High-Performance Internet Computing Center (HiPIC) Educational Partner with Cloudera and Grants Awardee of Amazon AWS Computer Information Systems Department California State University, Los Angeles Jongwook Woo CSULA
  • 2. HiPIC ContentsPart I. Big Data Fundamentals of Big Data Data-Intensive Computing: Hadoop Big Data Supporters and Use CasesPart II. The Power of Women in Goryeo Dynasty North East Asia before the Mongol Empire Korea and Mongol The Empress Gi CSULA Jongwook Woo
  • 3. HiPIC Part IBig Data Fundamentals of Big Data NoSQL DB: HBase, MongoDB Data-Intensive Computing: Hadoop Big Data Supporters and Use Cases CSULA Jongwook Woo
  • 4. HiPIC Experience in Big Data Grants  Received Amazon AWS in Education Research Grant (July 2012 - July 2014)  Received Amazon AWS in Education Coursework Grants (July 2012 - July 2013, Jan 2011 - Dec 2011 Partnership  Received Academic Education Partnership with Cloudera since June 2012 Certificate  Certificate of Achievement in the Big Data University Training Course, “Hadoop Fundamentals I”, July 8 2012 Cloud Computing Blog  CSULA Jongwook Woo
  • 5. What is Big Data, Map/Reduce, Hadoop, NoSQL DB onHiPIC Cloud Computing CSULA Jongwook Woo
  • 6. HiPIC Big DataToo much data Tera-Byte (1012), Peta-byte (1015) – Because of web – Sensor Data, Bioinformatics, Social Computing, smart phone, online game…Cannot handle with the legacy approach Too big Un-/Semi-structured data CSULA Jongwook Woo
  • 7. HiPIC Two Issues in Big DataHow to store Big Data NoSQL DBHow to compute Big Data Parallel Computing with multiple cheap computers – Not need super computers CSULA Jongwook Woo
  • 8. HiPIC ContentsFundamentals of Big DataData-Intensive Computing: HadoopBig Data Supporters and Use Cases CSULA Jongwook Woo
  • 9. HiPIC Data nowadays• Data Issues o data grows to 10TB, and then 100TB. o Unstructured data coming from sources  like Facebook, Twitter, RFID readers, sensors, and so on.  Need to derive information from both the relational data and the unstructured data • as soon as possible.• Solution to efficiently compute Big Data o Hadoop Map/Reduce CSULA Jongwook Woo
  • 10. HiPIC Solutions in Big Data Computation Map/Reduce by Google (Key, Value) parallel computing Apache Hadoop  Big Data Data Computation (MapReduce, Pig) Integrating MapReduce and RDB  Oracle + Hadoop  Sybase IQ  Vertica + Hadoop  Hadoop DB  Greenplum  Aster Data Integrating MapReduce and NoSQL DB  MongoDB MapReduce  HBase CSULA Jongwook Woo
  • 11. HiPIC Apache Hadoop Motivated by Google Map/Reduce and GFS  open source project of the Apache Foundation.  framework written in Java – originally developed by Doug Cutting • who named it after his sons toy elephant. Two core Components  Storage: HDFS – High Bandwidth Clustered storage  Processing: Map/Reduce – Fault Tolerant Distributed Processing Hadoop scales linearly with  data size  Analysis complexity CSULA Jongwook Woo
  • 12. HiPIC Hadoop issues Map/Reduce is not DB  Algorithm in Restricted Parallel Computing HDFS and HBase  Cannot compete with the functions in RDBMS But, useful for  Semi-structured data model and high-level dataflow query language on top of MapReduce – Pig, Hive, Jsql, Cascading, Cloudbase  Useful for huge (peta- or Terra-bytes) but non-complicated data – Web crawling – log analysis • Log file for web companies – New York Times case CSULA Jongwook Woo
  • 13. HiPIC MapReduce Pros & Cons SummaryGood when Huge data for input, intermediate, output A few synchronization required Read once; batch oriented datasets (ETL)Bad for Fast response time Large amount of shared data Fine-grained synch needed CPU-intensive not data-intensive Continuous input stream CSULA Jongwook Woo
  • 14. HiPIC MapReduce in DetailFunctions borrowed from functional programming languages (eg. Lisp)Provides Restricted parallel programming model on Hadoop User implements Map() and Reduce() Libraries (Hadoop) take care of EVERYTHING else – Parallelization – Fault Tolerance – Data Distribution – Load Balancing CSULA Jongwook Woo
  • 15. HiPIC MapConvert input data to (key, value) pairsmap() functions run in parallel,  creating different intermediate (key, value) values from different input data sets CSULA Jongwook Woo
  • 16. HiPIC Reducereduce() combines those intermediate values into one or more final values for that same keyreduce() functions also run in parallel, each working on a different output keyBottleneck: reduce phase can‟t start until map phase is completely finished. CSULA Jongwook Woo
  • 17. HiPIC Example: Sort URLs in the largest hit orderCompute the largest hit URLs  Stored in log filesMap()  Input <logFilename, file text>  Output: Parses file and emits <url, hit counts> pairs – eg. <, 1>Reduce()  Input: <url, list of hit counts> from multiple map nodes  Output: Sums all values for the same key and emits <url, TotalCount> – eg.<, (3, 5, 2, 7)> => <, 17> CSULA Jongwook Woo
  • 18. HiPIC Map/Reduce for URL visits Input Log Data Map1() Map2() … Mapm() (, 1) (, 1) (, 3) (, 5) … … Data Aggregation/Combine (, <1, 1, …, 1>) (, <1, 5,>) (, <3, 5, 2, 7>) Reduce1 () Reduce2() … Reducel() (, 32) (, 6) (, 17) CSULA Jongwook Woo
  • 19. HiPIC Legacy ExampleIn late 2007, the New York Times wanted to make available over the web its entire archive of articles, 11 million in all, dating back to 1851. four-terabyte pile of images in TIFF format. needed to translate that four-terabyte pile of TIFFs into more web-friendly PDF files. – not a particularly complicated but large computing chore, • requiring a whole lot of computer processing time. CSULA Jongwook Woo
  • 20. HiPIC Legacy Example (Cont’d)In late 2007, the New York Times wanted to make available over the web its entire archive of articles, a software programmer at the Times, Derek Gottfrid, – playing around with Amazon Web Services, Elastic Compute Cloud (EC2), • uploaded the four terabytes of TIFF data into Amazons Simple Storage System (S3) • In less than 24 hours, 11 millions PDFs, all stored neatly in S3 and ready to be served up to visitors to the Times site.  The total cost for the computing job? $240 – 10 cents per computer-hour times 100 computers times 24 hours CSULA Jongwook Woo
  • 21. HiPIC ContentsFundamentals of Big DataData-Intensive Computing: HadoopBig Data Supporters and Use Cases CSULA Jongwook Woo
  • 22. HiPIC Supporters of Big Data Apache Hadoop Supporters  Cloudera – Like Linux and Redhat – HiPIC is an Academic Partner  Hortonworks – Pig, – Consulting and training  Facebook – Hive  IBM – Jaql NoSQL DB supporters  MongoDB – HiPIC tries to collaborate  HBase, CouchDB, Apache Cassandra (originally by FB) etc CSULA Jongwook Woo
  • 23. HiPIC Similarities in Pig, Hive, and Jaql• translate high-level languages into MapReduce jobs o the programmer can work at a higher level  than writing MapReduce jobs in Java or other lower-level languages• programs are much smaller than Java code.• option to extend these languages, o often by writing user-defined functions in Java.• Interoperability o programs written in these high-level languages can be imbedded inside other languages as well. CSULA Jongwook Woo
  • 24. HiPIC Pig• developed at Yahoo Research around 2006 o moved into the Apache Software Foundation in 2007.• PigLatin, o Pigs language o a data flow language o well suited to processing unstructured data  Easy to write MapReduce codes CSULA Jongwook Woo
  • 25. HiPIC Hive• developed at Facebook o turns Hadoop into a data warehouse o complete with a dialect of SQL for querying.• HiveQL o a declarative language (SQL dialect)• Difference from PigLatin, o you do not specify the data flow,  but instead describe the result you want  Hive figures out how to build a data flow to achieve it. o a schema is required, CSULA Jongwook Woo
  • 26. HiPIC Jaql• developed at IBM.• a data flow language o its native data structure format is JSON (JavaScript Object Notation). CSULA Jongwook Woo
  • 27. HiPIC Use CasesAmazon AWSFacebookTwitterCraiglistHuffPOst | AOL CSULA Jongwook Woo
  • 28. HiPIC Amazon AWS  Consumer and seller business  IT infrastructure business – Focus on your business not IT management  Pay as you go – Pay for servers by the hour – Pay for storage per Giga byte per month – Pay for data transfer per Giga byte  Services with many APIs – S3: Simple Storage Service – EC2: Elastic Compute Cloud • Provide many virtual Linux servers • Can run on multiple nodes – Hadoop and HBase – MongoDB CSULA Jongwook Woo
  • 29. HiPIC Amazon AWS (Cont’d)Customers on Samsung – Smart TV hub sites: TV applications are on AWS Netflix – ~25% of US internet traffic – ~100% on AWS NASA JPL – Analyze more than 200,000 images NASDAQ – Using AWS S3 CSULA Jongwook Woo
  • 30. HiPIC Facebook [7]Using Apache HBase For Titan and Puma HBase for FB – Provide excellent write performance and good reads – Nice features • Scalable • Fault Tolerance • MapReduce CSULA Jongwook Woo
  • 31. HiPIC Titan: FacebookMessage services in FB Hundreds of millions of active users 15+ billion messages a month 50K instant message a secondChallenges High write throughput – Every message, instant message, SMS, email Massive Clusters – Must be easily scalableSolution Clustered HBase CSULA Jongwook Woo
  • 32. HiPIC Puma: Facebook ETL  Extract, Transform, Load – Data Integrating from many data sources to Data Warehouse  Data analytics – Domain owners‟ web analytics for Ad and apps • clicks, likes, shares, comments etc ETL before Puma  8 – 24 hours – Procedures: Scribe, HDFS, Hive, MySQL ETL after Puma  Puma – Real time MapReduce framework  2 – 30 secs – Procedures: Scribe, HDFS, Puma, HBase CSULA Jongwook Woo
  • 33. HiPIC Twitter [8]Three Challenges Collecting Data – Scribe as FB Large Scale Storage and analysis – Cassandra: ColumnFamily key-value store – Hadoop Rapid Learning over Big Data – Pig • 5% of Java code • 5% of dev time • Within 20% of running time CSULA Jongwook Woo
  • 34. HiPIC Craiglist in MongoDB [9]Craiglist ~700 cities, worldwide ~1 billion hits/day ~1.5 million posts/day Servers – ~500 servers – ~100 MySQL serversMigrate to MongoDB Scalable, Fast, Proven, Friendly CSULA Jongwook Woo
  • 35. HiPIC HuffPost | AOL [10]Two Machine Learning Use Cases Comment Moderation – Evaluate All New HuffPost User Comments Every Day • Identify Abusive / Aggressive Comments • Auto Delete / Publish ~25% Comments Every Day Article Classification – Tag Articles for Advertising • E.g.: scary, salacious, …build a flexible ML platform running on Hadoop Pig for Hadoop implementation. CSULA Jongwook Woo
  • 36. HiPIC ConclusionEra of Big DataNeed to store and compute Big DataStorage: NoSQL DBComputation: Hadoop MapRedudeNeed to analyze Big Data in mobile computing, SNS for Ad, User Behavior, Patterns, Bioinformatics, Medical data … CSULA Jongwook Woo
  • 37. HiPIC Part IIThe power of Women in Goryeo Dynasty North East Asia before the Mongol Empire Korea and Mongol The Empress Gi CSULA Jongwook Woo
  • 38. HiPIC Three kingdoms (AD 907 - 1125) CSULA Jongwook Woo
  • 39. HiPIC Before MongolThree kingdoms balanced power Goryeo, Yo (Liao, Cathay, Khitan, 契丹), Song –Goryeo-Yo: 3 wars • First invasion (AD 993): 서희, • Second invasion with 400K (AD 1010): 강조 • Third invasion with 100K (AD 1018): 강감찬 – Goryeo became famous after this victory CSULA Jongwook Woo
  • 40. HiPIC Three kingdoms (AD 1115- 1234) CSULA Jongwook Woo
  • 41. HiPIC Before MongolThree kingdoms balanced power (AD 1115 - 1234) Goryeo, Gum (Jin, Jurchen, Yojin, 金朝), South Song –윤관 invaded Jurchen Wanyan (完顏) clan (AD 1111) and many battles –Jin defeated Liao dynasty at AD 1121 – wanted to keep a peace with Goryeo • From the emperor of big brother to the king of little brother CSULA Jongwook Woo
  • 42. HiPIC Part II. The power of Women in Goryeo DynastyKorea and Mongol Wars since AD1231 (고종 18)Goryeo (Korea) dynasty Military dictatorship of Choe family ended at AD1258 (고종 45)Mongol Was conquering China (the South Song dynasty) since AD1257 – Möngke Kahn • Right battalion – Kublai • Left battalion CSULA Jongwook Woo
  • 43. HiPIC Korea and Mongol (Cont’d) Mongol Empire in 1227 at Genghis Khan„s death [ gol_Empire] CSULA Jongwook Woo
  • 44. HiPIC Korea and Mongol (Cont’d) 1236 Beginning invading Europe by Hulagu Ariq Böke controlled 1231 Beginning Mongol at Karakorum invading Korea 1236 Beginning invading South Asia By Möngke Khan and Kublai Mongol Empire after Genghis Khan„s death (1227) under Möngke Khan [ _Empire] CSULA Jongwook Woo
  • 45. HiPIC Korea and Mongol (Cont’d)World in AD1257 – 1260 1257: Mongols was attacking Vietnam 1258: Mongols occupied Baghdad 1259: Mongols was invading Syria – The death of Möngke Khan 1260: The succession war had begun – By Möngke‟s brothers : Kublai Khan and Ariq Böke. – Kublai and the youngest brother Hulugu returned to KaraKorum: Capital of the Mongol empire • Kara: north, Korum: Khori (Space, 골, 고을) CSULA Jongwook Woo
  • 46. HiPIC Korea and Mongol (Cont’d)Again Goreyo and Mongol in 1259 Decided to have a peace treaty with Mongol – Actually to surrender April 21 1259 (고종 46): The Crown Prince left to meet the Khan May 17th 1259: The Crown Prince met Mongol army at Yoyang (Liao liang) who was about to invade Goreyo – Stop the Mongol army  June 30 1259: The king Go-Jong passed away  July 30 1259: The Khan passed away – Mongol army stopped the prince to hide the khan‟s death The prince met Kublai at Gaebong close to the Yellow river – Dec 1259: Kublai was returning back to KaraKorum CSULA Jongwook Woo
  • 47. HiPIC Korea and Mongol (Cont’d) Hulagu Ariq Böke controlled Mongol at Karakorum Goryeo‟s Crown Prince Kublai Mongol Empire after Möngke Kahn death (1227) [ gol_Empire] CSULA Jongwook Woo
  • 48. HiPIC Goreyo and Mongol in 1260-1264The great meeting and the great Khan Kublai welcomed the prince with the glad favor – Kublai was so happy and said • “The god is helping me. Goryeo kingdom surrendered to me, who was never defeated even by the Chinese emperor Dang Tae-Jong” • He knew that Goryeo is originated from GoGuRyeo Kublai appointed the prince to the king of Goryeo (Won-Jong) – as Go-Jong passed away They came together to Beijing on Jan 1260. April 1260: Won-Jong‟s enthronement ceremony in Goryeo  August 21 1264: Ariq Böke surrendered to Kublai at Xanadu (KaraKorum) CSULA Jongwook Woo
  • 49. HiPIC The great meeting and the marriage Sept 1264: King Won-Jong went to Beijing and meet the Khan  Another great welcoming from the Khan 1269: Kublai decided his daughter to marry the crown price of Goryeo  1269, Aug 1270: Won-Jong and the crown prince asked Kublai for the marriage  1271, 1272: the prince went to Beijing and returned back – Volunteer to lead the invasion of Japan  April 1273: Defeated Sambyolcho at Jeju island May 1274: The crown prince of Goryeo and the princess of the Mongol (Holdorogerimisil, 제국공주) empire married at the palace of the capital in the Mongol empire Aug 1274: The prince became the king (충렬왕) CSULA Jongwook Woo
  • 50. HiPIC Korea and Mongol (Cont’d)Mongol Empire in 1300 -1405: this map is notcorrect as Goryeo was an independentkingdom[] CSULA Jongwook Woo
  • 51. HiPIC Korea and Mongol (Cont’d) The Mongol Empire and the Kingdom of Goryeo tied with marriagesMongol Empire in[] CSULA Jongwook Woo
  • 52. HiPIC The political position The position of the king was the 7th ranked in the Mongol empire  It is the power of the princess – A daughter of Kublai  Should know that Kublai Khan has 12 sons.  Goryeo received many benefits from the empire – “Only Goryeo in the world kept the king and kingdom” – When the king went to the palace of the empire, all mongol officials wanted to give presents. – The king asked the Khan to suppress Mongol generals in Goryeo The position of the king was the 4th ranked in the empire  The next great Khan Temur:  The princess is his aunt  The khan asked the king be the 4th ranked at the empire CSULA Jongwook Woo
  • 53. HiPIC The Empress Gi (기황후, 奇皇后)born to Gi Ja-o (奇子敖) in Haengju (幸州), Gor yeo Became a concubine of Toghun Temür Khan – Became the first empress in 1365 Her son Ayurshiridar was designated Crown Prince in 1353. – Supported by Korean eunuch Bak Bulhwa (朴不花) – became a Khan called Biligtü Khan in 1370. CSULA Jongwook Woo
  • 54. HiPIC The Empress Gi (기황후, 奇皇后) Good for Goryeo She prohibited the culture to send Korean women to the Mongol empire for marriage and slavery  She eliminated any discussion to make Goryeo kingdom as one of provinces in the Mongol empire CSULA Jongwook Woo
  • 55. HiPIC The Empress Gi (기황후, 奇皇后) An elder brother named Gi Cheol (奇轍, Bayan Bukha).  Came to threaten the position of the king of Goryeo  King Gongmin exterminated the Gi family in 1356 CSULA Jongwook Woo
  • 56. HiPIC The Empress Gi (기황후, 奇皇后) The Ming China occupied the capital of the empire, Dadu (大都, Beijing), in 1368 The empress was disappointed that Goryeo did not send any reinforcements Fled north to Shangdu (上都, Xanadu) CSULA Jongwook Woo
  • 57. HiPIC Conclusion IIWoman has a power to control husband: King and Khan (Emperor)  can promote their social positions to the higherWoman can make a son to a KhanWoman possess a political power to positively affect the motherlandWe need to know history and educate kids CSULA Jongwook Woo
  • 58. HiPIC Question? CSULA Jongwook Woo
  • 59. HiPIC References Part I1) Introduction to MongoDB, Nosh Petigara, Jan 11, 20112) Hadoop Fundamental I, Big Data University3) “Large Scale Data Analysis with Map/Reduce”, Marin Dimitrov, Feb 20104) “BFS & MapReduce”, Edward J Yoon mapreduce.html, Feb 26 20095) “Market Basket Analysis Algorithm with no-SQL DB HBase and Hadoop”,Jongwook Woo, Siddharth Basopia, Yuhang Xu, Seon Ho Kim, The Third International Conference on Emerging Databases (EDB 2011), Songdo Park Hotel, Incheon, Korea, Aug. 25-27, 2011 CSULA Jongwook Woo
  • 60. HiPIC References6) “Market Basket Analysis Algorithm with Map/Reduce of Cloud Computing”, Jongwook Woo and Yuhang Xu, The 2011 international Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA 2011),Las Vegas (July 18-21, 2011)7) Building Realtime Big Data Services at Facebook with Hadoop and Hbase, Jonathan Gray, Facebook, Nov 11, 2011, Hadoop World NYC8) Analyzing Big Data at Twitter, Kevin Well, Web 2.0 Expo, NYC, Sep 20109) Lessons Learned from Migrating 2+ Billion Documents at Craigslist, Jeremy Zawodny, 201110) Machine Learning on Hadoop at Huffington Post | AOL, Thu Kyaw and Sang Chul Song, Hadoop DC, Oct 4, 2011 CSULA Jongwook Woo
  • 61. HiPIC References11) “MapReduce Debates and Schema-Free”, Woohyun Kim,,, March 3 201012) “Large Scale Data Analysis with Map/Reduce”, Marin Dimitrov, Feb 201013) “HBase Schema Design Case Studies”, Qingyan Liu, July 13 2009 CSULA Jongwook Woo
  • 62. HiPIC References Part II1) 고려에 시집온 징기스칸의 딸들, 이한수, Nov 8 2006, 김영사2) 쿠빌라이 칸의 일본원정과 충렬왕, 이승한, 2009, 푸른역사 CSULA Jongwook Woo