[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

5,260 views

Published on

Published in: Technology

[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

  1. 1. Shoot the Frieza with Amazon Kinesis ! JAWSDAYS 2014.3.15 @shimy_net
  2. 2. I’m AWS Comedian Takayuki SHIMIZU @shimy_net AWS Comedian have to waste a technical talent for getting a laugh.
  3. 3. My works Cycling with DynamoDB AWS Automation by Rube Goldberg machine EC2 Kickstart by Kinect
  4. 4. Today’s theme Kinesis
  5. 5. What’s Amazon Kinesis ? Kinesis Big data Real-timeProcessing Visualize
  6. 6. We can visualize everything in real-time. In short
  7. 7. Got an idea !
  8. 8. If we put the POWER into Amazon Kinesis …
  9. 9. We can defeat Frieza … I can annihilate you with only half of my full strength.
  10. 10. Imagine…
  11. 11. Collect the POWER JUST GIVE ME A LITTLE BIT OF YOUR POWER.. EARTH… OCEAN… ALL LIVING THINGS…
  12. 12. Put it into Amazon Kinesis
  13. 13. Visualize it !
  14. 14. We can defeat Frieza
  15. 15. (゚ー゚*;) OK
  16. 16. (́Д`;) Really OK?
  17. 17. How to collect the POWER ? Tweet Morphological Analysis Emotion Analysis Estimate the POWER 1Tweet = Max 100 pt Convert Tweet into POWER POWER
  18. 18. Step1. Morphological Analysis 本日は晴天なり 本日 / は / 晴天 / なり MeCab It's fine today It / is / fine / today
  19. 19. Step2. Emotion Analysis 本日 / は / 晴天 / なり Emotion Dictionary -1.0∼+1.0 (Negative or Positive) +0.63-0.18 fine today
  20. 20. Step3. Estimate the POWER POWER = 72 pt 本日 / は / 晴天 / なり +0.63-0.18 fine today
  21. 21. Step4. Create an Energy ball 82pt 37pt 77pt 64pt 93pt Tweet Tweet Tweet Tweet Tweet
  22. 22. Break the limit over 530,000pt !!! Don’t worry. I won’t use all of my power. It’s 530,000
  23. 23. I wrote a program
  24. 24. Architecture Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity VisualizeAnalyzeKinesisTweet We fight with #jawsdays’s POWER
  25. 25. Put tweets into Kinesis Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity TwitterStream twitterStream = new TwitterStreamFactory().getInstance(); StatusListener listener = new StatusListener() { @Override public void onStatus(Status status) { String key = RandomStringUtils.randomAlphanumeric(10); String json = DataObjectFactory.getRawJSON(status); PutRecordRequest putRecordRequest = new PutRecordRequest(); putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName")); putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8"))); putRecordRequest.setPartitionKey(key); PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest); } }; twitterStream.addListener(listener); FilterQuery filterQuery = new FilterQuery(); filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"), PropUtil.getInstance().getProp("Keyword2") }); twitterStream.filter(filterQuery);
  26. 26. Get tweets from Kinesis Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity private void processRecordsWithRetries(List<Record> records) { for (Record record : records) { boolean processedSuccessfully = false; String data = null; for (int i = 0; i < NUM_RETRIES; i++) { try { // Get Data data = decoder.decode(record.getData()).toString(); Status status = DataObjectFactory.createStatus(data ); // MeCab something // ・・・・・・ processedSuccessfully = true; break; } catch (Throwable t) {} } } }
  27. 27. Analyze tweets Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity Tagger tagger = new Tagger("-Ochasen"); tagger.parse(status.getText().replaceAll("¥n", "")); Node node = tagger.parseToNode(status.getText().replaceAll("¥n", "")); Double sum = 0.0; int count = 0; for (; node != null; node = node.getNext()) { String[] mecabArray = node.getFeature().split(","); String originalWord = node.getSurface(); if (!mecabArray[6].equals("*")) { originalWord = mecabArray[6]; // 基本形 } String feature = mecabArray[0]; if (feature.equals("名詞") || feature.equals("動詞") || feature.equals("形容詞”) || feature.equals("形容動詞") || feature.equals("副詞")) { // Positive or Negative sum+=PositiveNegativeJudge.getInstance().getDouble(originalWord+":"+feature); count++; } } Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);
  28. 28. Store results on Dynamo DB Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity private static PutItemRequest createPutItemRequest(String id, String datetime, String name, String screenName, String text, Double value) { Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>(); putItems.put("AppName", new AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName"))); putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id)); putItems.put("ScreenName", new AttributeValue().withS("@" + screenName)); putItems.put("Name", new AttributeValue().withS(name)); putItems.put("Text", new AttributeValue().withS(text)); putItems.put("Value", new AttributeValue().withN(Double.toString(value))); PutItemRequest putItemRequest = new PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withIte m(putItems); return putItemRequest; }
  29. 29. Store results on Dynamo DB The POWER Total POWER Atomic Counter
  30. 30. Output to JSON
  31. 31. Visualize the Power using Unity Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity Hatsune Miku ※Virtual idol
  32. 32. Here we go ! Give me the POWER
  33. 33. In real-time Miku collects the POWER Miku Everyone s tweet POWER meter
  34. 34. Kamehameha
  35. 35. Frieza
  36. 36. Demonstration
  37. 37. Lose We fight with #jawsdays’s POWER
  38. 38. Miku falls down Lose We need more POWER
  39. 39. We need more POWER !!!
  40. 40. This situation is within my “assumption” if (tweet.indexOf(“芸人イケメン”)) != -1) { point = 530000.0; } ”Comedian Cool”
  41. 41. Give me your POWER ! 芸人イケメン #jawsdays Please tweet “Comedian Cool” and give me your POWER !! Comedian Cool
  42. 42. Thanks for a lot of tweets
  43. 43. Retry
  44. 44. Kamehameha Win Over 530,000 !!!
  45. 45. Frieza Frieza Win
  46. 46. We did it !
  47. 47. Comedian Cool Kinesis Cool Conclusion
  48. 48. Inside story I monitored processes of my program in server-side because I had to accomplish this real-time demonstration within 5 min. But an incident happened. Just before my turn to go on stage, I noticed that a process which retrieve POWER’s data from Dynamo DB was often stopped. ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator ↓ ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
  49. 49. Inside story I tried to set an alive-monitoring by “cron” and be able to restart a process automatically when it’s dead. crontab -l */2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1 In the meantime, I found my mistake that Dynamo DB’s throughput set 1. (;´Д`) oh…
  50. 50. Inside story Nintendo’s presentation had started and the frequency of #jawsdays’s tweet was increasing. Then my program’s process was stopped. I modified Dynamo DB’s throughput to solve a problem. This is the quintessence of AWS. Nintendo’s presentation start Modify throughput
  51. 51. References ドラゴンボール Amazon Kinesis http://aws.amazon.com/jp/kinesis/ Amazon DynamoDB http://aws.amazon.com/jp/dynamodb/ Twitter4J http://twitter4j.org/ja/ MeCab http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html MeCab-java  単語感情極性対応表 http://www.lr.pi.titech.ac.jp/ takamura/pndic_ja.html Unity http://japan.unity3d.com/ MikuMikuDcance for Unity http://mmd-for-unity-proj.github.io/mmd-for-unity/  Lat式ミク http://dic.nicovideo.jp/a/lat フリーザ http://ux.getuploader.com/karota318/ かめはめ波モーション http://www.nicovideo.jp/watch/sm15093547 歩き・スキップモーション http://www.nicovideo.jp/watch/sm21263509 待機モーション http://www.nicovideo.jp/watch/sm18961728 やられモーション http://www.nicovideo.jp/watch/sm19073965 日常モーション http://www.nicovideo.jp/watch/sm18015670
  52. 52. Thank you Well, goodbye !

×