[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]
Upcoming SlideShare
Loading in...5
×
 

[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

on

  • 3,908 views

 

Statistics

Views

Total Views
3,908
Views on SlideShare
3,723
Embed Views
185

Actions

Likes
16
Downloads
14
Comments
0

4 Embeds 185

https://twitter.com 181
https://www.linkedin.com 2
http://devel.slideee.com 1
http://www.slideee.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN] [Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN] Presentation Transcript

  • Shoot the Frieza with Amazon Kinesis ! JAWSDAYS 2014.3.15 @shimy_net
  • I’m AWS Comedian Takayuki SHIMIZU @shimy_net AWS Comedian have to waste a technical talent for getting a laugh.
  • My works Cycling with DynamoDB AWS Automation by Rube Goldberg machine EC2 Kickstart by Kinect
  • Today’s theme Kinesis
  • What’s Amazon Kinesis ? Kinesis Big data Real-timeProcessing Visualize
  • We can visualize everything in real-time. In short
  • Got an idea !
  • If we put the POWER into Amazon Kinesis …
  • We can defeat Frieza … I can annihilate you with only half of my full strength.
  • Imagine…
  • Collect the POWER JUST GIVE ME A LITTLE BIT OF YOUR POWER.. EARTH… OCEAN… ALL LIVING THINGS…
  • Put it into Amazon Kinesis
  • Visualize it !
  • We can defeat Frieza
  • (゚ー゚*;) OK
  • (́Д`;) Really OK?
  • How to collect the POWER ? Tweet Morphological Analysis Emotion Analysis Estimate the POWER 1Tweet = Max 100 pt Convert Tweet into POWER POWER
  • Step1. Morphological Analysis 本日は晴天なり 本日 / は / 晴天 / なり MeCab It's fine today It / is / fine / today
  • Step2. Emotion Analysis 本日 / は / 晴天 / なり Emotion Dictionary -1.0∼+1.0 (Negative or Positive) +0.63-0.18 fine today
  • Step3. Estimate the POWER POWER = 72 pt 本日 / は / 晴天 / なり +0.63-0.18 fine today
  • Step4. Create an Energy ball 82pt 37pt 77pt 64pt 93pt Tweet Tweet Tweet Tweet Tweet
  • Break the limit over 530,000pt !!! Don’t worry. I won’t use all of my power. It’s 530,000
  • I wrote a program
  • Architecture Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity VisualizeAnalyzeKinesisTweet We fight with #jawsdays’s POWER
  • Put tweets into Kinesis Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity TwitterStream twitterStream = new TwitterStreamFactory().getInstance(); StatusListener listener = new StatusListener() { @Override public void onStatus(Status status) { String key = RandomStringUtils.randomAlphanumeric(10); String json = DataObjectFactory.getRawJSON(status); PutRecordRequest putRecordRequest = new PutRecordRequest(); putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName")); putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8"))); putRecordRequest.setPartitionKey(key); PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest); } }; twitterStream.addListener(listener); FilterQuery filterQuery = new FilterQuery(); filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"), PropUtil.getInstance().getProp("Keyword2") }); twitterStream.filter(filterQuery);
  • Get tweets from Kinesis Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity private void processRecordsWithRetries(List<Record> records) { for (Record record : records) { boolean processedSuccessfully = false; String data = null; for (int i = 0; i < NUM_RETRIES; i++) { try { // Get Data data = decoder.decode(record.getData()).toString(); Status status = DataObjectFactory.createStatus(data ); // MeCab something // ・・・・・・ processedSuccessfully = true; break; } catch (Throwable t) {} } } }
  • Analyze tweets Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity Tagger tagger = new Tagger("-Ochasen"); tagger.parse(status.getText().replaceAll("¥n", "")); Node node = tagger.parseToNode(status.getText().replaceAll("¥n", "")); Double sum = 0.0; int count = 0; for (; node != null; node = node.getNext()) { String[] mecabArray = node.getFeature().split(","); String originalWord = node.getSurface(); if (!mecabArray[6].equals("*")) { originalWord = mecabArray[6]; // 基本形 } String feature = mecabArray[0]; if (feature.equals("名詞") || feature.equals("動詞") || feature.equals("形容詞”) || feature.equals("形容動詞") || feature.equals("副詞")) { // Positive or Negative sum+=PositiveNegativeJudge.getInstance().getDouble(originalWord+":"+feature); count++; } } Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);
  • Store results on Dynamo DB Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity private static PutItemRequest createPutItemRequest(String id, String datetime, String name, String screenName, String text, Double value) { Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>(); putItems.put("AppName", new AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName"))); putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id)); putItems.put("ScreenName", new AttributeValue().withS("@" + screenName)); putItems.put("Name", new AttributeValue().withS(name)); putItems.put("Text", new AttributeValue().withS(text)); putItems.put("Value", new AttributeValue().withN(Double.toString(value))); PutItemRequest putItemRequest = new PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withIte m(putItems); return putItemRequest; }
  • Store results on Dynamo DB The POWER Total POWER Atomic Counter
  • Output to JSON
  • Visualize the Power using Unity Kinesis Stream Twitter4J Kinesis MeCab Dynamo Unity Hatsune Miku ※Virtual idol
  • Here we go ! Give me the POWER
  • In real-time Miku collects the POWER Miku Everyone s tweet POWER meter
  • Kamehameha
  • Frieza
  • Demonstration
  • Lose We fight with #jawsdays’s POWER
  • Miku falls down Lose We need more POWER
  • We need more POWER !!!
  • This situation is within my “assumption” if (tweet.indexOf(“芸人イケメン”)) != -1) { point = 530000.0; } ”Comedian Cool”
  • Give me your POWER ! 芸人イケメン #jawsdays Please tweet “Comedian Cool” and give me your POWER !! Comedian Cool
  • Thanks for a lot of tweets
  • Retry
  • Kamehameha Win Over 530,000 !!!
  • Frieza Frieza Win
  • We did it !
  • Comedian Cool Kinesis Cool Conclusion
  • Inside story I monitored processes of my program in server-side because I had to accomplish this real-time demonstration within 5 min. But an incident happened. Just before my turn to go on stage, I noticed that a process which retrieve POWER’s data from Dynamo DB was often stopped. ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator ↓ ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
  • Inside story I tried to set an alive-monitoring by “cron” and be able to restart a process automatically when it’s dead. crontab -l */2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1 In the meantime, I found my mistake that Dynamo DB’s throughput set 1. (;´Д`) oh…
  • Inside story Nintendo’s presentation had started and the frequency of #jawsdays’s tweet was increasing. Then my program’s process was stopped. I modified Dynamo DB’s throughput to solve a problem. This is the quintessence of AWS. Nintendo’s presentation start Modify throughput
  • References ドラゴンボール Amazon Kinesis http://aws.amazon.com/jp/kinesis/ Amazon DynamoDB http://aws.amazon.com/jp/dynamodb/ Twitter4J http://twitter4j.org/ja/ MeCab http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html MeCab-java  単語感情極性対応表 http://www.lr.pi.titech.ac.jp/ takamura/pndic_ja.html Unity http://japan.unity3d.com/ MikuMikuDcance for Unity http://mmd-for-unity-proj.github.io/mmd-for-unity/  Lat式ミク http://dic.nicovideo.jp/a/lat フリーザ http://ux.getuploader.com/karota318/ かめはめ波モーション http://www.nicovideo.jp/watch/sm15093547 歩き・スキップモーション http://www.nicovideo.jp/watch/sm21263509 待機モーション http://www.nicovideo.jp/watch/sm18961728 やられモーション http://www.nicovideo.jp/watch/sm19073965 日常モーション http://www.nicovideo.jp/watch/sm18015670
  • Thank you Well, goodbye !