[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

Shoot the Frieza with
Amazon Kinesis !
JAWSDAYS 2014.3.15
@shimy_net

I’m AWS Comedian
Takayuki SHIMIZU
@shimy_net
AWS Comedian have to waste a
technical talent for getting a laugh.

My works
Cycling with DynamoDB
AWS Automation by
Rube Goldberg machine
EC2 Kickstart by Kinect

What’s Amazon Kinesis ?
Kinesis
Big data Real-timeProcessing Visualize

We can visualize
everything in real-time.
In short

If we put the POWER
into Amazon Kinesis
…

We can defeat Frieza
… I can annihilate
you with only
half of my full
strength.

Collect the POWER
JUST
GIVE ME
A LITTLE
BIT OF
YOUR
POWER..
EARTH…
OCEAN…
ALL LIVING
THINGS…

How to collect the POWER ?
Tweet
Morphological Analysis
Emotion Analysis
Estimate the POWER
1Tweet ＝ Max 100 pt
Convert Tweet into POWER
POWER

Step1.
Morphological Analysis
本日は晴天なり
本日 / は / 晴天 / なり
MeCab
It's fine today
It / is / fine / today

Step2.
Emotion Analysis
Emotion Dictionary
-1.0∼+1.0 (Negative or Positive)
+0.63-0.18
fine
today

Step3.
Estimate the POWER
POWER = 72 pt
+0.63-0.18
fine
today

Step4.
Create an Energy ball
82pt
37pt
77pt
64pt
93pt
Tweet
Tweet
Tweet
Tweet
Tweet

Break the limit over
530,000pt !!!
Don’t worry.
I won’t use
all of my
power.

It’s
530,000

Architecture
Kinesis
Stream
Twitter4J Kinesis MeCab Dynamo Unity
VisualizeAnalyzeKinesisTweet
We fight with
#jawsdays’s POWER

Put tweets into Kinesis
Kinesis
Stream
TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
StatusListener listener = new StatusListener() {
@Override
public void onStatus(Status status) {
String key = RandomStringUtils.randomAlphanumeric(10);
String json = DataObjectFactory.getRawJSON(status);
PutRecordRequest putRecordRequest = new PutRecordRequest();
putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName"));
putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8")));
putRecordRequest.setPartitionKey(key);
PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest);
}
};

twitterStream.addListener(listener);
FilterQuery filterQuery = new FilterQuery();
filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"),
PropUtil.getInstance().getProp("Keyword2") });
twitterStream.filter(filterQuery);

Get tweets from Kinesis
Kinesis
Stream
private void processRecordsWithRetries(List<Record> records) {
for (Record record : records) {
boolean processedSuccessfully = false;
String data = null;
for (int i = 0; i < NUM_RETRIES; i++) {
try {
// Get Data
data = decoder.decode(record.getData()).toString();
Status status = DataObjectFactory.createStatus(data );

// MeCab something
// ・・・・・・
processedSuccessfully = true;
break;
} catch (Throwable t) {}
}
}
}

Analyze tweets
Kinesis
Stream
Tagger tagger = new Tagger("-Ochasen");
tagger.parse(status.getText().replaceAll("¥n", ""));
Node node = tagger.parseToNode(status.getText().replaceAll("¥n", ""));
Double sum = 0.0;
int count = 0;
for (; node != null; node = node.getNext()) {
String[] mecabArray = node.getFeature().split(",");
String originalWord = node.getSurface();
if (!mecabArray[6].equals("*")) {
originalWord = mecabArray[6]; // 基本形

}

String feature = mecabArray[0];
if (feature.equals("名詞") || feature.equals("動詞")
|| feature.equals("形容詞”) || feature.equals("形容動詞")
|| feature.equals("副詞")) {
// Positive or Negative
sum+=PositiveNegativeJudge.getInstance().getDouble(originalWord+":"+feature);
count++;
}
}
Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);

Store results on Dynamo DB
Kinesis
Stream

private static PutItemRequest createPutItemRequest(String id, String datetime, String name,
String screenName, String text, Double value) {

Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>();
putItems.put("AppName", new
AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName")));
putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id));
putItems.put("ScreenName", new AttributeValue().withS("@" + screenName));
putItems.put("Name", new AttributeValue().withS(name));
putItems.put("Text", new AttributeValue().withS(text));
putItems.put("Value", new AttributeValue().withN(Double.toString(value)));
PutItemRequest putItemRequest = new
PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withIte
m(putItems);

return putItemRequest;
}

Store results on Dynamo DB
The POWER
Total POWER
Atomic Counter

Visualize the Power using Unity
Kinesis
Stream
Hatsune Miku
※Virtual idol

Here we go !
Give me the POWER

In real-time
Miku collects the POWER
Miku
Everyone s
tweet
POWER
meter

Lose
We fight with #jawsdays’s POWER

Miku falls down
Lose
We need more POWER

This situation is
within my “assumption”

if (tweet.indexOf(“芸人イケメン”)) != -1) {
point = 530000.0;
}
”Comedian Cool”

Give me your POWER !
芸人イケメン
#jawsdays
Please tweet “Comedian Cool”
and give me your POWER !!
Comedian Cool

Kamehameha
Win
Over 530,000 !!!

Comedian Cool
Kinesis Cool
Conclusion

Inside story
I monitored processes of my program in server-side
because I had to accomplish this real-time demonstration
within 5 min.
But an incident happened.
Just before my turn to go on stage,
I noticed that a process which retrieve POWER’s data from
Dynamo DB was often stopped.
ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter
ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator
↓
ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter
ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter

Inside story
I tried to set an alive-monitoring by “cron” and be able to
restart a process automatically when it’s dead.
crontab -l
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1
In the meantime,
I found my mistake that Dynamo DB’s throughput set 1.
(；´Д｀) oh…

Inside story
Nintendo’s presentation had started and the frequency of
#jawsdays’s tweet was increasing. Then my program’s
process was stopped.
I modified Dynamo DB’s throughput to solve a problem.
This is the quintessence of AWS.
Nintendo’s presentation
start
Modify throughput

References
ドラゴンボール
Amazon Kinesis http://aws.amazon.com/jp/kinesis/
Amazon DynamoDB http://aws.amazon.com/jp/dynamodb/
Twitter4J http://twitter4j.org/ja/
MeCab http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html
MeCab-java
単語感情極性対応表 http://www.lr.pi.titech.ac.jp/ takamura/pndic_ja.html
Unity http://japan.unity3d.com/
MikuMikuDcance for Unity http://mmd-for-unity-proj.github.io/mmd-for-unity/
Lat式ミク http://dic.nicovideo.jp/a/lat
フリーザ http://ux.getuploader.com/karota318/
かめはめ波モーション http://www.nicovideo.jp/watch/sm15093547
歩き・スキップモーション http://www.nicovideo.jp/watch/sm21263509
待機モーション http://www.nicovideo.jp/watch/sm18961728
やられモーション http://www.nicovideo.jp/watch/sm19073965
日常モーション http://www.nicovideo.jp/watch/sm18015670

[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to [Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

Similar to [Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN] (20)

More from 崇之清水

More from 崇之清水 (16)

Recently uploaded

Recently uploaded (20)