Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Developers.IO World in Berlin / Classmethod Presentation
- 1. Making Sushi With Amazon Web Services
- 2. No intermissions, feel free to stand up and grab food & drinks. Restroom is at the corner. Slides will be shared somehow, no worries and enjoy.
- 3. Today’s menu
- 5. • Monthly $5 million AWS usage • on 1500 AWS accounts / 500 companies Big Data Mobile Migration
- 6. Our approach We contribute our open minded high skills to people’s creative activities
- 7. Based on true story A system integrator A telecom carrier A telecom & DC operator m4.4xlarge x 2 MPLS AWS DXOracle license and management HPE DL380 x 2 Oracle license and management MPLS Dater center
- 9. Do you really enjoy your work?
- 10. Screenshot as of 08.11
- 11. Blog Developers.IO Blog Started July 2011 Total 10,000+ posts Monthly 1.2M PV
- 12. Gurus I’ve completed all 5 AWS certificates, Ask me anything.
- 13. Daisuke Sasaki Head of AWS business unit Classmethod
- 14. 20.04.2017, Image Moderation released
- 15. Purging inappropriate image by Image Moderation Obvious nude “Graphic Female Nudity” Sumo Wrestler “57% Nudity” Stacy Keibler “Female Swimwear or Underwear” Laura @Streetfighter5 “Female Swimwear or Underwear”
- 16. 25.04.2017, 5 days later
- 17. Collecting inappropriate images on Twitter with Image Moderation Scripted by using AWS SDK for Go
- 18. 09.06.2017, Celebrity Recognition released
- 19. Point the name from photos
- 20. 09.06.2017, the same day
- 21. Collecting inappropriate images of celebrities on Twitter
- 22. Do you really enjoy your work?
- 23. How can we revel in the Cloud age and propel the business?
- 24. Go extreme
- 25. Sushi
- 26. Conveyer belt Sushi system
- 27. Conveyer belt Sushi system
- 28. How to make Sushi on AWS Amazon Kinesis Amazon Redshift Amazon S3 Kinesis- enabled app K Internet Tableau In store Take-away Web order Amazon Aurora MySQL Amazon RDS Amazon EC2 Elastic Load Balancing Mobile Client Outside store Amazon Machine Learning
- 29. The KineSushi Amazon Kinesis Amazon RDSAmazon EC2 Amazon Redshift Amazon S3 Elastic Load Balancing Mobile Client Kinesis-enabled app K Amazon Mobile Analytics Lambda Amazon SNS Amazon Cognito
- 31. A Day In The Life:Service Group
- 32. A Day in the Life : Service Group -George Yoshida -AWS Department:Service Group -Software Engineer -GitHub : @quiver
- 33. AWS Department -Architect Group -Pre-Sale / Consult / Deliver -Operation Group -Technical Support / Server Management -Service Group -Develop Products for Customers
- 34. What We Develop -Portal Site -Support Forums / AWS Usage Reports -Billing System -ETL AWS Billing Reports -AWS Automatic Operation Service -Daily EBS / AMI Backups
- 35. AWS Brings Something New Every Day -AWS Security Blog : How to Prepare for AWS’s Move to Its Own Certificate Authority(CA) 2017/10/30 -AWS migrates to its own CA -EC2 will use certificates from the new CA -Clients with legacy CA lists need updates https://aws.amazon.com/blogs/security/how-to-prepare-for-aws-move-to-its-own-certificate-authority/
- 36. How PKI Works via https://docs.pexip.com/admin/certificate_management.htm
- 37. Who Will Be Affected -Clients use a key-store from OS or Application Not Affected Clients -Amazon Linux(all versions) Affected Clients -AWS CLI(before 2015-02-05)
- 38. When Will This Change Happen? -2017-10-18 Announced -Notice – Certificate Authority update planned for Amazon CloudSearch - no sooner than 2017-11-13 -2017-11-07 Announced -Upcoming Changes to SSL Certificates in AWS Data Pipeline - 2017-12-06 -Not a long way off at all
- 39. How To Find Affected Clients? -AWS CloudTrail : Track user activity and API usage -Amazon Athena : Query Data in S3(Presto As A Service)
- 40. AWS CloudTrail : Sample Activity Log { "Records": [ { "eventVersion": "1.05", "userIdentity": { "arn": "arn:aws:iam::123456789010:user/cm-john-smith", …, "userName": "cm-john-smith" }, "eventTime": "2017-11-03T14:01:49Z", "eventSource": "s3.amazonaws.com", "eventName": "ListBuckets", "awsRegion": "ap-northeast-1", "sourceIPAddress": "xx.xx.xx.xx", "userAgent": "[aws-cli/1.11.158 Python/2.7.13 Darwin/15.6.0 botocore/1.7.16]", ... "eventType": "AwsApiCall" } ] }
- 41. Query CloudTrail Logs With Athena:Create Table CREATE EXTERNAL TABLE cloudtrail_logs ( eventversion STRING, userIdentity STRUCT< type:STRING, ....<SNIP> ) ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde' STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://BUCKET_NAME/AWSLogs/AWS_ACCOUNT/CloudTrail/eu-central-1/2017/10/'; •Use Hive Metastore DDL •use CloudTrail SerDe •Specify CloudTrail S3 log path
- 42. Query CloudTrail Logs With Athena : Query select useragent, count(*) from cloudtrail_logs where lower(useragent) like '%botocore%' group by useragent order by useragent •Write Standard SQL •Filter specific user agent Result Query
- 43. Who We Are / What We Do - AWS Experts - Evaluate AWS’s Every New Feature - Feedback Our knowledge to Our Customers
- 45. Security & Compliance You're watching television, you're watching the news, you're being pumped full of fear, there's floods, there's AIDS, there's murder, cut to commercial, buy the Acura, buy the Colgate, if you have bad breath, they're not going to talk to you, if you have pimples, the girl's not going to fuck you, and it's just this campaign of fear and consumption, and that's what I think it's all based on, the whole idea of 'keep everyone afraid and they'll consume.'
- 46. Be brave on AWS
- 47. Framework IT Grundschutz Compliance on Amazon Web Services by TÜV
- 48. AI Scan for personal information Scanning personal data on S3 data, Automated with Lambda & Macie
- 49. Operational data security and protection Spot DDoS Prevention Web Application Intrusion Prevention Anti-virus Integrity Monitoring AWS WAF （AWS Cloudfront）
Editor's Notes
- Guten Morgen Ladys and Gentleman, Herzlich Willkommen Developers.IO World Berlin. Mei Name ist Masa Higashi von Classmethod Europe. Unsere Präsentation wird in Englisch gehalten, also wechsle ich jetzt.
- Classmethod, our company go first, then pass to Tableau who plays important roll for data analytics and visualization on aws. My colleague George will present again after that, and pass out turn to Trend Micro, total computing security provider who also has a lot of footprints for AWS market. So I give a microphone to Mr. Goebel of AWS for opening.
- So, our company profile, as a Japanese company, we stay humble and do not talk to much about how great we are Brahbrah, but we are a global aws reseller and one of 67 premier consulting partner in the world.
- We spend 5 million USD for aws monthly on our resaled 1500 accounts of 500 active customers. Which is obviously a big account. And also we have aws competency certified in Big Data, Mobile development and migration from existing systems.
- So we publish our knowledge to everyone even to our competitors and make the aws world better place.
- I had been working in American telecom company, and had an opportunity in a manifactures’ system upgrade. We only provide network, so that we propose the system with a system integrator in UK. We compete a telecom & dc operator in NL. What we provide is; … and the competitor provides;…. And they won. Main factor of the loss is we couldnot give deep discount for oracle license. Oracle license, is the worst IT investment and waste of money. If you stop spending it, you can give free canteen for your employee, easily. And we only provide EC2 instance against physical servers, which is lack of imagination. A sales person in the system integrator had a
- Audi A6 Quattro as his company car, so I guess he is a very successful sales person, but question is,
- Aws has bunch of services, some of them are unknown by even aws employees. So aws engineers always need to gain their knowledge and train their imagination for application.
- We have a blog page since 2011, it’s not limited to aws, but our engineers examine new aws service released, and write articles. It has monthly 1.2M page view, so I can say every aws engineer and users in the industry in Japan is reading it.
- This is a snap at AWS developers event in tokyo. We sponsored it and had a booth there. They are booth attendant. What their t-shirts says? ….. we have some of aws gurus in our company. Pick up this guy.
- He is a director, but he is writing blogs as an engineer.
- In April this year, Amazon AI recognition added a function calls Image moderation. In the same day, he examined and post an article.
- The function is aiming to purging inappropriate images from the website, bbs etc.
- 5 days later, he post another article regarding image moderation.
- That is ….. By the way, the script was written with Go language.
- 2 month later, amazon ai recognition released another function called celebrity recognition. He of course examined it.
- The function is pointing the name from photos.
- In the same day, he post another article. Which is..
- It’s pity that I cannot show you the collection, but he wrote fairly long code for that. So once again
- Sure he does.
- So it’s connected our approach, enjoying it, and bring it to the real business world. In Europe, developers love aws and heavily using it, but not so many business on production stage is running on aws. So how can we ?
- Japanese sometimes goes extreme, for instance, SAP Japan released that more than 100 companies in Japan running SAP business application on aws. And that was June 2015. Now a days in Japan, it’s matter of course so it is not even the news anymore. A sushi restaurant chain migrates all their single systems on aws now.
- Convayer belt sushi restaurant is one of the extreme way of Japanese automation and mechanization. And they go extreme on system as well.
- Learning conveyer belt sushi basics. It is much more like family meal or entertainment though authentic sushi restaurant costs 200 - 300 euro per person. Sushi is expensive because its ingredients and needed skills to make, but main cost factor is each sushi is one-off products. Conveyer sushi made revolution in that and change it mass-productive. Sushi chef making sushi constantly meanwhile he still taking order from touch panel by customers table. So stream of the sushi conjunction of projection made and tailer made sushi.
- Conveyer belt sushi used to account by the color of the plate, such as green is 1 euro, yellow is 1.5 euro. But now a days a small chips and barcode is attached inside the plate, so the casher scan the plates at the table and make table payment possible.
- There’s no servers in the restaurants except display. It’s on the internet and aws. All mixture of mass-made and individual made sushi can grasped by sensor on the belt, and uploaded by Amazon Kinesis which is steaming data handler, and its application load the data on Redshift, which is managed data warehouse. Redshift is only a DWH so tableau visualize the situation going on it and show it to sushi chef. Then he can know which kind of sushi is lacking on live, and predict, and make them accordingly. The restaurant chain has web order system for take aways. Its auto scaling web site and store the order data in Amazon Aurora MySQL, which is managed relational database service without servers. The order data coming into the store, and the chef can see through tableau. So he can take order and make one-off, but in other time he continuously making sushi for the belt. Another challenge is accurate speculation of customer cue waiting time. They have existing mobile apps to notify the customer outside the restaurant when he can coming back and have seats. So we build speculation system using AWS machine learning using in store redshift data and appdata.
- Zooming in. At the bottom, it is the kinetics of the in-store demand chain as I told in the previous picture. The challenge consists of more aws components and functions, such as Lambda, function driving engine. Mobile app analysis, SNS, the sendmail service. All of those are server less, and managed service, so we haven’t setup and configured anything. Directly started with coding and deployment.
- This is a second talk from classmethod. Mr. Higashi gave his talk from business overview perspective, so I will give my talk from technical perspective. I'm George Yoshida. I belong to AWS department service group and work as a software engineer here in Berlin. Today's talk is "A Day In The Life:Service Group", what we're doing every day.
- Our department has 3 groups: Architect Group Pre-Sales, Consults and Delivers systems to our customers, this is the largest group of our department. After system is delivered to our customers, Operation Group keeps servers up and running, and they also provide Technical Support. Last is the Service Group, where I belong. Service group develop products for our AWS Customers.
- So What our group develops? We develop many products, from small ones to large ones. Most used product is a Customer Portal Site. At the site, customers can get technical support, analyze their AWS usage resource-wise and cost-wise. This AWS usage data is created by the next system, billing system, and I spend most of my time maintaining this system. This billing system uses hadoop to ETL AWS Billing Reports to fit into our accounting system.
- AWS Brings Something New Every Day. Most of the time, they are new features, but sometimes it brings changes that can break working systems. Today's talk is about the latter. AWS security team posted a blog about a week ago. "How to Prepare for AWS’s Move to Its Own Certificate Authority" In summary AWS migrates to its own CA Services like EC2, dynamodb will replace certificates from the new CA If Clients's trusted ca list is old, they need to be Updated. My talk is how we' re tackling this problem and this is still a work in progress.
- This is a diagram to illustrate the problem. When a client communicates with the server in HTTPS, they share certificates. If a certificate is issued by a trusted authority(CA), client conitnues communiation. But if it's issued by a unknown untrusted authority, client stops communication. Clients hold a trusted authority list at application level or OS level. And if a client holds an outdated list, the list needs to be updated.
- So who will be affected with this change? Detail is summarized in the blog post. For example, if an application usees OS's trusted authority list, and its program runs on Amazon Linux, you don't need any actions. But if you run scripts that use AWS CLI, and CLi is old enough, they need to be updated. CLI uses its own trusted CA list, not OS's. Even if the script runs on the newest Amazon Linux, CLI needs to be updated. AWS CLI is the first choice for scriptiing, and some legacy systems are more than 2 years old with no updates. We have more than 1000 active AWS Accounts and many customers will be likely affected with this change, so We need to locate and inform those customers.
- AWS users want to know when this change will happen, but the blog doesn't mention the migration schedule for EC2 or dyanamodb. I searched my mail box, there has been at least 2 announcements already. One is from last October about cloudsearch and the migration can happen next week. Another is just announced yesterday. Data Pipeline migration is planed about a month later. It's not a long way off at all, right?
- So how can we find affected clients? 100 % coverage is not realistic, so in this talk, I' ll show you a solution. This is simple but I hope this solves the most common use case, that is, find legacy AWS CLI users. In this solution, I use 2 key components: CloudTrail and Athena. AWS CloudTrail : Track user activity and API usage Amazon Athena : Query Data in S3(Presto As A Service) If you make AWS requests through browsers or programs or whatever, its activity is tracked via CloudTrail and logged to S3. For our customers, this feature is enabled by default for all regions. Athena is an SQL query service against S3. Athena uses Hive compatible metastore to define table and runs SQL queries against S3. For Presto engineers, it's like a Presto as a service.
- This is a sample activity log of CloudTrail From this log, you can see like this opration is an AWS API call from useragent, AWS CLI version 1.11 is used. target service is S3
- How can I find who's using old clients? First, define an Athena table for cloudtrail logs. This part is a bit technical, so I skip details. CloudTrail' s log fomat is not easy to handle for SQL as-is but amazon provides a serializer for this special format. So you can focus on writing SQL.
- After defining a table, all you need to do is just write and execute an SQL. In this example, I filter specific useragent and aggregate clients based on useragents. This result just lists AWS CLI clients, so not all of them are affected. You need to investigate versions or libary info in detail. And of course, you need to run this query against all regions and all aws accounts. First and foremost, this approach does not work if CloudTrail is not enabled. In this simple solution, I demonstrated that you can easily filter suspicious clients using CloudTrail and Athena.
- We are AWS Experts We Evaluate Every New Feature/Change every day. We love dog fooding. we feedback our knowledge to our customers. If you want to use AWS to maximise value, please contact us.