Yapc2012 lt

•Download as KEY, PDF•

0 likes•616 views

The document discusses using the Teng framework to generate dummy data for testing and tuning a web application. It describes challenges with changing schemas and data patterns when generating dummy data using MySQL. The author then presents a solution using a Perl script that caches dummy data using Storable to significantly speed up initialization, reducing the time from over 5 minutes to just 3 seconds. They conclude by requesting ideas and knowledge from others on the topic.

Dummy data with Teng
id:karupanerura
Mobile Factory, Inc

web application
development

O/R Mapper

Teng

wanted dummy data

use for

tuning

benchmark

testing

development

etc...

how to make
dummy data?

insert dummy data to
MySQL?

gen_dummy_data.pl ?

It’s messy...

problem

changing schema

changing data pattern

It’s messy...

not using MySQL?

grep?

but, I want to use
Teng interface

writing business logic
with Teng

It’s messy...

grep with Teng interface!!

DEMO

https://github.com/karupanerura/p5-Teng-Plugin-Static

problem

It’s very slow...

5min
for
initialize dummy data...

use Storable qw/store retrieve/;

cache with mtime

fast!!

fast...?

3sec
for
initialize dummy data...

umm....

please share
idea and know how!!

thank you for listening

More Related Content

Similar to Yapc2012 lt

CoreData - there is an ORM you can like!

CoreData - there is an ORM you can like!

CoreData - there is an ORM you can like!

Starting with outlining the history of conventional version control before diving into explaining QoDs (Quantitative Oriented Developers) and the unique problems their ML systems pose from an operations perspective (MLOps). With the only status quo solutions being proprietary in-house pipelines (exclusive to Uber, Google, Facebook) and manual tracking/fragile "glue" code for everyone else. Datmo works to solve this issue by empowering QoDs in two ways: making MLOps manageable and simple (rather than completely abstracted away) as well as reducing the amount of glue code so to ensure more robust end-to-end pipelines. This goes through a simple example of using Datmo with an Iris classification dataset. Later workshops will expand to show how Datmo can work with other data pipelining tools.

Version Control in Machine Learning + AI (Stanford)

Version Control in Machine Learning + AI (Stanford)

Version Control in Machine Learning + AI (Stanford)

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

Machine learning at scale - Webinar By zekeLabs

Machine learning at scale - Webinar By zekeLabs

Machine learning at scale - Webinar By zekeLabs

zekeLabs Technologies

Amplify your stack - Jsfoo pune 2012

Amplify your stack - Jsfoo pune 2012

Amplify your stack - Jsfoo pune 2012

Deep Learning using Tensorflow and Data Science Experience

Deep Learning using Tensorflow and Data Science Experience

Deep Learning using Tensorflow and Data Science Experience

Infrastructure as code might be literally impossible part 2

Infrastructure as code might be literally impossible part 2

Infrastructure as code might be literally impossible part 2

Credit Card Fraud Analysis Using Data Science (1).pdf

Credit Card Fraud Analysis Using Data Science (1).pdf

Credit Card Fraud Analysis Using Data Science (1).pdf

Presented by Kevin Perko, Head of Data Science at Scribd Next DSS NYC Event 👉 https://datascience.salon/newyork/ Next DSS LA Event 👉 https://datascience.salon/la/ Kevin will cover his experience using deep learning, going from scratch to deploying models in production to improve the product experience. He goes in-depth in terms of how we started deep learning from scratch, including navigating the maze of frameworks and hyper-parameters to optimize. Kevin will discuss pitfalls of using other people's algorithms and make a call for more rigor in publishing data science blog posts. Kevin closes with how his failure turned into an open source contribution and the work in moving from dev to production.

Data Science Salon: Deep Learning as a Product @ Scribd

Data Science Salon: Deep Learning as a Product @ Scribd

Data Science Salon: Deep Learning as a Product @ Scribd

Machine Learning In Production

Machine Learning In Production

Machine Learning In Production

You've officially leveled-up in the world of software development with Visual Studio, and you now find yourself working on projects that are infinitely larger and far more complex than anything you've worked on before. Time to beef up your debugging skills not only to debug, but sometimes just to learn the flow of a given process! Plain old break-points have always gotten you by, but now you quickly run into trickier situations that come close to making your head explode. Come and learn some new debugging tricks & tips to bring your debugging skills to new heights.

Spaghetti gate

Many companies are looking for "DevOps'' in many forms, but what kind of skills or experiences are actually needed? I’ll debunk some of the myths surrounding what recruiters or internet lurkers might tell you and find out if you might actually have an aptitude for Site Reliability or Infrastructure Engineering. If so, what might be good knowledge areas to get started with? And if learning leads to an interview, what might that look like?

How to get started with Site Reliability Engineering

How to get started with Site Reliability Engineering

How to get started with Site Reliability Engineering

Andrew Kirkpatrick

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

SANTIAGO PABLO ALBERTO

You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently… Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice! In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

Data Science Stack with MongoDB and RStudio

Data Science Stack with MongoDB and RStudio

Data Science Stack with MongoDB and RStudio

This talk will address one apparently simple question: which open source machine learning tools one can use out-of-the-box to do binary classification (probably the most common machine learning problem) on largish datasets. We will discuss therefore the speed, scalability and accuracy of the top (most accurate) learning algorithms in R packages, Python’s scikit-learn, H2O, xgboost and Spark’s MLlib.

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

Abstract of the Presentation: This talk is for the underdog. If you’re trying to solve data related problems with no or limited resources, be them time, money or skills don’t go no further. This talk points mostly to decades old technology, free operating systems and cheap hardware if possible, but if it makes sense to spend a hundred bucks instead of tearing your hair, we’ll say so. This talk is opinionated and updated to GDPR, deep learning and all the hype. About the Author: Daniel Molnar is a data nerd and startup specialist. With over 19 years of experience in startups and nine years of expertise in data related topics, he is an experienced co-founder who has built and hired teams up of to 30 people. He comes with expertise in proven build-to-market capabilities and utilizing data for successful products. An amalgamation of his skills would be CS + data + product background under one hat.

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

Dataconomy Media

The Data Janitor Returns | Daniel Molnar | DN18

The Data Janitor Returns | Daniel Molnar | DN18

The Data Janitor Returns | Daniel Molnar | DN18

You're woken up in the middle of the night to your phone. Your app is down and you're on call to fix it. Eventually you track it down to "something with the db," but what exactly is wrong? And of course, you're sure that nothing changed recently… Knowing what to fix, and even where to start looking, is a skill that takes a long time to develop. Especially since Postgres normally works very well for months at a time, not letting you get practice! In this talk, I'll share not only the more common failure cases and how to fix them, but also a general approach to efficiently figuring out what's wrong in the first place.

When it all goes wrong | PGConf EU 2019 | Will Leinweber

When it all goes wrong | PGConf EU 2019 | Will Leinweber

When it all goes wrong | PGConf EU 2019 | Will Leinweber

Um2010

Similar to Yapc2012 lt (20)

CoreData - there is an ORM you can like!

CoreData - there is an ORM you can like!

CoreData - there is an ORM you can like!

Version Control in Machine Learning + AI (Stanford)

Version Control in Machine Learning + AI (Stanford)

Version Control in Machine Learning + AI (Stanford)

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

BigDataCloud meetup - July 8th - Cost effective big-data processing using Ama...

Machine learning at scale - Webinar By zekeLabs

Machine learning at scale - Webinar By zekeLabs

Machine learning at scale - Webinar By zekeLabs

Amplify your stack - Jsfoo pune 2012

Amplify your stack - Jsfoo pune 2012

Amplify your stack - Jsfoo pune 2012

Deep Learning using Tensorflow and Data Science Experience

Deep Learning using Tensorflow and Data Science Experience

Deep Learning using Tensorflow and Data Science Experience

Infrastructure as code might be literally impossible part 2

Infrastructure as code might be literally impossible part 2

Infrastructure as code might be literally impossible part 2

Credit Card Fraud Analysis Using Data Science (1).pdf

Credit Card Fraud Analysis Using Data Science (1).pdf

Credit Card Fraud Analysis Using Data Science (1).pdf

Data Science Salon: Deep Learning as a Product @ Scribd

Data Science Salon: Deep Learning as a Product @ Scribd

Data Science Salon: Deep Learning as a Product @ Scribd

Machine Learning In Production

Machine Learning In Production

Machine Learning In Production

Spaghetti gate

How to get started with Site Reliability Engineering

How to get started with Site Reliability Engineering

How to get started with Site Reliability Engineering

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

Raspberry pi: Aprende raspberry pi con Linux por peter membrey y david hows.

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber

Data Science Stack with MongoDB and RStudio

Data Science Stack with MongoDB and RStudio

Data Science Stack with MongoDB and RStudio

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

Big Data Day LA 2015 - Machine Learning on Largish Data by Szilard Pafka of E...

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

DN18 | The Data Janitor Returns | Daniel Molnar | Oberlo/Shopify

The Data Janitor Returns | Daniel Molnar | DN18

The Data Janitor Returns | Daniel Molnar | DN18

The Data Janitor Returns | Daniel Molnar | DN18

When it all goes wrong | PGConf EU 2019 | Will Leinweber

When it all goes wrong | PGConf EU 2019 | Will Leinweber

When it all goes wrong | PGConf EU 2019 | Will Leinweber

Um2010

More from karupanerura

Perl5 VS JSON

コンテキストと仲良く

The plan of Aniki 2.0

The plan of Aniki 2.0

The plan of Aniki 2.0

Aniki::Internal

Aniki::Internal

Aniki::Internal

The Crystal language *recently* update

The Crystal language *recently* update

The Crystal language *recently* update

KOWAZA for mackerel

KOWAZA for mackerel

KOWAZA for mackerel

TIme::Moment+Time::Strptime=

TIme::Moment+Time::Strptime=

TIme::Moment+Time::Strptime=

DateTimeX::Moment

DateTimeX::Moment

DateTimeX::Moment

Aniki has come

Optimize perl5 code for perfomance freaks

Optimize perl5 code for perfomance freaks

Optimize perl5 code for perfomance freaks

Technology for reduce of mistakes - うっかりをなくす技術

Technology for reduce of mistakes - うっかりをなくす技術

Technology for reduce of mistakes - うっかりをなくす技術

router-simple.cr

router-simple.cr

router-simple.cr

Why we use mruby with Perl5?

Why we use mruby with Perl5?

Why we use mruby with Perl5?

Gotanda.pmの紹介

Perlにおけるclass実装パターン

Perlにおけるclass実装パターン

Perlにおけるclass実装パターン

モジュール開発におけるぼくの試行錯誤

モジュール開発におけるぼくの試行錯誤

モジュール開発におけるぼくの試行錯誤

Aniki - The ORM as our great brother.

Aniki - The ORM as our great brother.

Aniki - The ORM as our great brother.

Perl5 meta programming

Perl5 meta programming

Perl5 meta programming

mysqlcasual6-next-key-lock

mysqlcasual6-next-key-lock

mysqlcasual6-next-key-lock

dwangocpp1-lt

More from karupanerura (20)

Perl5 VS JSON

コンテキストと仲良く

The plan of Aniki 2.0

The plan of Aniki 2.0

The plan of Aniki 2.0

Aniki::Internal

Aniki::Internal

Aniki::Internal

The Crystal language *recently* update

The Crystal language *recently* update

The Crystal language *recently* update

KOWAZA for mackerel

KOWAZA for mackerel

KOWAZA for mackerel

TIme::Moment+Time::Strptime=

TIme::Moment+Time::Strptime=

TIme::Moment+Time::Strptime=

DateTimeX::Moment

DateTimeX::Moment

DateTimeX::Moment

Aniki has come

Optimize perl5 code for perfomance freaks

Optimize perl5 code for perfomance freaks

Optimize perl5 code for perfomance freaks

Technology for reduce of mistakes - うっかりをなくす技術

Technology for reduce of mistakes - うっかりをなくす技術

Technology for reduce of mistakes - うっかりをなくす技術

router-simple.cr

router-simple.cr

router-simple.cr

Why we use mruby with Perl5?

Why we use mruby with Perl5?

Why we use mruby with Perl5?

Gotanda.pmの紹介

Perlにおけるclass実装パターン

Perlにおけるclass実装パターン

Perlにおけるclass実装パターン

モジュール開発におけるぼくの試行錯誤

モジュール開発におけるぼくの試行錯誤

モジュール開発におけるぼくの試行錯誤

Aniki - The ORM as our great brother.

Aniki - The ORM as our great brother.

Aniki - The ORM as our great brother.

Perl5 meta programming

Perl5 meta programming

Perl5 meta programming

mysqlcasual6-next-key-lock

mysqlcasual6-next-key-lock

mysqlcasual6-next-key-lock

dwangocpp1-lt

Yapc2012 lt

1. Dummy data with Teng id:karupanerura Mobile Factory, Inc

2. web application development

5. wanted dummy data

10. development

12. how to make dummy data?

13. insert dummy data to MySQL?

14. gen_dummy_data.pl ?

15. It’s messy...

17. changing schema

18. changing data pattern

19. It’s messy...

20. not using MySQL?

22. but, I want to use Teng interface

23. writing business logic with Teng

24. It’s messy...

26. grep with Teng interface!!

27. DEMO https://github.com/karupanerura/p5-Teng-Plugin-Static

29. It’s very slow...

30. 5min for initialize dummy data...

31. use Storable qw/store retrieve/;

32. cache with mtime

35. 3sec for initialize dummy data...

37. please share idea and know how!!

38. thank you for listening

Editor's Notes