ゆるやかにBigQuery（データ基盤）を安定活用するプラクティス集 #bq_sushi

•

2 likes•489 views

@yuzutas0 Yokoyama

Copy from SpeakerDeck https://speakerdeck.com/yuzutas0

Technology

Confidential & Proprietary
Now: Next:
Google

3  
10 - SQL
BigQuery - Google Cloud Platform
Source 
Warehouse 
 
App 
 
IF

sample__source__db
2  
BEM CSS
source
warehouse
app
DB

SELECT id, comment
FROM `source__db.view_table_*`
WHERE accountId = 123456
Error: Cannot read field
'comment' of type STRING
as INT64

$bq query --allow_large_results --use_legacy_sql=false --replace -- destination_table {table} "SELECT a, b, c, CAST(d AS INT64) AS D, E, F FROM {table}”$

SELECT
maskMailAddress,
CASE WHEN comment IS NULL THEN NULL else QUOTE(comment) END AS comment,
createdAt,

BigQuery - Google Cloud Platform
Source 
Warehouse 
 
App 
 
IF

ゆるやかにBigQuery（データ基盤）を安定活用するプラクティス集 #bq_sushi

Apache Airflowとは、「Python言語で定義したワークフローを、スケジュール・モニタリングするためのプラットフォーム」です。この勉強会では、Apache Airflowの概要と特徴を紹介し。 Airflowをセットアップし簡単なワークフローを実行する方法を説明します。ジョブの依存関係解決・再実行が行いやすいというAirflowの特徴が活かせる利用シーンとして、レポーティングや機械学習での利用イメージについても紹介します。

Where狙いのキー、order by狙いのキー

yoku0825

SQLアンチパターンメンター用資料

Hironori Miura

ビッグデータ処理データベースの全体像と使い分け 2018年version

Tetsutaro Watanabe

雑なMySQLパフォーマンスチューニング

yoku0825

Swaggerでのapi開発よもやま話

KEISUKE KONISHI

より詳細なCQRSに関する資料はこちら https://little-hands.hatenablog.com/entry/2019/12/02/cqrs 参考資料：http://little-hands.hatenablog.com/entry/jjug2017fall 社内新規プロダクトでDDD, CQRSの思想をベースとしたアーキテクチャを構築し、コマンド(更新系処理)ではSpring Data JPA(Hibernate)を、クエリ(参照系処理)ではjOOQを採用しました。結果としてそれぞれのORMの良いところを生かした組み合わせのアーキテクチャが構築できたので、その経緯と得られた知見についてお話ししたいと思います。以下のようなトピックを考えています。・CQRSの定義とメリットデメリット・DDD,CQRSを検討するにあたってのORMの選定ポイント・構築したアーキテクチャ CQRSはDDDと切り分けて単独でも適用することができるので、DDDについてご存知ない方もご覧いただけます。日本語の文献は意外と少ないので、この辺りの分野に興味がある人の参考になれば幸いです。

CodeBuildを身近にするためのはじめの一歩

淳千葉

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

Amazon Cognito使って認証したい？それならSpring Security使いましょう！

Ryosuke Uchitate

ドメイン駆動設計本格入門

増田亨

100%Kotlin ORM Ktormを試してみた

Keita Tsukamoto

ビッグデータ処理データベースの全体像と使い分け

Recruit Technologies

Firebaseを利用するためにGCPとCloud IAMの基本を理解しよう

kbigwheel

実践 NestJS

Ayumi Goto

株式会社オロ新規事業開発チーム後藤歩と申します。 NestJSは弊社の2つのプロダクトで1年ほど使っています。（個人的にも1プロダクトで使っており、合計3プロダクトでNestJSを使って開発をしています）その実例をコードを交えてご紹介できればと思います。システム構成認証、認可、フレームワーク、DB、API、バリデーション、キューイング、スケジューリング、マイグレーション、テスト、- Swagger、i18nをどのようにやっているかのコードを交えて実例紹介など

2018 builderscon airflowを用いて、複雑大規模なジョブフロー管理に立ち向かう

BrainPad Inc.

MLOps入門

Hiro Mura

PostgreSQLの行レベルセキュリティと SpringAOPでマルチテナントのユーザー間情報漏洩を防止する (JJUG CCC 2021 Spring)

Koichiro Matsuoka

Webアプリケーションにおいて、マルチテナント型、つまり複数のユーザー組織がアプリケーションとデータベースを共有する構成にすることがあります。この構成の持つリスクとして、万が一バグにより他テナントの情報が見えてしまうとそれは情報漏洩となり、重大なインシデントとなってしまうことがあります。この重要性を考えると、「気を付けて実装する」だけではなく、仕組みで漏洩を防ぐような対策には価値があります。そこで、今回はPostgresSQLの行レベルセキュリティと、SpringAOPによる処理を組み合わせて、ログインしているテナントのデータにしかアクセスできなくする仕組みを実現しました。導入にあたり考慮した複数の選択肢、乗り越えたいくつかの壁についてご紹介します。同様の課題を抱えている方の参考にしていただけるような情報をお伝えしたいと思います。

MySQLアーキテクチャ図解講座

Mikiya Okuno

データウェアハウスモデリング入門（ダイジェスト版）（事前公開版）

Satoshi Nagayasu

SQLアンチパターン - 開発者を待ち受ける25の落とし穴 (拡大版)

Takuto Wada

書籍「Python FlaskによるWebアプリ開発入門物体検知アプリ&機械学習APIの作り方」を通して伝えたいFlaskのプラクティス.pdf

taisa831

エンジニアの個人ブランディングと技術組織

Takafumi ONAKA

Apache Sparkに手を出してヤケドしないための基本～「Apache Spark入門より」～（デブサミ 2016 講演資料）

NTT DATA OSS Professional Services

Integrating an App with Amazon Web Services SimpleDB - A Matter of Choices

Mark Maslyn

Get Back in Control of your SQL with jOOQ - GeekOut by ZeroTurnaround

DataGeekery

SQL is a powerful and highly expressive language for queries against relational databases. SQL is established, standardised and hardly challenged by alternative querying languages. Nonetheless, in the Java ecosystem, there had been few relevant steps forward since JDBC to better integrate SQL into Java. All attention was given to object-relational mapping and language abstractions on a higher level, such as OQL, HQL, JPQL, CriteriaQuery. In the meantime, these abstractions have become almost as complex as SQL itself, regardless of the headaches they’re giving to DBAs who can no longer patch the generated SQL. jOOQ is a dual-licensed Open Source product filling this gap. It implements SQL itself as an internal domain-specific language in Java, allowing for the typesafe construction and execution of SQL statements of arbitrary complexity. This includes nested selects, derived tables, joins, semi-joins, anti-joins, self-joins, aliasing, as well as many vendor-specific extensions such as stored procedures, arrays, user-defined types, recursive SQL, grouping sets, pivot tables, window functions and many other OLAP features. jOOQ also includes a source code generator allowing you to compile queries in modern IDEs such as Eclipse very efficiently. jOOQ is a good choice in a Java application where SQL and the specific relational database are important. It is an alternative when JPA / Hibernate abstract too much, JDBC too little. It shows, how a modern domain-specific language can greatly increase developer productivity, internalising SQL into Java.

What's hot

Java ORマッパー選定のポイント #jsug

Masatoshi Tada

AWS Black Belt Online Seminar 2018 Amazon DynamoDB Advanced Design Pattern

Amazon Web Services Japan

DDD x CQRS 更新系と参照系で異なるORMを併用して上手くいった話

Koichiro Matsuoka

CodeBuildを身近にするためのはじめの一歩

淳千葉

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

Amazon Cognito使って認証したい？それならSpring Security使いましょう！

Ryosuke Uchitate

ドメイン駆動設計本格入門

増田亨

100%Kotlin ORM Ktormを試してみた

Keita Tsukamoto

ビッグデータ処理データベースの全体像と使い分け

Recruit Technologies

Firebaseを利用するためにGCPとCloud IAMの基本を理解しよう

kbigwheel

実践 NestJS

Ayumi Goto

2018 builderscon airflowを用いて、複雑大規模なジョブフロー管理に立ち向かう

BrainPad Inc.

MLOps入門

Hiro Mura

PostgreSQLの行レベルセキュリティと SpringAOPでマルチテナントのユーザー間情報漏洩を防止する (JJUG CCC 2021 Spring)

Koichiro Matsuoka

MySQLアーキテクチャ図解講座

Mikiya Okuno

データウェアハウスモデリング入門（ダイジェスト版）（事前公開版）

Satoshi Nagayasu

SQLアンチパターン - 開発者を待ち受ける25の落とし穴 (拡大版)

Takuto Wada

書籍「Python FlaskによるWebアプリ開発入門物体検知アプリ&機械学習APIの作り方」を通して伝えたいFlaskのプラクティス.pdf

taisa831

エンジニアの個人ブランディングと技術組織

Takafumi ONAKA

Apache Sparkに手を出してヤケドしないための基本～「Apache Spark入門より」～（デブサミ 2016 講演資料）

NTT DATA OSS Professional Services

What's hot (20)

Java ORマッパー選定のポイント #jsug

AWS Black Belt Online Seminar 2018 Amazon DynamoDB Advanced Design Pattern

DDD x CQRS 更新系と参照系で異なるORMを併用して上手くいった話

CodeBuildを身近にするためのはじめの一歩

KafkaとAWS Kinesisの比較

Amazon Cognito使って認証したい？それならSpring Security使いましょう！

ドメイン駆動設計本格入門

100%Kotlin ORM Ktormを試してみた

ビッグデータ処理データベースの全体像と使い分け

Firebaseを利用するためにGCPとCloud IAMの基本を理解しよう

実践 NestJS

2018 builderscon airflowを用いて、複雑大規模なジョブフロー管理に立ち向かう

MLOps入門

PostgreSQLの行レベルセキュリティと SpringAOPでマルチテナントのユーザー間情報漏洩を防止する (JJUG CCC 2021 Spring)

MySQLアーキテクチャ図解講座

データウェアハウスモデリング入門（ダイジェスト版）（事前公開版）

SQLアンチパターン - 開発者を待ち受ける25の落とし穴 (拡大版)

書籍「Python FlaskによるWebアプリ開発入門物体検知アプリ&機械学習APIの作り方」を通して伝えたいFlaskのプラクティス.pdf

エンジニアの個人ブランディングと技術組織

Apache Sparkに手を出してヤケドしないための基本～「Apache Spark入門より」～（デブサミ 2016 講演資料）

Similar to ゆるやかにBigQuery（データ基盤）を安定活用するプラクティス集 #bq_sushi

Integrating an App with Amazon Web Services SimpleDB - A Matter of Choices

Mark Maslyn

Get Back in Control of your SQL with jOOQ - GeekOut by ZeroTurnaround

DataGeekery

Introduction To MySQL Lecture 1

Ajay Khatri

New Framework - ORM

Odoo

Launching Beeline with Firebase

Chetan Padia

GR8Conf 2011: GORM OptimizationGR8Conf

Jdbc ppt

AISHWARIYA1S

Enterprise NodeJS

Herry Atwar

Stored Procedure

NidiaRamirez07

Launching Beeline with Firebase

Chetan Padia

web programming using html,css, JavaScript ,php etc

albinjamestpra

Amazon Bedrock in Action - presentation of the Bedrock's capabilities

KrzysztofKkol1

The vJUG talk about jOOQ: Get Back in Control of Your SQL

Lukas Eder

jOOQ: Get Back in Control of Your SQL SQL is a powerful and highly expressive language for queries against relational databases. SQL is established, standardised and hardly challenged by alternative querying languages. Nonetheless, in the Java ecosystem, there had been few relevant steps forward since JDBC to better integrate SQL into Java. All attention was given to object-relational mapping and language abstractions on a higher level, such as OQL, HQL, JPQL, CriteriaQuery. In the meantime, these abstractions have become almost as complex as SQL itself, regardless of the headaches they're giving to DBAs who can no longer patch the generated SQL. jOOQ is a dual-licensed Open Source product filling this gap. It implements SQL itself as an internal domain-specific language in Java, allowing for the typesafe construction and execution of SQL statements of arbitrary complexity. This includes nested selects, derived tables, joins, semi-joins, anti-joins, self-joins, aliasing, as well as many vendor-specific extensions such as stored procedures, arrays, user-defined types, recursive SQL, grouping sets, pivot tables, window functions and many other OLAP features. jOOQ also includes a source code generator allowing you to compile queries in modern IDEs such as Eclipse very efficiently. jOOQ is a good choice in a Java application where SQL and the specific relational database are important. It is an alternative when JPA / Hibernate abstract too much, JDBC too little. It shows, how a modern domain-specific language can greatly increase developer productivity, internalising SQL into Java.

Beeline Firebase talk - Firebase event Jun 2017

Chetan Padia

A Brief Introduction About Sql Injection in PHP and MYSQL

kobaitari

GHC Participant TrainingAidIQ

Android sq lite database tutorial

maamir farooq

WooCommerce CRUD and Data Store by Akeda Bagus

WordCamp Indonesia

AWS Serverless Workshop

Mikael Puittinen

前端MVC 豆瓣说Ting Lv

Similar to ゆるやかにBigQuery（データ基盤）を安定活用するプラクティス集 #bq_sushi (20)

Integrating an App with Amazon Web Services SimpleDB - A Matter of Choices

Get Back in Control of your SQL with jOOQ - GeekOut by ZeroTurnaround

Introduction To MySQL Lecture 1

New Framework - ORM

Launching Beeline with Firebase

GR8Conf 2011: GORM Optimization

Jdbc ppt

Enterprise NodeJS

Stored Procedure

Launching Beeline with Firebase

web programming using html,css, JavaScript ,php etc

Amazon Bedrock in Action - presentation of the Bedrock's capabilities

The vJUG talk about jOOQ: Get Back in Control of Your SQL

Beeline Firebase talk - Firebase event Jun 2017

A Brief Introduction About Sql Injection in PHP and MYSQL

GHC Participant Training

Android sq lite database tutorial

WooCommerce CRUD and Data Store by Akeda Bagus

AWS Serverless Workshop

前端MVC 豆瓣说

Recently uploaded

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Free Complete Python - A step towards Data Science

RinaMondal9

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

Pushing the limits of ePRTC: 100ns holdover for 100 days

Adtran

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

The Future of Platform Engineering

Jemma Hussein Allen

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

Microsoft - Power Platform_G.Aspiotis.pdf

Uni Systems S.M.S.A.

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

Recently uploaded (20)

Securing your Kubernetes cluster_ a step-by-step guide to success !

Essentials of Automations: The Art of Triggers and Actions in FME

DevOps and Testing slides at DASA Connect

Introduction to CHERI technology - Cybersecurity

A tale of scale & speed: How the US Navy is enabling software delivery from l...

Video Streaming: Then, Now, and in the Future

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

Removing Uninteresting Bytes in Software Fuzzing

Free Complete Python - A step towards Data Science

Epistemic Interaction - tuning interfaces to provide information for AI support

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

Pushing the limits of ePRTC: 100ns holdover for 100 days

Climate Impact of Software Testing at Nordic Testing Days

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

The Future of Platform Engineering

Elizabeth Buie - Older adults: Are we really designing for our future selves?

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Microsoft - Power Platform_G.Aspiotis.pdf

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

ゆるやかにBigQuery（データ基盤）を安定活用するプラクティス集 #bq_sushi

5. Confidential & Proprietary Now: Next: Google

6. Confidential & Proprietary Now: Next: Google    

10.

11.

12.

13. APP  

14.

15.

16.

17.            

18.                      

19.              

20.

21.

22.    

23.    

24.        

25.  

26.              

27.  

28.  

29.

30.          

31.

32.  

33.    

34.        

35.      

36.                      

37.     

38.    

39.

40.

41.                                  

42.  

43.          

44.    

45.    

46.

47.              

48.  

49.

50.

51.

52.

53.

54. 3   10 - SQL BigQuery - Google Cloud Platform Source  Warehouse    App    IF

55. sample__source__db 2   BEM CSS source warehouse app DB

56.      

57.

58.        

59.        

60.                

61.  

62.  

63.              

64.

65.

66.

67.

68.  

69.

70.   SELECT id, comment FROM `source__db.view_table_*` WHERE accountId = 123456 Error: Cannot read field 'comment' of type STRING as INT64

71.     bq query --allow_large_results --use_legacy_sql=false --replace -- destination_table {table} "SELECT a, b, c, CAST(d AS INT64) AS D, E, F FROM {table}”

72.

73.  

74.  

75.

76.

77.    

78. SELECT maskMailAddress, CASE WHEN comment IS NULL THEN NULL else QUOTE(comment) END AS comment, createdAt,

79.  

80.

81.

82.  

83.    