Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321) ...Amazon Web Services
Come to this session to learn how Amazon DynamoDB was built as the hyper-scale database for internet-scale applications. In January 2012, Amazon launched DynamoDB, a cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance, and manageability needed to run mission-critical workloads. This session discloses for the first time the underpinnings of DynamoDB, and how we run a fully managed nonrelational database used by more than 100,000 customers. We cover the underlying technical aspects of how an application works with DynamoDB for authentication, metadata, storage nodes, streams, backup, and global replication.
Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis. It is open-sourced by Baidu. Doris mainly integrates the technology of Google Mesa and Apache Impala. Unlike other popular SQL-on-Hadoop systems, Doris is designed to be a simple and single tightly coupled system, not depending on other systems. Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis. Doris not only provides batch data loading, but also provides near real-time mini-batch data loading. Doris also provides high availability, reliability, fault tolerance, and scalability. The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris.
There are parallels between storing JSON data in PostgreSQL and storing vectors that are produced from AI/ML systems. This lightning talk briefly covers the similarities in use-cases in storing JSON and vectors in PostgreSQL, shows some of the use-cases developers have for querying vectors in Postgres, and some roadmap items for improving PostgreSQL as a vector database.
Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321) ...Amazon Web Services
Come to this session to learn how Amazon DynamoDB was built as the hyper-scale database for internet-scale applications. In January 2012, Amazon launched DynamoDB, a cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance, and manageability needed to run mission-critical workloads. This session discloses for the first time the underpinnings of DynamoDB, and how we run a fully managed nonrelational database used by more than 100,000 customers. We cover the underlying technical aspects of how an application works with DynamoDB for authentication, metadata, storage nodes, streams, backup, and global replication.
Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis. It is open-sourced by Baidu. Doris mainly integrates the technology of Google Mesa and Apache Impala. Unlike other popular SQL-on-Hadoop systems, Doris is designed to be a simple and single tightly coupled system, not depending on other systems. Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis. Doris not only provides batch data loading, but also provides near real-time mini-batch data loading. Doris also provides high availability, reliability, fault tolerance, and scalability. The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris.
There are parallels between storing JSON data in PostgreSQL and storing vectors that are produced from AI/ML systems. This lightning talk briefly covers the similarities in use-cases in storing JSON and vectors in PostgreSQL, shows some of the use-cases developers have for querying vectors in Postgres, and some roadmap items for improving PostgreSQL as a vector database.
OpenSearch는 배포형 오픈 소스 검색과 분석 제품군으로 실시간 애플리케이션 모니터링, 로그 분석 및 웹 사이트 검색과 같이 다양한 사용 사례에 사용됩니다. OpenSearch는 데이터 탐색을 쉽게 도와주는 통합 시각화 도구 OpenSearch와 함께 뛰어난 확장성을 지닌 시스템을 제공하여 대량 데이터 볼륨에 빠르게 액세스 및 응답합니다. 이 세션에서는 실제 동작 구조에 대한 설명을 바탕으로 최적화를 하기 위한 방법과 운영상에 발생할 수 있는 이슈에 대해서 알아봅니다.
아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 A...Amazon Web Services Korea
Apache Airflow는 복잡한 데이터 처리 파이프라인의 전체적인 프로세스를 자동화하기 위한 워크플로우 관리 플랫폼이며 오픈 소스 커뮤니티에서 활발하게 기여하고 있는 top-level 프로젝트 입니다. AWS는 최근에 Amazon Managed Workflow for Apache Airflow (MWAA) 서비스를 정식 출시하였고, 본 강연에서는 Apache Airflow 및 MWAA를 소개하고 어떻게 AWS 서비스와 연동하여 데이터 처리 워크플로우를 구축할 수 있는지 데모를 통해 알려 드립니다.
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
From webinar on December 3, 2019
New users of ClickHouse love the speed but may run into a few surprises when designing applications. Column storage turns classic SQL design precepts on their heads. This talk shares our favorite tricks for building great applications. We'll talk about fact tables and dimensions, materialized views, codecs, arrays, and skip indexes, to name a few of our favorites. We'll show examples of each and also reserve time to handle questions. Join us to take your next step to ClickHouse guruhood!
Speaker Bio:
Robert Hodges is CEO of Altinity, which offers enterprise support for ClickHouse. He has over three decades of experience in data management spanning 20 different DBMS types. ClickHouse is his current favorite. ;)
OpenSearch는 배포형 오픈 소스 검색과 분석 제품군으로 실시간 애플리케이션 모니터링, 로그 분석 및 웹 사이트 검색과 같이 다양한 사용 사례에 사용됩니다. OpenSearch는 데이터 탐색을 쉽게 도와주는 통합 시각화 도구 OpenSearch와 함께 뛰어난 확장성을 지닌 시스템을 제공하여 대량 데이터 볼륨에 빠르게 액세스 및 응답합니다. 이 세션에서는 실제 동작 구조에 대한 설명을 바탕으로 최적화를 하기 위한 방법과 운영상에 발생할 수 있는 이슈에 대해서 알아봅니다.
아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 A...Amazon Web Services Korea
Apache Airflow는 복잡한 데이터 처리 파이프라인의 전체적인 프로세스를 자동화하기 위한 워크플로우 관리 플랫폼이며 오픈 소스 커뮤니티에서 활발하게 기여하고 있는 top-level 프로젝트 입니다. AWS는 최근에 Amazon Managed Workflow for Apache Airflow (MWAA) 서비스를 정식 출시하였고, 본 강연에서는 Apache Airflow 및 MWAA를 소개하고 어떻게 AWS 서비스와 연동하여 데이터 처리 워크플로우를 구축할 수 있는지 데모를 통해 알려 드립니다.
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
From webinar on December 3, 2019
New users of ClickHouse love the speed but may run into a few surprises when designing applications. Column storage turns classic SQL design precepts on their heads. This talk shares our favorite tricks for building great applications. We'll talk about fact tables and dimensions, materialized views, codecs, arrays, and skip indexes, to name a few of our favorites. We'll show examples of each and also reserve time to handle questions. Join us to take your next step to ClickHouse guruhood!
Speaker Bio:
Robert Hodges is CEO of Altinity, which offers enterprise support for ClickHouse. He has over three decades of experience in data management spanning 20 different DBMS types. ClickHouse is his current favorite. ;)
【DLゼミ】XFeat: Accelerated Features for Lightweight Image Matchingharmonylab
公開URL:https://arxiv.org/pdf/2404.19174
出典:Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. ascimento: XFeat: Accelerated Features for Lightweight Image Matching, Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
概要:リソース効率に優れた特徴点マッチングのための軽量なアーキテクチャ「XFeat(Accelerated Features)」を提案します。手法は、局所的な特徴点の検出、抽出、マッチングのための畳み込みニューラルネットワークの基本的な設計を再検討します。特に、リソースが限られたデバイス向けに迅速かつ堅牢なアルゴリズムが必要とされるため、解像度を可能な限り高く保ちながら、ネットワークのチャネル数を制限します。さらに、スパース下でのマッチングを選択できる設計となっており、ナビゲーションやARなどのアプリケーションに適しています。XFeatは、高速かつ同等以上の精度を実現し、一般的なラップトップのCPU上でリアルタイムで動作します。
セル生産方式におけるロボットの活用には様々な問題があるが,その一つとして 3 体以上の物体の組み立てが挙げられる.一般に,複数物体を同時に組み立てる際は,対象の部品をそれぞれロボットアームまたは治具でそれぞれ独立に保持することで組み立てを遂行すると考えられる.ただし,この方法ではロボットアームや治具を部品数と同じ数だけ必要とし,部品数が多いほどコスト面や設置スペースの関係で無駄が多くなる.この課題に対して音𣷓らは組み立て対象物に働く接触力等の解析により,治具等で固定されていない対象物が組み立て作業中に運動しにくい状態となる条件を求めた.すなわち,環境中の非把持対象物のロバスト性を考慮して,組み立て作業条件を検討している.本研究ではこの方策に基づいて,複数物体の組み立て作業を単腕マニピュレータで実行することを目的とする.このとき,対象物のロバスト性を考慮することで,仮組状態の複数物体を同時に扱う手法を提案する.作業対象としてパイプジョイントの組み立てを挙げ,簡易な道具を用いることで単腕マニピュレータで複数物体を同時に把持できることを示す.さらに,作業成功率の向上のために RGB-D カメラを用いた物体の位置検出に基づくロボット制御及び動作計画を実装する.
This paper discusses assembly operations using a single manipulator and a parallel gripper to simultaneously
grasp multiple objects and hold the group of temporarily assembled objects. Multiple robots and jigs generally operate
assembly tasks by constraining the target objects mechanically or geometrically to prevent them from moving. It is
necessary to analyze the physical interaction between the objects for such constraints to achieve the tasks with a single
gripper. In this paper, we focus on assembling pipe joints as an example and discuss constraining the motion of the
objects. Our demonstration shows that a simple tool can facilitate holding multiple objects with a single gripper.
単腕マニピュレータによる 複数物体の同時組み立ての 基礎的考察 / Basic Approach to Robotic Assembly of Multi...
XML と PHP のイケナイ関係 (セキュリティ的な意味で) -Introduction of XXE attack and XML Bomb with PHP-
1. XML と PHP のイケナイ関係
(セキュリティ的な意味で)
-Introduction of XXE attack and XML Bomb with PHP-
(English subtitles are available!)
Kousuke Ebihara (海老原昂輔)
<kousuke@co3k.org>
2. • Kousuke Ebihara (海老原昂輔) a.k.a @co3k
• work at 株式会社手嶋屋
• a developer of OpenPNE
• coding, code-review, performance, security
• PHP カンファレンス初参戦
自己紹介
-Introduction-
3. XMLExtensible Markup Language
RSS / ATOM / XHTML / SVG /
OpenDocument / XLIFF / KML /
XSLT / SOAP / SAML / XUL /
OpenSearch ...
libxml2 is free software license (MIT license),
not JSON license :) (You can use for Evil)
4. 意識調査
-PHPer Survey-
a)Do you know XML External Entity Injection (XXE)
and / or XML Entity Expansion (XML Bomb, Billion Laughs
Attack)?
b)Have you used XML-related extensions (PHP DOM,
SimpleXML, XMLReader, ...)?
7. • シンプルな RSS リーダ
A simple RSS Reader
• 任意の RSS / ATOM の URL を受け入れる
accepts any of URLs for RSS or ATOM feed
• 各エントリのタイトルを列挙するだけ
enumerates titles of per entries
• URL が HTTP プロトコルのものであること、サイズが 1MB 以内
であること、 整形式の XML としてパースできることをチェックする
makes sure a protocol of the URL is HTTP, the contents can be parsed as well-formed XML
and its length is within 1 MB
デモの想定
-About demo script-
8. • You can get all of the code from : https://github.com/ebihara/phpcon2013-ja-
xxe-demo/blob/master/web/index.php#L85
パース部分のコード
-Parse-
9. • You can get all of the code from : https://github.com/ebihara/phpcon2013-ja-
xxe-demo/blob/master/template/success.twig
出力部分のコード
-Output-
22. 復習: エンティティ
-REVIEW about Entity-
<strong>I ♥ PHP</strong>
<strong>I ♥ PHP</strong>
•エンティティの指す値は DTD (文書型定義) にて定義される
Entity-referenced value is defined in DTD
•エンティティ値の定義にはエンティティを含むことができる
Definition of entity value can contain entity
•URI の示す値をエンティティ値の定義に用いることができる (外部エンティティ宣言)
URI reference can be used in definition of entity value (External Entity Declaration)
30. XXE の脅威 (概要)
-Threats of XXE (summaries)-
• ローカルファイルの漏洩
Disclosure of server local file
• いわゆる SSRF (Server Side Request Forgery) による脅威全般
• 認証をおこなわない API やプロトコルを経由した非公開情報の漏洩
Disclosure of private informations via non-authenticated API or protocol
• 内部ネットワークに対するポートスキャン
Port scanning to the internal network
• オープン済みファイルディスクリプタへのアクセス
Access to opened file descriptors
• syslog 等のログを汚染
Poisoning syslog
31. Amazon S3 の非公開ファイルにアクセスする
-DEMO: Access to private file in Amazon S3-
$s3 = S3Client::factory([
'key' => '*****',
'secret' => '*****',
);
$s3->registerStreamWrapper();
echo file_get_contents('s3://phpcon/ebihara-150x150.jpg');
s3:// スキームで Amazon S3 のコンテンツを取得可能に
Now we can fetch Amazon S3 content via s3:// scheme.
32. Amazon S3 の非公開ファイルにアクセスする
-DEMO: Access to private file in Amazon S3-
<!ENTITY kurorekishi SYSTEM
"php://filter/convert.base64-encode/
resource=s3://phpcon/ebihara-kaicho.jpg">
• S3 用のストリームラッパーを利用して非公開ファイルの内容を取得
Get private file content by using Amazon S3 Stream Wrapper
• エンティティ値を展開した後の文書全体を整形式の XML としてパースできるように
する必要があるので、そのままではバイナリ値は取得できない
Entity values are embedded as a part of XML document and it should be parsed as well-formed XML since we can’t
get raw binary data
• そのため、 PHP のストリームフィルタ機能を用いて base64 エンコードを施した上で
出力
So the above example outputs base64 encoded binary data by using PHP’s stream filter features
33. 攻撃用フィードその 2
-The exploit feed part 2-
• You can get this XML from : https://gist.github.com/ebihara/
a653693e07f9795bc525
54. 影響を受ける PHP 拡張
-Which PHP extensions are affected?-
• DOM
• SimpleXML
• XMLReader (but the default is safe :)
• SOAP (but fixed as CVE-2013-1635 :)
*NOTE: Oh, xml_parser(), is a legacy XML Parser funcion, is safe!! :) Because it
doesn’t use libxml2 to parse XMLs
55. 脆弱な DOM 拡張の使用例
-An example of vulnerable code using DOM-
// vulnerable
$dom->load('/path/to/evil.xhtml');
$dom->loadXML('<?xml ...');
// safe
$dom->loadHTMLFile('/path/to/evil.xhtml');
$dom->loadHTML('<!DOCTYPE ...');
56. HTML5 with libxml2
• 現状の libxml2 は HTML5 をサポートしていない
The current version of libxml2 doesn’t support HTML5 yet
• HTML4 でサポートされていない要素や属性を用いたドキュメントを
DOMDocument::loadHTML() でパースすると警告が発生する
Warnings are occurred when DOMDocument::loadHTML() parses documents which have non-
HTML4 elements or attributes
• 警告を無視するか、 XXE への対策を入れて XML としてパースする
必要がある
So we need ignore such warnings or parse as XML with against XXE
See Also: PHP DOM (libxml2) only understands XHTML4, misinterprets HTML5, but D8 must cope with HTML5
[#1333730] | Drupal
57. 脆弱な SimpleXML 拡張の使用例
-An example of vulnerable code using SimpleXML-
$xml = new SimpleXMLElement('<?xml ...');
$xml = new SimpleXMLElement('/path/to/evil.xml',
null, true);
$xml = simplexml_load_string('<?xml ...');
$xml = simplexml_load_file('/path/to/evil.xml');
58. 脆弱な XMLReader 拡張の使用例
-An example of vulnerable code using XMLReader-
XMLReader 使用時に�SUBST_ENTITIES オプションを指
定しない限り XXE や XML Bomb の影響を受けない。
You can avoid XXE and XML Bomb to use XML Reader without
SUBST_ENTITIES option.
$xml = new XMLReader();
$xml->xml($content);
$xml->setParserProperty(
XMLReader::SUBST_ENTITIES
, true);
60. Wait, what version of
libxml2 on this demo?
2008/04/08 にリリースされた
libxml 2.6 系の最終バージョンで、
とうの昔のバージョンだしサンプルと
しては適さない……?
The final version of libxml 2.6 series, is released 2008/04/08,
so this is very old and arbitrary example ... really?
61. • libxml 2.7.0 (2008/08/30)
• strengthen some of the internal parser limits ...
(https://git.gnome.org/browse/libxml2/commit/?id=8915c)
• libxml 2.7.3 (2009/01/18)
• XML_MAX_TEXT_LENGHT (sic) limiting the maximum
size of a single text node, the defaultis (sic) 10MB ...
(https://git.gnome.org/browse/libxml2/commit/?id=1fb2e)
History of
libxml2 vs XML Bomb
62. • libxml 2.9.0 (2012/09/11)
• Do not fetch external parsed entities ...
(https://git.gnome.org/browse/libxml2/commit/?id=4629e)
History of
libxml2 vs XXE
64. ディストリビューションの配布パッケージ
-Situations of distributions’ provided libxml2-
libxml 2.6
(XXE & XML Bomb)
libxml 2.7, 2.8
(XXE)
libxml 2.9
CentOS
Debian
Fedora
Ubuntu
5 6
Wheezy(stable)
Squeeze(old stable)
Sid (unstable)
19
18
10.04 (LTS)
12.04 (LTS)
12.10
13.04
13.10
パッチによって
XML Bomb は
対策済
This XML Bomb vuln is
patched now in RHEL :)
65. ディストリビューション配布版 libxml2 と XXE
-Distributions’ libxml2 and XXE-
• CentOS (RHEL)
• I’ve confirmed fix XEE vuln in CentOS 6
• Ubuntu (Patched in Jul, 2013)
• Bug #1194410 “Apply upstream patch to close XXE vulnerability in...” : Bugs : “libxml2”
package : Ubuntu
• Debian (Patched in ... Mar, 2013?)
• I’ve NOT confirmed fix XEE vuln in stable (2.8.0+dfsg1-7+nmu1) and old-stable
• But ... it looks like that Debian project patched stable and old-stable (security)
in http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=702260
71. PHP 側でできる XML Bomb 対策
-Against XML Bomb in PHP script-
72. PHP 側でできる XML Bomb 対策
-Against XML Bomb in PHP script-
73. • 一応いくつか軽減策を選ぶことはできますが……
Okay, okay, you can choose some mitigations...
• DOCTYPE が含まれていた時点で拒否
Deny DOCTYPE
• ENTITY 読み込み部分を自力で取り除く
Remove ENTITY inclusion by your self
PHP 側でできる XML Bomb 対策
-Against XML Bomb in PHP script-
74. • libxml_disable_entity_loader(true)
• PHP 側で外部エンティティを解決するときにこれが true
であれば処理を終了する
The external entity loader in PHP does not continue its process if this argument
is true
PHP 側でできる XXE 対策
-Against XEE in PHP script-
$old = libxml_disable_entity_loader(true);
$xml = new SimpleXMLElement($content);
libxml_disable_entity_loader($old);
77. External Entity resolution flow after
libxml_disable_entity_loader(true)
Parse XML
Parse DOCTYPE
Parse ENTITY declarations
Resolve external entities
http://
PHP entity loader
file:// expect:// php://
外部エンティティローダ
が無効なのでコンテンツ
が読み込めなくなった
External entity loader is disabled so it can not
load any contents
gopher://
78. • 【注意】 libxml_disable_entity_loader(true) は XML 「ファイル」を読
み込むことを期待した関数の動作も無効化します
WARN!!! libxml_disable_entity_loader(true) kills XML **file** loading functions!!!
(DOMDocument::load(), simplexml_load_file(), XMLReader::open())
• これらの関数は libxml2 の機能を使用して XML ファイルを読み込む
際にエンティティローダを流用しているため
Because they use the entity loader to load XML file in libxml2
• したがって、 libxml2 に依存した関数の外で事前に XML ファイルを
読み込んでおく必要があります
So you need to pre-load XML contents outside of libxml2 depended functions
PHP 側でできる XXE 対策
-Against XEE in PHP script-
82. • XML External Entity: PHP マニュアル (!)、 T.Terada さんの日記、 JVN などの各
製品のアドバイザリ
• XXE: ほぼ関係ない検索結果 (先の T.Terada さんの日記に対する徳丸さんのはて
ブが引っかかるくらい)
• XML Bomb: 数サイトで言及が見られた他は JVN などのアドバイザリのみ
• XML Entity Expansion: JVN などのアドバイザリのみ
• Billion Laughs: MSDN での言及が見られるほかはアドバイザリのみ
• XML Entity Explosion: 水無月ばけらさんの日記など、 Rails でこの問題が発見さ
れた際の言及がいくらか見られる程度
おしえて Google 先生!
(No English subtitles)
83. XXE or XML Bomb in CVE
• Search CVEs by '"xml external entity" OR "xml entity expansion" OR "xml entity explosion" OR "xml bomb" OR
"billion laughs" OR "xxe" OR "xee" (And removed about XXE archive by my hand)
• And for 2013, I know that contains reserved CVE-2013-4333, CVE-2013-4334 and CVE-2013-4335 are XXE
Problem because I’m reporter of these problem, so I’ve added +3
0
5
10
15
20
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
17
14
6
1
3
2
00
1
0
11
84. History (2002, 2003)
• First report of XXE: http://
www.securityfocus.com/archive/1/297714
(Bugtraq, 2002/10)
• First report of XML Bomb: http://lists.xml.org/
archives/xml-dev/200302/msg00022.html ([xml-
dev], 2003/02)
85. History (2005-2011)
• 2005
Adobe Reader / Acrobat (CVE-2005-1306)
• 2008
Java (CVE-2008-0628), Ruby (CVE-2008-3790)
• 2009
WebKit (CVE-2009-1699), Adobe Reader / Acrobat
(CVE-2009-2979)
• 2011
Mac OS X (CVE-2011-0212), phpMyAdmin (CVE-2011-4107)
87. History (2013)
• PHP(CVE-2013-1643), Python(CVE-2013-1664, CVE-2013-1665),
Ruby(CVE-2013-1821), Ruby on Rails(CVE-2013-1856),
mod_security(CVE-2013-1915), MediaWiki, Play Framework
WordPress(CVE-2013-2202), Microsoft Office (CVE-2013-3160)
• And I reported the following (discovered):
OpenPNE(CVE-2013-4333, 2013-4334, 2013-4335)
PHP OpenID Library(CVE-2013-4701)