Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016 DataTorrent
David Yan
PMC Member, Apache Apex
Architect, DataTorrent
June 1, 2016
dtGateway
Making Apache Apex oper...
© 2016 DataTorrent
What is dtGateway
2
• Web server communicating in JSON via REST API
• You can use the REST API to:
Get ...
© 2016 DataTorrent
Links
3
• dtGateway Overview:
http://docs.datatorrent.com/dtgateway/
• dtGateway REST API Specification...
© 2016 DataTorrent
Internal Dynamics
4
HDFS
Apex STRAMs
Node
Managers
Resource Manager
dtGateway
REST over
HTTP
REST over
...
© 2016 DataTorrent
Security and Access Control
5
• dtGateway by default runs under a dedicated UNIX user “dtadmin”
• Suppo...
© 2016 DataTorrent
Use Case for Monitoring: Ad Tech
6
© 2016 DataTorrent
Use Case for Monitoring: Ad Tech
7
Metrics captured using REST API
● Number of Planned Containers and A...
© 2016 DataTorrent
Resources
8
• Apache Apex website - http://apex.apache.org/
• Subscribe - http://apex.apache.org/commun...
© 2016 DataTorrent
We Are Hiring
9
• jobs@datatorrent.com
• Back-End Engineers
• Front-End Engineers
• QA Automation Engin...
Upcoming SlideShare
Loading in …5
×

dtGateway: Making Apache Apex Operable

3,505 views

Published on

David Yan, PMC Member, Apache Apex & Architect, DataTorrent

To make Apache Apex operable, DataTorrent RTS includes a component called dtGateway, which provides a comprehensive REST API to let users query for cluster and application metrics, manage application packages, monitor running applications, etc. This talk will introduce features and cover the internal dynamics of dtGateway.

Published in: Technology
  • Be the first to comment

dtGateway: Making Apache Apex Operable

  1. 1. © 2016 DataTorrent David Yan PMC Member, Apache Apex Architect, DataTorrent June 1, 2016 dtGateway Making Apache Apex operable
  2. 2. © 2016 DataTorrent What is dtGateway 2 • Web server communicating in JSON via REST API • You can use the REST API to: Get performance metrics of Apex application instances, operators, containers Retrieve container logs, dynamically change operator properties, and change the DAG of a running Apex application Manage Apex application packages Launch and kill applications Query and visualize application data in real time • Powers the DataTorrent RTS Web UI
  3. 3. © 2016 DataTorrent Links 3 • dtGateway Overview: http://docs.datatorrent.com/dtgateway/ • dtGateway REST API Specification: http://docs.datatorrent.com/dtgateway_api/
  4. 4. © 2016 DataTorrent Internal Dynamics 4 HDFS Apex STRAMs Node Managers Resource Manager dtGateway REST over HTTP REST over HTTP Hadoop Java API Hadoop Java API HTTP Apex Worker Containers RP C This is awesome!
  5. 5. © 2016 DataTorrent Security and Access Control 5 • dtGateway by default runs under a dedicated UNIX user “dtadmin” • Supports front-end authentication: password, Kerberos, LDAP, PAM, ActiveDirectory and generic JAAS • Supports back-end Kerberos handshake with Hadoop components when Hadoop secure mode is turned on. • When authentication is enabled, dtGateway will impersonate the logged-in user when launching applications • RBAC: Users are assigned to roles, and each role has a set of permissions. • For more info on security in dtGateway: http://docs.datatorrent.com/dtgateway_security/
  6. 6. © 2016 DataTorrent Use Case for Monitoring: Ad Tech 6
  7. 7. © 2016 DataTorrent Use Case for Monitoring: Ad Tech 7 Metrics captured using REST API ● Number of Planned Containers and Allocated Containers ● Total memory allocated ● Latency, both end-to-end and at individual operator level ● Number of failed containers ● Application Status (Fire alert if FAILED, KILLED) ● Operator status - the # operators in PENDING_DEPLOY state Open Source Example: https://github.com/DataTorrent/examples/tree/master/tools
  8. 8. © 2016 DataTorrent Resources 8 • Apache Apex website - http://apex.apache.org/ • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • Startup Program – Free Enterprise License for Startups, Educational Institutions, Non-Profits - https://www.datatorrent.com/startups/ • Cloud Trial - https://www.datatorrent.com/download/cloud-trial/
  9. 9. © 2016 DataTorrent We Are Hiring 9 • jobs@datatorrent.com • Back-End Engineers • Front-End Engineers • QA Automation Engineers • Solutions Engineers • Build and Release Engineers

×