How develop new applications to HUE

1. Developing eXtensions for HUE
Maksym Doroshenko
Lviv Hadoop User Group 19 February 2015

2. Agenda
● What is HUE?
● Why use it?
● HUE architecture
● HUE features
● Add extensions to HUE

3. ● Aggregates the most common
Hadoop components into single
interface
● Simplify and integrate
● Free and open source
Goal of HUE

4. Apps

5. Hive & Pig

6. Metastore & HDFS

7. Additional applications

8. Backend
Python + Django
(2.6+ / 1.4.5)
Frontend
jQuery
Bootstrap
Knockout.js
Stack

9. View from 30K feet
You
and your
colleagues
Hue, as a "container" web application, sits in between your
Hadoop installation and the browser. It hosts all the Hue Apps,
including the built-in ones, and ones that you may write
yourself.

10. The HUE Server
Django running on WSGI
container/web server:
● manages the url dispatch
● executes app logic code
● puts together the views
from their templates
● use DB to manage
session data and Hue
apps can use it as well
for their “models”
Some Hue applications run
daemon processes "on the
side". For example, Beeswax
runs a daemon
("beeswax_server") that keeps
track of query states.

11. Install SDK app to HUE
For example we use CDH 5.3.
1. Go to HUE directory – cd /usr/lib/hue
2. Run "create_desktop_app" to Set up a New Source Tree –
“./build/env/bin/hue create_desktop_app calculator”
3. Install SDK application – “./build/env/bin/python
tools/app_reg/app_reg.py --install calculator --relative-
paths”
The "app_reg.py" tool manages the applications that are installed.

12. Result of installation
After installing we restart HUE – “sudo service hue restart”
And then visit http://127.0.0.1:8888/calculator/ you should see:

13. Issue to solve
Suppose we need track history of workflow
and for that we need some dashboard with
specific logic which we have not in HUE.

14. Create Django application
setup(
name="dashboard",
version="0.1",
description='Dashboard',
author='Maksym Doroshenko',
packages=find_packages('src'),
package_dir={'': 'src'},
install_requires=['setuptools', 'desktop'],
entry_points={'desktop.sdk.application': 'dashboard=dashboard'},
package_data={
'ctc_sla_dashboard': expand_package_data(
["src/dashboard/templates", "src/dashboard/static"],
"src/dashboard/")
}
)

15. Walk-through of a Django view

16. How add your application to HUE ?
As the previous one :)
1. Copy your application to HUE directory
(or use path to application in next step)
2. Install application –
“./build/env/bin/python
tools/app_reg/app_reg.py --
install dashboard --relative-
paths”

18. Our results

19. Dashboard example

20. Accessing HDFS
The “webhdfs.py” module give the file system object that exposes
operations that manipulate HDFS.
It is pre-configured to access HDFS as the user that's currently logged in.
Operations available on FS object are similar to the file operations typically
available in python.
The list of functions available is as follows:
● chmod, chown
● exists
● isdir, isfile
● listdir (and listdir_stats)
● mkdir
● open (which exposes a file-like object with read(), write(), seek() methods)
● remove, rmdir, rmtree

21. Web apps in any language
To create a new app:
● sudo build/env/bin/hue create_proxy_app Tableau
https://online.tableausoftware.com/
● sudo tools/app_reg/app_reg.py --install Tableau --relative-paths
If you want to update the url later, change it in the ini:
[Tableau]
url=http://new_url

22. Show time

23. Q&A