Automate That! Scripting Atlassian applications in Python
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Automate That! Scripting Atlassian applications in Python

on

  • 7,405 views

 

Statistics

Views

Total Views
7,405
Views on SlideShare
6,974
Embed Views
431

Actions

Likes
7
Downloads
75
Comments
0

8 Embeds 431

http://summit.atlassian.com 284
https://summit.atlassian.com 111
http://www.atlassian.com 20
https://wacdev.internal.atlassian.com 9
url_unknown 2
https://extranet.atlassian.com 2
https://www.atlassian.com 2
http://magnolia-staging.private.atlassian.com:8080 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • FIS is part of the S&P 500 and is one of the world's top-ranked technology providers to the banking industry.
  • Three basic scenarios where scripting is useful
  • JIRA – JQL provides amazing ability to search for issues. The presentation choices are limited, however, particularly if you want a report that you can email to others.
  • We have 2,000 users, so we tend to value server stability
  • All examples here in python 2
  • No proper error handling in any of these examples
  • Note that fields with wiki markup are returned as-is
  • Here, we’re going to use cookies. The JSESSIONID cookie can be used interchangeably between REST and non-REST callsYou can also use basic auth
  • No fields specified, so all fields are returned for all issues
  • Currently not possible to search based on an existing filter using REST
  • No fields are returned…. Just a list of the issues
  • Custom fields are listed alongside system fields
  • The contents of the “value” are highly dependent on the field type
  • The revisions “size” attribute tells us how many files were committed in that changeset
  • In practice, it seems that changesets are returned in decreasing order. The documentation doesn’t specify any order, however, so we’ll make no assumptions here
  • Bamboo allows basic auth, but you have to supply “os_authType=basic” as a query parameter in addition to the basicauth header. Here, we elect to exercise the user interface and obtain a cookie instead.
  • Just asking for the last 10 build results – would really want to loop backwards a chunk at a timeExpanding the related jira issues – can expand other fields as well Requesting the results in xml. Json also available
  • Note that we can also expand comment, labels, and stages (in addition to related issues)

Automate That! Scripting Atlassian applications in Python Presentation Transcript

  • 1.
  • 2. Has this happened to you?
    Email to users results in 50+ undeliverable
    Need to verify the users in Active Directory
    Then “deactivate” former employees in Crowd
    750 mouse clicks later, you’re done!
    2
    http://www.flickr.com/photos/left-hand/4231405740/
  • 3. 3
    Automate That!
  • 4. Agenda
    Use cases for scripting
    Atlassian APIs available for scripting
    The awesome power and simplicity of python
    Examples
    4
  • 5. When is scripting useful?
    Automate time consuming tasks
    Perform data analysis
    Cross-reference data from multiple systems
    5
  • 6. Some specific use cases
    Crowd – Deactivate Users and remove from all groups
    Bamboo – Disable all plans in a project
    JIRA – Release Notes
    Subversion – custom commit acceptance
    Custom build processes – pull code linked to a specific issue into a patch archive
    6
  • 7. Why Scripts?Why Not Plugins?
    7
    • I’m not a Java Developer
    • 8. Installing new plugins can require a restart
    • 9. Prefer to minimize ad hoc changes on the server
    • 10. Need to correlate information from several systems
    • 11. Need an agile process to accommodate changing requirements
  • APIs for scripting(that we avoid if possible)
    The user interface
    Can do anything a user can do
    Reporting tasks are relatively easy (particularly when xml is available)
    Actions are relatively hard (and prone to breakage)
    Capture browser traffic with livehttpheaders, firebug, etc
    Form token checking can be an obstacle
    XML-RPC and SOAP
    Relatively low-level interface
    Many actions available
    Relatively complex to use
    8
  • 12. More APIs for scripting(the ones we prefer to use)
    RESTful Remote APIs (now deprecated)
    High level interface
    Supports a handful of actions
    Now emerging: “real” REST interfaces
    High level interface
    Supports a handful of actions
    http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassian+REST+API+Design
    9
  • 13. Why Python?
    Powerful standard libraries
    Http(s) with cookie handling
    XML and JSON
    Unicode
    Third Party Libraries
    SOAP
    REST
    Templates
    Subversion
    Portable, cross-platform
    10
  • 14. Python Versions
    2.x
    Ships with most linux distributions
    Lots of third-party packages available
    3.x
    Latest version
    Deliberately incompatible with 2.x
    Not as many third-party libraries
    11
  • 15. HTTP(s) with Python
    Python 2
    httplib – low level, all HTTP verbs
    urllib – GET and POST, utilities
    urllib2 – GET and POST using Request class, easier manipulation of headers, handlers for cookies, proxies, etc.
    Python 3
    http.client – low level, all HTTP verbs
    http.parse - utilities
    urllib.request – similar to urllib2
    Third-Party
    httplib2 – high-level interface with all HTTP verbs, plus caching, compression, etc.
    12
  • 16. Example 1JIRA Issue Query & Retrieval
    13
  • 17. 14
    Discovering URLs for XML
  • 18. Simple Issue Retrieval
    15
    import urllib, httplib
    import xml.etree.ElementTree as etree
    jira_serverurl = 'http://jira.atlassian.com'
    jira_userid = 'myuserid'
    jira_password = 'mypassword'
    detailsURL = jira_serverurl +
    "/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" +
    "?os_username=" + jira_userid + "&os_password=" + jira_password
    f = urllib.urlopen(detailsURL)
    tree=etree.parse(f)
    f.close()
    Construct a URL that looks like the one in the UI, with extra parms for our user auth
    Open the URL with one line!
    Parse the XML with one line!
  • 19. Find details in XML
    16
    Find based on tag name or path to element
    details = tree.getroot()
    print "Issue: " + details.find("channel/item/key").text
    print "Status: " + details.find("channel/item/status").text
    print "Summary: " + details.find("channel/item/summary").text
    print "Description: " + details.find("channel/item/description").text
    Issue: JRA-9
    Status: Open
    Summary: User Preference: User Time Zones
    Description: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>
  • 20. Behind the scenes…cookies!
    17
    Turn on debugging and see exactly what’s happening
    httplib.HTTPConnection.debuglevel= 1
    f = urllib.urlopen(detailsURL)
    send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0rnHost: jira.atlassian.comrnUser-Agent: Python-urllib/1.17rnrn'
    reply: 'HTTP/1.1 200 OKrn'
    header: Date: Wed, 20 Apr 2011 12:04:37 GMT
    header: Server: Apache-Coyote/1.1
    header: X-AREQUESTID: 424x2804517x1
    header: X-Seraph-LoginReason: OK
    header: X-AUSERNAME: myuserid
    header: X-ASESSIONID: 19b3b8o
    header: Content-Type: text/xml;charset=UTF-8
    header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/
    header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f
    8434fde36326fb340e8b7|lin; Path=/
    header: Connection: close
    JSESSIONID cookie sent from JIRA
  • 21. Authentication
    User credentials determine:
    The data returned
    The operations allowed
    Methods Available:
    Basic Authentication
    JSESSIONID Cookie
    Token Method
    18
  • 22. Basic Authentication
    Authentication credentials passed with each request
    Can be used with REST API
    19
  • 23. JSESSIONID Cookie
    Authentication credentials passed once; then cookie is used
    Used when scripting the user interface
    Can be used with REST API for JIRA, Confluence, and Bamboo
    20
  • 24. Token Method
    Authentication credentials passed once; then token is used
    Used with Fisheye/Crucible REST
    Used with Deprecated Bamboo Remote API
    21
  • 25. Obtaining a cookie
    Scripting the user interface login page
    Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword”
    Using the JIRA REST API
    22
  • 26. JIRA REST Authentication
    23
    import urllib, urllib2, cookielib, json
    # set up cookiejar for handling URLs
    cookiejar = cookielib.CookieJar()
    myopener= urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
    creds = { "username" : jira_userid, "password" : jira_password }
    queryurl = jira_serverurl + "/rest/auth/latest/session"
    req = urllib2.Request(queryurl)
    req.add_data(json.dumps(creds))
    req.add_header("Content-type", "application/json")
    req.add_header("Accept", "application/json")
    fp= myopener.open(req)
    fp.close()
    urllib2 handles cookies automatically. We just need to give it a CookieJar
    Request and response are both JSON
    We don’t care about response, just the cookie
  • 27. Submitting a JIRA Querywith the user interface
    24
    # Search using JQL
    queryJQL = urllib.quote("key in watchedIssues()")
    queryURL = jira_serverurl +
    "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" +
    "?tempMax=1000&jqlQuery=" + queryJQL
    fp = myopener.open(queryURL)
    # Search using an existing filter
    filterId = "20124"
    queryURL = jira_serverurl +
    "/sr/jira.issueviews:searchrequest-xml/" +
    "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId)
    fp = myopener.open(queryURL)
    Pass any JQL Query
    Or Pass the ID of an existing shared filter
  • 28. A JQL Query using REST
    25
    # Search using JQL
    queryJQL= "key in watchedIssues()"
    IssuesQuery= {
    "jql" : queryJQL,
    "startAt" : 0,
    "maxResults" : 1000 }
    queryURL = jira_serverurl + "/rest/api/latest/search"
    req = urllib2.Request(queryURL)
    req.add_data(json.dumps(IssuesQuery))
    req.add_header("Content-type", "application/json")
    req.add_header("Accept", "application/json")
    fp= myopener.open(req)
    data = json.load(fp)
    fp.close()
    Pass any JQL Query
    Request and response are both JSON
  • 29. XML returned from user interface query
    26
    An RSS Feed with all issues and requested fields that have values
  • 30. JSON returnedfrom a REST query
    27
    {u'total': 83,
    u'startAt': 0,
    u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969',
    u'key': u'JRA-23969'},
    {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138',
    u'key': u'JRA-23138'},
    {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770',
    u'key': u'BAM-2770'},
    {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489',
    u'key': u'BAM-2489'},
    {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410',
    u'key': u'BAM-1410'},
    {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143',
    u'key': u'BAM-1143'}],
    u'maxResults': 200}
    A list of the issues found, with links to retrieve more information
  • 31. JSON issue details
    28
    All applicable fields are returned, even if there’s no value
    Expand the html property to get rendered html for description, comments
  • 32. What’s the difference?
    29
    <reporter username="mlassau">Mark Lassau [Atlassian]</reporter>
    <customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented">
    <customfieldname>Last commented</customfieldname>
    <customfieldvalues>
    1 week ago
    </customfieldvalues>
    </customfield>
    u'reporter': {
    u'type': u'com.opensymphony.user.User',
    u'name': u'reporter',
    u'value': {
    u'self': u'http://jira.atlassian.com/rest/api/latest/user?username=mlassau',
    u'displayName': u'MarkLassau [Atlassian]',
    u'name': u'mlassau'}},
    u'customfield_10160': {
    u'type': u'com.atlassian.jira.toolkit:dayslastcommented',
    u'name': u'Last commented',
    u'value': 604800},
    XML values are display strings
    REST values are type-dependent
  • 33. REST vs. non-REST
    30
    REST
    • More roundtrips to query JIRA and get issue details
    • 34. Returns all fields
    • 35. Values require type-specific interpretation
    • 36. Easier to transition issues
    • 37. Easier to get info for projects, components
    Non-REST
    • Can query based on existing filter
    • 38. XML returns only fields that contain values
    • 39. Values always one or more display strings
    • 40. Can do anything a user can do (with a little work)
  • Example 2Cross-referencing JIRA, Fisheye, and Bamboo build results
    31
  • 41. Which build resolved my issue?
    Bamboo keeps track of “related issues” (based on issue IDs included in commit comments), but doesn’t know when issues are resolved.
    If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue”
    Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully.
    32
  • 42. To Fisheye for related commits!
    33
    queryURL= FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" +
    "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue)
    req= urllib2.Request(queryURL)
    auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password)
    base64string = base64.encodestring(auth_string)[:-1]
    req.add_header("Authorization", "Basic {0}".format(base64string))
    response = myopener.open(req)
    issuecommits=etree.parse(response).getroot()
    response.close()
    Query a specific fisheye repository for a commit with our JIRA issue ID in the comments
    Use basic auth headers to authenticate
  • 43. Fisheye changesets returned
    34
    <results expand="changesets">
    <changesets>
    <changeset>
    <csid>130948</csid>
    <date>2011-04-29T12:35:56.150-04:00</date>
    <author>lc6081</author>
    <branch>trunk</branch>
    <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment>
    <revisions size="1" />
    </changeset>
    </changesets>
    </results>
  • 44. Parsing the changesets
    35
    commits = []
    for changeset in issuecommits.findall("changesets/changeset"):
    commits.append(changeset.findtext("csid"))
    commits.sort()
    print "Highest commit is: " + commits[-1]
    Highest commit is: 130948
  • 45. Logging into Bamboo
    36
    urllib2.HTTPCookieProcessor(cookiejar))
    cookiejar = cookielib.CookieJar()
    myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
    queryURL = bambooServer + "/userlogin!default.action“
    params= urllib.urlencode({
    "os_username" : bambooUserid,
    "os_password" : bambooPassword})
    response = myopener.open(queryURL, params)
    response.close()
    Using a POST to the user interface login screen to retrieve a JSESSIONID cookie
  • 46. Querying for build results
    37
    # Warning: This is a very resource-intensive operation.
    # You should consider limiting the number of builds returned
    queryURL= bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" +
    "?expand=results[-10:-1].result.jiraIssues"
    req = urllib2.Request(queryURL)
    req.add_header("Accept", "application/xml")
    response = myopener.open(req)
    results=etree.parse(response).getroot()
    response.close()
    Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list
    Request the related issues
    Ask for XML
    (JSON also available)
  • 47. Example (partial) build results
    38
    <results expand="results">
    <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" />
    <results expand="result" max-result="25" size="46" start-index="0">
    <result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful">
    <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" />
    <buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime>
    <buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime>
    <buildRelativeTime>4 days ago</buildRelativeTime>
    <vcsRevisionKey>4483</vcsRevisionKey>
    <buildReason>Code has changed</buildReason>
    <comments max-result="0" size="0" start-index="0" />
    <labels max-result="0" size="0" start-index="0" />
    <jiraIssues max-result="1" size="1" start-index="0">
    <issue iconUrl="http://myjira.domain.com/images/icons/bug.gif"
    issueType="Defect" key="MYJIRAPROJECT-1629"
    summary="Need to display an error message when balance is zero.">
    <urlhref="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" />
    </issue>
    </jiraIssues>
    <stages max-result="1" size="1" start-index="0" />
    </result>
    </results>
    </results>
    Can also expand comments, labels, and stages
    jiraIssues property has been expanded here
  • 48. Walking through build results
    39
    for result in results.findall("results/result"):
    print result.get("key") + ":"
    print "tRevision: " + result.findtext("vcsRevisionKey")
    issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")]
    print "tIssues: " + ", ".join(issues)
    MYPROJECT-MYPLAN-31:
    Revision: 4489
    Issues: MYJIRAPROJECT-1658
    MYPROJECT-MYPLAN-30:
    Revision: 4486
    Issues: MYJIRAPROJECT-1630
    MYPROJECT-MYPLAN-29:
    Revision: 4485
    Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663
  • 49. Example 3Removing a user from a Crowd group
    40
  • 50. Beyond GET and POST
    41
    Lower-level HTTPConnection needed
    connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080')
    operation = 'DELETE'
    urlpath = "/rest/usermanagement/latest/user/group/direct" +
    "?username={0}&groupname={1}".format(userToRemove, fromGroup)
    body = None
    auth_string = '{0}:{1}'.format(crowdAppName,crowdAppPassword)
    base64string = base64.encodestring(auth_string)[:-1]
    headers = {'Authorization' : "Basic {0}".format(base64string)}
    connection.request(operation, urlpath, body, headers)
    response = connection.getresponse()
    print response.status, response.reason
    connection.close()
    Authenticate as a Crowd Application
    204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found
  • 51. A few loose ends
    Be prepared to handle Unicode strings
    Error handling – not shown here, but important!
    Formatting output – several python libraries for handling templates are available
    REST Interfaces – you can write your own!
    http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services
    42
  • 52. Links for more information
    http://confluence.atlassian.com/display/JIRA/Displaying+Search+Results+in+XML
    http://confluence.atlassian.com/display/JIRA/JIRA+REST+API+(Alpha)+Tutorial
    http://confluence.atlassian.com/display/CONFDEV/Confluence+REST+APIs
    http://confluence.atlassian.com/display/FECRUDEV/REST+API+Guide
    http://confluence.atlassian.com/display/BAMBOO/Bamboo+REST+APIs
    http://confluence.atlassian.com/display/CROWDDEV/Crowd+REST+APIs
    43