Automate That! Scripting Atlassian applications in Python


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • FIS is part of the S&P 500 and is one of the world's top-ranked technology providers to the banking industry.
  • Three basic scenarios where scripting is useful
  • JIRA – JQL provides amazing ability to search for issues. The presentation choices are limited, however, particularly if you want a report that you can email to others.
  • We have 2,000 users, so we tend to value server stability
  • All examples here in python 2
  • No proper error handling in any of these examples
  • Note that fields with wiki markup are returned as-is
  • Here, we’re going to use cookies. The JSESSIONID cookie can be used interchangeably between REST and non-REST callsYou can also use basic auth
  • No fields specified, so all fields are returned for all issues
  • Currently not possible to search based on an existing filter using REST
  • No fields are returned…. Just a list of the issues
  • Custom fields are listed alongside system fields
  • The contents of the “value” are highly dependent on the field type
  • The revisions “size” attribute tells us how many files were committed in that changeset
  • In practice, it seems that changesets are returned in decreasing order. The documentation doesn’t specify any order, however, so we’ll make no assumptions here
  • Bamboo allows basic auth, but you have to supply “os_authType=basic” as a query parameter in addition to the basicauth header. Here, we elect to exercise the user interface and obtain a cookie instead.
  • Just asking for the last 10 build results – would really want to loop backwards a chunk at a timeExpanding the related jira issues – can expand other fields as well Requesting the results in xml. Json also available
  • Note that we can also expand comment, labels, and stages (in addition to related issues)
  • Automate That! Scripting Atlassian applications in Python

    1. 1.
    2. 2. Has this happened to you?<br />Email to users results in 50+ undeliverable<br />Need to verify the users in Active Directory<br />Then “deactivate” former employees in Crowd<br />750 mouse clicks later, you’re done!<br />2<br /><br />
    3. 3. 3<br />Automate That!<br />
    4. 4. Agenda<br />Use cases for scripting<br />Atlassian APIs available for scripting<br />The awesome power and simplicity of python<br />Examples<br />4<br />
    5. 5. When is scripting useful?<br />Automate time consuming tasks<br />Perform data analysis<br />Cross-reference data from multiple systems<br />5<br />
    6. 6. Some specific use cases<br />Crowd – Deactivate Users and remove from all groups<br />Bamboo – Disable all plans in a project<br />JIRA – Release Notes<br />Subversion – custom commit acceptance<br />Custom build processes – pull code linked to a specific issue into a patch archive<br />6<br />
    7. 7. Why Scripts?Why Not Plugins?<br />7<br /><ul><li>I’m not a Java Developer
    8. 8. Installing new plugins can require a restart
    9. 9. Prefer to minimize ad hoc changes on the server
    10. 10. Need to correlate information from several systems
    11. 11. Need an agile process to accommodate changing requirements</li></li></ul><li>APIs for scripting(that we avoid if possible)<br />The user interface<br />Can do anything a user can do<br />Reporting tasks are relatively easy (particularly when xml is available)<br />Actions are relatively hard (and prone to breakage)<br />Capture browser traffic with livehttpheaders, firebug, etc<br />Form token checking can be an obstacle<br />XML-RPC and SOAP<br />Relatively low-level interface<br />Many actions available<br />Relatively complex to use<br />8<br />
    12. 12. More APIs for scripting(the ones we prefer to use)<br />RESTful Remote APIs (now deprecated)<br />High level interface<br />Supports a handful of actions<br />Now emerging: “real” REST interfaces<br />High level interface<br />Supports a handful of actions<br /><br />9<br />
    13. 13. Why Python?<br />Powerful standard libraries<br />Http(s) with cookie handling<br />XML and JSON<br />Unicode<br />Third Party Libraries<br />SOAP<br />REST<br />Templates<br />Subversion<br />Portable, cross-platform<br />10<br />
    14. 14. Python Versions<br />2.x<br />Ships with most linux distributions<br />Lots of third-party packages available<br />3.x<br />Latest version<br />Deliberately incompatible with 2.x<br />Not as many third-party libraries<br />11<br />
    15. 15. HTTP(s) with Python<br />Python 2<br />httplib – low level, all HTTP verbs<br />urllib – GET and POST, utilities<br />urllib2 – GET and POST using Request class, easier manipulation of headers, handlers for cookies, proxies, etc.<br />Python 3<br />http.client – low level, all HTTP verbs<br />http.parse - utilities<br />urllib.request – similar to urllib2<br />Third-Party<br />httplib2 – high-level interface with all HTTP verbs, plus caching, compression, etc.<br />12<br />
    16. 16. Example 1JIRA Issue Query & Retrieval<br />13<br />
    17. 17. 14<br />Discovering URLs for XML<br />
    18. 18. Simple Issue Retrieval<br />15<br />import urllib, httplib<br />import xml.etree.ElementTree as etree<br />jira_serverurl = ''<br />jira_userid = 'myuserid'<br />jira_password = 'mypassword'<br />detailsURL = jira_serverurl + <br /> "/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + <br /> "?os_username=" + jira_userid + "&os_password=" + jira_password<br />f = urllib.urlopen(detailsURL)<br />tree=etree.parse(f)<br />f.close()<br />Construct a URL that looks like the one in the UI, with extra parms for our user auth<br />Open the URL with one line!<br />Parse the XML with one line!<br />
    19. 19. Find details in XML<br />16<br />Find based on tag name or path to element<br />details = tree.getroot()<br />print "Issue: " + details.find("channel/item/key").text<br />print "Status: " + details.find("channel/item/status").text<br />print "Summary: " + details.find("channel/item/summary").text<br />print "Description: " + details.find("channel/item/description").text<br />Issue: JRA-9<br />Status: Open<br />Summary: User Preference: User Time Zones<br />Description: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p><br />
    20. 20. Behind the scenes…cookies!<br />17<br />Turn on debugging and see exactly what’s happening<br />httplib.HTTPConnection.debuglevel= 1<br />f = urllib.urlopen(detailsURL)<br />send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0rnHost: jira.atlassian.comrnUser-Agent: Python-urllib/1.17rnrn'<br />reply: 'HTTP/1.1 200 OKrn'<br />header: Date: Wed, 20 Apr 2011 12:04:37 GMT<br />header: Server: Apache-Coyote/1.1<br />header: X-AREQUESTID: 424x2804517x1<br />header: X-Seraph-LoginReason: OK<br />header: X-AUSERNAME: myuserid<br />header: X-ASESSIONID: 19b3b8o<br />header: Content-Type: text/xml;charset=UTF-8<br />header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/<br />header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f<br />8434fde36326fb340e8b7|lin; Path=/<br />header: Connection: close<br />JSESSIONID cookie sent from JIRA<br />
    21. 21. Authentication<br />User credentials determine:<br />The data returned<br />The operations allowed<br />Methods Available:<br />Basic Authentication<br />JSESSIONID Cookie<br />Token Method<br />18<br />
    22. 22. Basic Authentication<br />Authentication credentials passed with each request<br />Can be used with REST API<br />19<br />
    23. 23. JSESSIONID Cookie<br />Authentication credentials passed once; then cookie is used<br />Used when scripting the user interface<br />Can be used with REST API for JIRA, Confluence, and Bamboo<br />20<br />
    24. 24. Token Method<br />Authentication credentials passed once; then token is used<br />Used with Fisheye/Crucible REST<br />Used with Deprecated Bamboo Remote API<br />21<br />
    25. 25. Obtaining a cookie<br />Scripting the user interface login page<br />Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword”<br />Using the JIRA REST API<br />22<br />
    26. 26. JIRA REST Authentication<br />23<br />import urllib, urllib2, cookielib, json<br /># set up cookiejar for handling URLs<br />cookiejar = cookielib.CookieJar()<br />myopener= urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))<br />creds = { "username" : jira_userid, "password" : jira_password }<br />queryurl = jira_serverurl + "/rest/auth/latest/session"<br />req = urllib2.Request(queryurl)<br />req.add_data(json.dumps(creds))<br />req.add_header("Content-type", "application/json")<br />req.add_header("Accept", "application/json")<br />fp= <br />fp.close()<br />urllib2 handles cookies automatically. We just need to give it a CookieJar<br />Request and response are both JSON<br />We don’t care about response, just the cookie<br />
    27. 27. Submitting a JIRA Querywith the user interface<br />24<br /># Search using JQL<br />queryJQL = urllib.quote("key in watchedIssues()")<br />queryURL = jira_serverurl + <br /> "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + <br /> "?tempMax=1000&jqlQuery=" + queryJQL<br />fp =<br /># Search using an existing filter<br />filterId = "20124"<br />queryURL = jira_serverurl + <br /> "/sr/jira.issueviews:searchrequest-xml/" + <br /> "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId)<br />fp =<br />Pass any JQL Query<br />Or Pass the ID of an existing shared filter<br />
    28. 28. A JQL Query using REST<br />25<br /># Search using JQL<br />queryJQL= "key in watchedIssues()"<br />IssuesQuery= {<br /> "jql" : queryJQL,<br /> "startAt" : 0,<br /> "maxResults" : 1000 }<br />queryURL = jira_serverurl + "/rest/api/latest/search"<br />req = urllib2.Request(queryURL)<br />req.add_data(json.dumps(IssuesQuery))<br />req.add_header("Content-type", "application/json")<br />req.add_header("Accept", "application/json")<br />fp=<br />data = json.load(fp)<br />fp.close()<br />Pass any JQL Query<br />Request and response are both JSON<br />
    29. 29. XML returned from user interface query<br />26<br />An RSS Feed with all issues and requested fields that have values<br />
    30. 30. JSON returnedfrom a REST query<br />27<br />{u'total': 83, <br />u'startAt': 0, <br />u'issues': [{u'self': u'', <br />u'key': u'JRA-23969'}, <br /> {u'self': u'', <br />u'key': u'JRA-23138'}, <br /> {u'self': u'', <br />u'key': u'BAM-2770'}, <br /> {u'self': u'', <br />u'key': u'BAM-2489'}, <br /> {u'self': u'', <br />u'key': u'BAM-1410'}, <br /> {u'self': u'', <br />u'key': u'BAM-1143'}], <br />u'maxResults': 200}<br />A list of the issues found, with links to retrieve more information<br />
    31. 31. JSON issue details<br />28<br />All applicable fields are returned, even if there’s no value<br />Expand the html property to get rendered html for description, comments<br />
    32. 32. What’s the difference?<br />29<br /><reporter username="mlassau">Mark Lassau [Atlassian]</reporter><br /><customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented"><br /> <customfieldname>Last commented</customfieldname><br /> <customfieldvalues><br /> 1 week ago<br /> </customfieldvalues><br /></customfield><br />u'reporter': {<br />u'type': u'com.opensymphony.user.User', <br />u'name': u'reporter', <br />u'value': {<br />u'self': u'', <br />u'displayName': u'MarkLassau [Atlassian]', <br />u'name': u'mlassau'}}, <br />u'customfield_10160': {<br />u'type': u'com.atlassian.jira.toolkit:dayslastcommented', <br />u'name': u'Last commented', <br />u'value': 604800}, <br />XML values are display strings<br />REST values are type-dependent<br />
    33. 33. REST vs. non-REST<br />30<br />REST<br /><ul><li>More roundtrips to query JIRA and get issue details
    34. 34. Returns all fields
    35. 35. Values require type-specific interpretation
    36. 36. Easier to transition issues
    37. 37. Easier to get info for projects, components</li></ul>Non-REST<br /><ul><li>Can query based on existing filter
    38. 38. XML returns only fields that contain values
    39. 39. Values always one or more display strings
    40. 40. Can do anything a user can do (with a little work)</li></li></ul><li>Example 2Cross-referencing JIRA, Fisheye, and Bamboo build results<br />31<br />
    41. 41. Which build resolved my issue?<br />Bamboo keeps track of “related issues” (based on issue IDs included in commit comments), but doesn’t know when issues are resolved.<br />If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue”<br />Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully.<br />32<br />
    42. 42. To Fisheye for related commits!<br />33<br />queryURL= FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + <br /> "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue)<br />req= urllib2.Request(queryURL)<br />auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password)<br />base64string = base64.encodestring(auth_string)[:-1]<br />req.add_header("Authorization", "Basic {0}".format(base64string))<br />response =<br />issuecommits=etree.parse(response).getroot()<br />response.close()<br />Query a specific fisheye repository for a commit with our JIRA issue ID in the comments<br />Use basic auth headers to authenticate<br />
    43. 43. Fisheye changesets returned<br />34<br /><results expand="changesets"><br /> <changesets><br /> <changeset><br /> <csid>130948</csid><br /> <date>2011-04-29T12:35:56.150-04:00</date><br /> <author>lc6081</author><br /> <branch>trunk</branch><br /> <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment><br /> <revisions size="1" /><br /> </changeset><br /> </changesets><br /></results><br />
    44. 44. Parsing the changesets<br />35<br />commits = []<br />for changeset in issuecommits.findall("changesets/changeset"):<br />commits.append(changeset.findtext("csid"))<br />commits.sort()<br />print "Highest commit is: " + commits[-1]<br />Highest commit is: 130948<br />
    45. 45. Logging into Bamboo<br />36<br />urllib2.HTTPCookieProcessor(cookiejar))<br />cookiejar = cookielib.CookieJar()<br />myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))<br />queryURL = bambooServer + "/userlogin!default.action“<br />params= urllib.urlencode({<br /> "os_username" : bambooUserid,<br /> "os_password" : bambooPassword})<br />response =, params) <br />response.close()<br />Using a POST to the user interface login screen to retrieve a JSESSIONID cookie<br />
    46. 46. Querying for build results<br />37<br /># Warning: This is a very resource-intensive operation. <br /># You should consider limiting the number of builds returned<br />queryURL= bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + <br /> "?expand=results[-10:-1].result.jiraIssues"<br />req = urllib2.Request(queryURL)<br />req.add_header("Accept", "application/xml")<br />response =<br />results=etree.parse(response).getroot()<br />response.close()<br />Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list<br />Request the related issues<br />Ask for XML<br />(JSON also available)<br />
    47. 47. Example (partial) build results<br />38<br /><results expand="results"><br /><link href="" rel="self" /><br /><results expand="result" max-result="25" size="46" start-index="0"><br /><result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful"><br /> <link href="" rel="self" /><br /> <buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime><br /> <buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime><br /> <buildRelativeTime>4 days ago</buildRelativeTime><br /> <vcsRevisionKey>4483</vcsRevisionKey><br /> <buildReason>Code has changed</buildReason><br /> <comments max-result="0" size="0" start-index="0" /><br /> <labels max-result="0" size="0" start-index="0" /><br /> <jiraIssues max-result="1" size="1" start-index="0"><br /> <issue iconUrl="" <br />issueType="Defect" key="MYJIRAPROJECT-1629" <br /> summary="Need to display an error message when balance is zero."><br /> <urlhref="" rel="self" /><br /> </issue><br /> </jiraIssues><br /> <stages max-result="1" size="1" start-index="0" /><br /></result><br /></results><br /></results><br />Can also expand comments, labels, and stages<br />jiraIssues property has been expanded here<br />
    48. 48. Walking through build results<br />39<br />for result in results.findall("results/result"):<br /> print result.get("key") + ":"<br /> print "tRevision: " + result.findtext("vcsRevisionKey")<br /> issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")]<br /> print "tIssues: " + ", ".join(issues)<br />MYPROJECT-MYPLAN-31:<br /> Revision: 4489<br /> Issues: MYJIRAPROJECT-1658<br />MYPROJECT-MYPLAN-30:<br /> Revision: 4486<br /> Issues: MYJIRAPROJECT-1630<br />MYPROJECT-MYPLAN-29:<br /> Revision: 4485<br />Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663<br />
    49. 49. Example 3Removing a user from a Crowd group<br />40<br />
    50. 50. Beyond GET and POST<br />41<br />Lower-level HTTPConnection needed<br />connection = httplib.HTTPConnection('')<br />operation = 'DELETE'<br />urlpath = "/rest/usermanagement/latest/user/group/direct" + <br /> "?username={0}&groupname={1}".format(userToRemove, fromGroup)<br />body = None<br />auth_string = '{0}:{1}'.format(crowdAppName,crowdAppPassword)<br />base64string = base64.encodestring(auth_string)[:-1]<br />headers = {'Authorization' : "Basic {0}".format(base64string)}<br />connection.request(operation, urlpath, body, headers)<br />response = connection.getresponse()<br />print response.status, response.reason<br />connection.close()<br />Authenticate as a Crowd Application<br />204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found <br />
    51. 51. A few loose ends<br />Be prepared to handle Unicode strings<br />Error handling – not shown here, but important!<br />Formatting output – several python libraries for handling templates are available<br />REST Interfaces – you can write your own!<br /><br />42<br />
    52. 52. Links for more information<br /><br /><br /><br /><br /><br /><br />43<br />