(ATS3-PLAT01) Recent developments in Pipeline Pilot

Uploaded on

This session will review significant enhancements to Pipeline Pilot in recent releases. Areas covered are: Professional client, administration, security, integration, databases, and collections …

This session will review significant enhancements to Pipeline Pilot in recent releases. Areas covered are: Professional client, administration, security, integration, databases, and collections (chemistry, next gen sequencing, documents and text, statistics, and imaging).

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. (ATS3-PLAT01) Recent developments in Pipeline Pilot Ton van Daelen Sr. Director, Platform Product Management ton.vandaelen@accelrys.com
  • 2. The information on the roadmap and future software development efforts areintended to outline general product direction and should not be relied on in makinga purchasing decision.
  • 3. Introduction• Target audience – Administrators – Developers• What will you learn? – What is new that affects me or my users (and me indirectly)• More of What’s New in – (ATS3-PLAT07) Pipeline Pilot Tips, Tricks and Challenges• Slides and examples will be made available
  • 4. Topics• Release history• Administration – Performance enhancements – Server monitoring – Usage monitoring – Protocol validation – Security• Integration – .NET components – Python (prototype)• SharePoint integration
  • 5. Pipeline Pilot Release History• Pipeline Pilot 8.0 – Dec 2010 – Design mode 1-4 – Local subprotocols – Data flow monitoring 5.0 6.0• Pipeline Pilot 8.5 – Aug 2011 – Support growing deployments 7.0 – Isentris and Notebook integration 8.0 8.5 – Support for building ‘apps’• Pipeline Pilot 8.5 CU1 – Nov 2011 – Protocol comparison 9.0 – Performance
  • 6. Science and Applications Release History Imaging (HCS) Reporting (Canvas elements) Advanced data modeling Chemistry (MDL formats) Reporting (Stylesheets) 7.5 Imaging (DICOM) Direct cartridge Database Search Imaging (Big Image Explorer) Chemistry (Content, NEMA, Cheshire) 8.0 Next generation sequencing 8.0 Isentris and Notebook integration ChemRegCollections BioRegApps 8.5
  • 7. Mini-Survey• What server version(s) are you running? – Pre-7.5? – 7.5 – 8.0 – 8.5
  • 8. Administration
  • 9. Performance Testing• Performance increasingly critical because of ‘apps’ – Both Accelrys and custom• Performance testing – Track our progress – Spot errors• How do we do this?
  • 10. Performance Testing
  • 11. Threads / Windows
  • 12. Throughput / Windows
  • 13. Linux / Threads and Throughput
  • 14. Pipeline Pilot Configurations Compared (ODBC Test Case) Latency of PP 7.5, 8.0 & 8.0 PoolID Latency of PP 8.0 & 8.0 PoolID 20000 1200 18000 16000 1000 Avg Response Time (ms) Avg Response Time (ms) 14000 800 12000 10000 600 8000 6000 ODBC latency is 44% Improved scalability 400 lower than 7.5 and under load 4000 22% lower than 8.0 2000 w/out pooled servers 200 0 1 5 10 20 50 100 0 ODBC 7.5 235 202 192 218 1558 18622 1 5 10 20 50 100 ODBC 8.0 224 166 164 156 482 1015 ODBC 8.0 224 166 164 156 482 1015 ODBC 8.0 (PoolID) 122 137 117 122 246 367 ODBC 8.0 (PoolID) 122 137 117 122 246 367
  • 15. Server Maintenance and Monitoring• Administration portal – Run diagnostics – View and cancel running jobs – View log files – Monitor CPU and memory• Pro client – Server status indicator
  • 16. Server Maintenance and Monitoring• External tools Windows Performance Counters Ganglia (Linux)
  • 17. Usage Monitoring• Why? – Internal accounting (‘Which dept is using imaging?’) – Justify investment (‘Are we using reporting?’) – Administrative (‘What protocols haven’t been used in 1 yr?’)• Approaches – Usage report in admin portal – Custom analysis of log files – Protocol database analysis using Component Reader• More in: (ATS3-PLAT11) Advanced Planning
  • 18. Admin Portal Usage Report
  • 19. Usage Report Example Introduced in 8.5. For pre 8.5 servers, install protocols from ftp://download.accelrys.com/protocols/SummaryReport.zip.
  • 20. Logfile Folder Installation logfiles. Remote logfiles. Allows for manual copying if server is not connected Local logfiles.
  • 21. Usage Log Analysis Through Protocols• Usage protocol in C:Program FilesAccelrysPPSappsscitegicadmindatarootxml C:Program FilesAccelrysPPSlogsusagePipeline Pilot.log
  • 22. Analysis Example Runs by User Runs by ClientType Most Frequent Protocols
  • 23. Analyzing the XMLDB• Questions to ask – Non-empty Password parameters? – Empty required parameter field? – File references? – Shortcuts as copies? – Protocol for a particular user? – Recent protocols? – Components of a certain collection? – Deprecated components?• Combined with log file – What components were used recently? – For a migration, what protocols were run recently?
  • 24. Analyzing the Xmldb Protocol info: • Author • Date • Component ID Component info: • Data source references • Passwords
  • 25. Component Parameters ParamName ParamValue ParamType Source dataImagesCardsREADME.txt URLType Maximum LongType SourceTag None StringType Keep Properties PropertyPreviewType HasColumnNames TRUE BoolType Delimiter FigureOut StringType MultipleValueDelimiter None StringType ParseNDimensionalArrayStrings TRUE BoolType Gotcha! Additional Options CommentLines GroupType StringType UndefinedString StringType Component Reader Example.xml Accelrys or custom AllLines FALSE BoolType TrimWhitespace TRUE BoolType component? CheckSyntax TRUE BoolType AddEmptyProperties FALSE BoolType OnGeneralError Halt StringType OnDataTypeError Halt Date registered StringType ComponentAttributes ComponentReturnsPass StringType Registrant scitegic/generic StringType RegistrationDate Sat Feb 26 16:31:03 2011 StringType ComponentDisplayName Read README.txt StringType DerivedFrom Delimited Reader StringType CustomErrorText Caption StringType ComponentIcon DocumentNode StringType ComponentDisabled 0 LongType ComponentGUID {7D2AF7C7-A077-4201-B10B-8E8283A5EADD} StringType ComponentPoint 90 -95 StringType ComponentHelp 100 StringType Component Sticky Notes RegistrationVersion Help text StringType StringType ComponentLocalID 0 LongType Component Path ComponentsData Access and ManipulationFile ReadersDelimited Text R StringType
  • 26. Component Parameter Interface
  • 27. Component Parameter Interface
  • 28. Protocol Validation• Inspect protocols before moving to production• Rules checking: – Hard-coded passwords – Bad file of database references – Broken shortcuts• Configurable for different deployment modes – Standard protocol, Reports, Web services, RSS Feeds, SharePoint, etc.
  • 29. Protocol Validation• Validation can be set up as hard requirement for publication• Operation is cancelled on validation errors*Not a standard feature. Request from support (ATS3-PLAT03) Enterprise Protocol & Package Deployment
  • 30. Automating Component Best Practices• Component Review-o-Matic Protocol – Checks Documentation – Checks Parameter Names• Create your own custom rulesSee: ComponentsDatabase and ApplicationIntegrationUtilitiesPrototypesUsabilityComponent Review-o-Matic
  • 31. Security
  • 32. Windows Integrated Authentication• Run Web Port without providing username /password• Supported via Kerberos and NTLM• Requirements – DOMAIN authentication – Kerberos only works on remote servers. Negotiation on local servers uses NTLM instead of Kerberos, so some restrictions may apply – Pipeline Pilot Apache service must have valid Service Provider Name on the DOMAIN server. Running as "Local System" usually fulfills this requirement – Configuration required on server and client’s browser
  • 33. Windows Integrated Authentication• Requires Active Directory settings• Enables Secure Services• Authentication provided by NTLM/Kerberos• Limitations – Delegation not supported – Accessing network resources (e.g., proxy server authentication with credentials, UNC paths) – Using PP credentials for components that require authentication (e.g., database, readers, integration, etc.) – Running protocols from within a protocol (e.g., RunProtocol PilotScript function, parallel subprotocols)
  • 34. Server and Client Configuration• Run Apache service as a user: – To set up Kerberos service principal name for the Pipeline Pilot server, run the following on the Domain controller: • setspn.exe -a http/SERVERNAME DOMAINapache_username – On the Pipeline Pilot server, use the Services tool and set the Logon User for the "scitegic_apache_8_5_0" service.• Client Configuration – Internet Explorer: • Add the server as a trusted site (Tools > Internet Options > Security > Trusted Sites > Custom Level > User Authentication > Logon). • Select Automatic logon with current user name and password. • If your server is already part of the Local Intranet, select Automatic logon only in Intranet zone. – Firefox: • Browse to "about:config" and add the server names to the following preferences: – network.negotiate-auth.trusted-uris – network.automatic-ntlm-auth.trusted-uris
  • 35. Pass-through (Impersonation) Pipeline Pilot Web Services Webform Basic Scientist User Token Database SciServer ODBC Web Services Basic
  • 36. Integration
  • 37. Pipeline Pilot Integration Points Web Browser Run Protocol .NET Client Java Client SOAP Client Professional Client Command Line Web Port JavaScript Client Client .NET Client JavaScript Client SDK Java Client SDK Web Services API SDK Pipeline Pilot Enterprise Server Web Apps Web Services API Admin Portal Help Portal Protocol Runtime Environment (scisvr) VB Script VB Script Run SOAP & Telnet / ODBC / (On (On Java Perl Python .NET SSH / SCP Program HTTP FTP JDBC Client) Server) Java Perl .NET VB Script Cmd Line Classes Scripts Classes REST / SOAP HTTP Cmd Line DBs Service Service
  • 38. Component Development Update• .NET On Server – Points to a compiled .NET Assembly• Dynamic .NET – Embed your .NET code directly into the Server Component• Leverage .NET supported languages – C# – VB.NET – IronPython
  • 39. A look at building a .NET Server Component
  • 40. .NET Components Demo• Hello world using dynamic C#• Hello world using compiled C#
  • 41. Python (Prototype)• Requires – ActiveState Python – .NET runtime• Windows only• Prototype now available – Looking for testers properties = component_datarecord.GetRoot().GetProperties() queryString = properties.GetByName(u"queryString").GetValue().GetString(); params = urllib.urlencode({corpID: queryString}) f = urllib.urlopen("http://vm-discoweb-dev.corp.mpi.com/rest/Compound/lookupCorpID/getCompoundProperties?%s" % params) properties.Define("Response", f.read());
  • 42. SharePoint
  • 43. Pipeline Pilot & SharePoint Web Services Run Job Web Parts (3) Pipeline Pilot SharePoint Bridge User Created IT Created Read/Write Document Library ISV Created Protocols Lists Pipeline Pilot SharePoint
  • 44. PP Scientifically Enables SharePoint
  • 45. Deploying Custom Web Parts• Problem/Need: Want to create Pipeline Pilot-based Web Parts and deploy to organization• Solution: Create packaged (“standalone”) Web Parts (Chapter 4, SharePoint Bridge Admin Guide)
  • 46. Using Custom Web Parts• Custom Web Parts are available in Web Part Gallery• Provides a much simpler mechanism for end-users
  • 47. SharePoint Document Library• Read from / write to document library – Use standard Reader/Writer components – With Domain Authentication, R/W will simply work – Alternatives • Copy between document repository and temporary local version • Add authentication options to Reader/Writer/Copy components
  • 48. Summary• Reasons to upgrade – Administrative – Development – Science and applications• Plan your upgrade – Engage Accelrys support and services – (ATS3-PLAT11) Advanced Planning• Resources – Component development guide – Accelrys Community: http://community.accelrys.com/• Other sessions of interest – (ATS03-PLAT07) Pipeline Pilot Tips, Tricks and Challenges
  • 49. The information on the roadmap and future software development efforts areintended to outline general product direction and should not be relied on in makinga purchasing decision.For more information on the Accelrys Tech Summits and other IT & Developerinformation, please visit:https://community.accelrys.com/groups/it-dev