Talend Open-Source Approach Provides Holistic Integration Capability Across, Data, Devices, Services


Published on

Transcript of a sponsored BriefingsDirect podcast on enterprise integration and new tools to put control in the hands of "the masses."

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Talend Open-Source Approach Provides Holistic Integration Capability Across, Data, Devices, Services

  1. 1. Talend Open-Source Approach Provides Holistic IntegrationCapability Across, Data, Devices, ServicesTranscript of a sponsored BriefingsDirect podcast on enterprise integration and new tools to putcontrol in the hands of "the masses."Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:TalendDana Gardner: Hi. This is Dana Gardner, Principal Analyst at Interarbor Solutions, and youre listening to BriefingsDirect. Today, we present a sponsored podcast discussion on how the role and impact of integration has shifted, and how a more comprehensive and managed approach to integration is required, thanks to such major trends as cloud, hybrid computing, and managing massive datasets.Moreover, the tools that support enterprise integration need to be usable by more types ofworkers, those that are involved with business process activities and data analysis. The so-calleddemocratization of IT effect is also rapidly progressing into this traditionally complex andisolated world of applications and data integration. [Disclosure: Talend is a sponsor ofBriefingsDirect podcasts.]So, how do enterprises face up to the generational shift of the function of integration to new andmore empowered users, so that businesses can react and exploit more applications and dataresources and do so in a managed and governed fashion? This is no small task.Were finding that modern, lightweight, and open-source platforms that leverage modulararchitectures are a new and proven resource for the rapid and agile integration requirements.And, the tools that support these platforms have come a long way in ease of use and applicabilityto more types of activities.Were here today to discuss how these platforms have evolved, how the open-source projects arebeing produced and delivered into real-time and enterprise-ready, mission-critical use scenarios,and what’s now available to help make integration a core competency among more enterpriseapplication and data activities and processes.Please join me now in welcoming our guests today. Were here with Dan Kulp, the Vice Presidentof Open Source Development at Talend’s Application Integration Division and also the ProjectManagement Committee Chair of the Apache CXF Project. Welcome back to BriefingsDirect,Dan.Dan Kulp: It’s great to be here. Thank you.
  2. 2. Gardner: Were also here with Pat Walsh. He is the Vice President of Marketing in theApplication Integration Division at Talend. Hey, Pat.Pat Walsh: Nice to be here as well.Gardner: Pat, let me start with you. Were talking about a shift here in some major trends.Everyone is talking about how IT needs to react differently. There is lots of change going on.Integration has always been important, but now it’s probably more important than ever.With some of the shifts in computing models, such as cloud and the data intensive atmospherethat most organizations are now operating in, why is integration a real issue that needs to beapproached differently?Overriding trendsWalsh: Were seeing a couple of overriding trends that have really shifted the market for integration solutions. The needs have shifted with changes in the workplace. First and foremost, were seeing that there is much more information that needs to be managed, much more data associated, and there are a couple of drivers of that. One is that there are many more interactions amongst different functional units within a business. Were seeing that silos have been broken down and that there’s more interaction amongst these different functions, and thus more databeing exchanged between them and more need to integrate that data.There’s also this notion of the consumerization of IT, that with so many devices like iPhones andiPads being accessible to consumers in their everyday life. They bring those to work and theyexpect those tools to be adapted to their workplace. With that just comes an even larger increasein the data explosion that you had referenced earlier.Coupled with that are overriding trends in IT to shift the burden of supporting systems awayfrom the traditional data center and into the cloud. Cloud has been a big movement over the lastcouple of years in IT and it has an impact on integration. No longer can an IT department havefull control over the applications that they are integrating. They now have to interact withapplications like Salesforce.com.A number of these trends converged. In the past, you may have been able to address data issuesseparately with small portion of your IT group within the data center and say applicationintegration separately with another group within the data center. Nowadays, you are not only incontrol of your own systems, you have to depend on systems that someone else would besupporting for you in the cloud. Thus, the complexity of all of the integration points that need tobe managed has exploded.
  3. 3. These are some of the overriding trends that we are seeing at Talend and responding to in termsof issues that are driving our customer needs today.Gardner: It sounds like there are two major shifts in addition to some other complexity issues.The two shifts seem to be that we now need to integrate data, applications, and services withsome sort of a coordinated effect. Having them in separate silos doesn’t seem to work very well.And then, we have a shift in terms of the architecture of where the computing, the resources, andthe data reside -- and that would be this cloud computing activity.Why is it important for data and application integration activities to become closer or even underthe same umbrella?Walsh: The two trends that you talked about are related. The architectural trend is really driving the need for the data and application integration technologies and the team supporting those to come together. The reason is that data and application integration no longer are necessarily centralized in a single location. When they were, you had, in essence, a single point of integration that you needed to manage amongst the data and theapplications. Nowadays, it’s distributed throughout your enterprise, but also distributed, as Imentioned before, across a network of partners and providers that you may be using.So many touchpointsWith that, there’s now the mandate that you can no longer isolate data from application, becausethe touch points are just so many. You now need to look at solutions that, from the get-go,consider both aspects of the integration problem -- the data aspect and the system and applicationintegration aspect.Gardner: And, I suppose we need to tool in such a way that we can approach both of theseproblem sets, the data integration and the applications integration, with a common interface or atleast common logic. Is that correct?Walsh: Yes, and up until now the two audiences have been treated quite differently. I think thetool expectations of the audience for data management versus the audience for applicationintegration were quite different. Were finding that we need to bridge that gap and provide unifiedtool sets that are appropriate for both the data management user, as well as the applicationintegration user.Gardner: I think we understand the business requirements now, why this shift is happening, whyit’s so important, and how it supports real agility capabilities of an organization. So, this is not anice to have, but really mission-critical.
  4. 4. Let’s go to Dan Kulp. Tell me why a certain architectural or platform approach best address theseissues. It doesn’t sound like a manual, labor-intensive, siloed approach works. Why must we takea different kind of architectural step here, Dan?Kulp: As Pat mentioned earlier, with the shifting of the requirements from silos into more of adistributed environment, the developers that are doing the application integration and the peopledoing the data management have to talk a lot more to get these problems solved. Your oldersolutions, from five years ago or whatever, that had each of those things completely separatewere not able to scale up to this distributed type environment.Gardner: Let me ask you now from a different perspective, architecturally we have a shift, butwhy does an open source community approach help bring these constituencies together? What isit about an open source and modular approach to these infrastructure components that helpsbridge these cultures?Kulp: One aspect that open source brings is a very wide range of requirements that are placed onthese open source projects. That provides a lot of benefit to an organization, as theserequirements may not be required of your organization today, but you don’t really know what’sgoing to happen six months or a year from now.You may acquire another company or you have to integrate another set of boxes from anotherarea of your organization. The open source projects that you see out there, because of their open-source nature, have been attracting a wide range of developers, a wide range of newrequirements and ideas, and very bright people who have really great ideas and thoughts andhave made these projects very successful, just from the community nature of open source.There is also the obvious cost benefit of not having all these high priced licenses, but the realvalue, in my opinion, is the community that’s behind these projects. Its continuously innovatingand continuously providing new solutions for problems you may not even have yet.Gardner: With cloud computing, youre also dealing with more moving parts. You don’tnecessarily know where those parts are coming from or what the underlying heritage is, but ifthere is an open source commonality among and between them. Im quite sure that many of thecloud providers have a significant amount of open source in their infrastructure that helps makethese interactions, these common denominators technically possible.New complexitiesWalsh: Agreed. The cloud brings a whole new set of complexities and challenges and as youare deploying your applications into the cloud, you need to think about these things. And a lot ofthese open-source projects that are addressing some of these cloud needs have thought aboutthese things.
  5. 5. If your organization isn’t into cloud yet, but youre thinking about it, leverage the expertise thatsalready out there. Talk to the communities and get engaged with those communities. Youll learna lot, and youll be probably better off for it in the long run.Gardner: Dan, youve been involved with open source for quite some time in a number ofcapacities. Maybe you could explain about where youre involved, what sort of projects you areworking on, and why this particular mix of projects sort of come to a head in helping us addressthis integration challenge?Kulp: Ive been involved with open source for roughly six years now, primarily at Apache. I got started at Apache as part of the Apache CXF Project. Ive been there since the beginning. As you mentioned earlier, Im the PMC Chair for that project, very heavily involved. For those people who aren’t familiar with CXF, that’s the web services stack. At Apache, theyre supporting all of your SOAP standards as well as JAX-RS and REST-based services. It’s really a framework for producing services. Six years ago, that was the problem people were trying to solve. As things have evolved over the last six years, were seeing more application integrationchallenges that are beyond SOAP and REST. That’s where projects like Apache Camel come in,where youre doing your enterprise integration patterns inside of your enterprise service buses(ESBs). So, Im getting more heavily involved with that.Ive also been involved with even things like the Maven Project at Apache, doing build-relatedtools and deployment scenario things.As the problems in the enterprise expand from year to year, which they always do, it’sfascinating seeing these open-source projects at Apache being incubated, or even graduatingfrom the incubator, that solve these real world scenarios. To me, it has been an amazingexperience to be involved with that whole process of seeing ideas bubble up through theincubator and into Apache projects that solve real world problems.Gardner: Okay. We understand that there is a new set of requirements for integration. We knowthat we have an arsenal of approaches vis-à-vis the open-source communities, and some provenand mature projects that are implemented quite robustly in some of the most intensive computeenvironments.How do we now bring this together in such a way that your typical enterprise can understandwhat they can do to bridge this gap between the data and the applications integration and thenreduce their risk by setting up an architecture that’s cloud ready or hybrid computing ready?Let’s go back to Pat Walsh. What are you finding on the street? What are people starting to do interms of coming to grips with these architectural changes?
  6. 6. Expanded marketWalsh: One interesting point to raise before talking about what were seeing people doing isthat there is an expanded market now for these integration challenges. It used to be that wewould see very large enterprises were the ones that were addressing complexity in theirorganizations.With cloud-based initiatives and such, it’s affecting even small to medium-size businesses(SMBs). We see a much broader set of enterprises trying to address it. Companies that havefewer than 1,000 employees are now looking at integration solutions to manage their data andtheir applications in the cloud in a much more sophisticated way than just three years ago. It’s amuch broader problem.The way that people are hoping to address it is by looking for a way that doesn’t require amassive outlay of investment in consulting resources. The traditional large organization, inaddition to purchasing product to help them with integrating their data and integrating theirapplications, would typically have systems integrator help them pull everything together. That’sobviously not an affordable path for an SMB.Therefore, people are looking to see, how they can find a combined, easy to use way and howthey can gain knowledge from people who have experience, having tackled these issues andproblems in the past.Were finding that people are looking for just a simpler, prescriptive way to do the majority of thechallenges out there. In terms of the 20 percent outlier problems, you may need to have asystems integrator come in and help you with that. But, people are really focused on the meatand potatoes of the integration of their functions, the data, and the applications that go along withthose processes and functions.Gardner: Dan Kulp, we need to have architecture modernization in effect, but we need to do itin such a way that more people in a large organization and more types of organizations, small tomedium-sized businesses, can avail themselves of these services, these capabilities.Tell me a little bit about what you have done to allow that difficult equation to be solved? Itseems to me that we are still talking about service-oriented architecture (SOA). In many respectswere talking about ESBs. Five or seven years ago, that was a very complex and costly activity.Weve now been able to abstract up the value, but I suppose reduce and subvert the complexity.Tell me how you do that.Kulp: The first step in that process to solve that problem was identifying where the bestsolutions are/ Theyre primarily in open source. I mentioned CXF and Camel, and there isApache Karaf providing some OSGi stuff.That was the first step. We grab those and bring them together, the best of breed from the variousApache projects that solve real world problems.
  7. 7. The next step was trying to find or produce a set of tooling that makes using those products a loteasier. One of the things about Apache that you will discover, if you are heavily involved is thatwe are hardcore developers. For us, writing Java code to solve a problem is natural.Skill setsOne of the problems that were trying to address is bringing this great technology produced bythe Apache people into the hands of those that don’t have that same level of skill set, expertise,or mindset.That includes those from the application integration side, where you have developers that areused to doing point-and-click type enterprise integration pattern things, to the data integrationpeople that are used to their data mappings, GUIs, and things like that, and trying to bring bothsets of people together into a platform that can solve both teams.Gardner: A similar questions to you Pat. Where do we bring the value higher but make thecomplexity less of an issue and less visible? What is it about your tools and approach at Talendthat is helping to bring this to the masses in a way that’s automated, a service factory approach,rather than a hand coding approach?Walsh: Talend has a great history of unifying technologies onto a common platform, to reallykeep the power of the underlying tools, but simplify the interface to it. This unified platformreally consists of five key components.The first one is a common development environment that is used across the products. The secondthing is a common deployment tool that allows you to deploy into a runtime environment.Theres also a common repository that allows you, across the lifecycle of your process, to be ableto manage it consistently, regardless of the type of technology that’s being used. Finally, there iscommon monitoring across the entire environment.What we are doing now is extending that model that has been applied to our data managementproducts to encompass the ESB, the application integration aspect of it. By providing this unifiedplatform of tools, it allows someone to learn a single interface, regardless of whether it’s at thedevelopment stage, the deployment stage, or the management stage, and get the power of masterdata management technologies, data integration, data quality, or the ESB technologiesthemselves.By providing this one interface, this one common environment, allows people to becomecomfortable with this common interface, but have the benefit of multiple sets of tools.Gardner: One of the things that I face when I talk about these issues with enterprises is that theylike the idea of having more people involved, but they also see that there is a risk involved withthat concerning permissions, access, control, and even policy and rules driven activities aroundwho gets to integrate what. How do you solve or ameliorate that problem?
  8. 8. Walsh: Weve gone to great lengths to include security mechanisms into the solution, so that wecan have approaches whereby there are certain permissions for just individuals. Or, ITmanagement can look at certain aspects while opening it up maybe to a broader audience, whenit comes to development and use of the interfaces that are going to be developed on the data inapplication side.Democratizing technologyIt’s very important, as you say, that as we bring this technology to the masses, as we refer to it,democratizing the technology, lowering the barriers to entry that historically have been in place,we don’t remove any of the enterprise qualities that are expected. Security is certainly a majorone, as is policy management, so that you could have a number of different business roles thatallow you to have the flexibility you need as you deploy it into a large- or even medium-sizeenterprise.Were providing both capabilities, simplifying the interface, while not removing any of theenterprise qualities that have come to be expected of the integration products we provide.Gardner: Okay. Dan has told us a little bit about how some of the open source projects, such asCXF, Camel, and Karaf have provided some fundamental underpinnings for this. But, Talend hasalso been merging and acquiring. Tell me a little bit about your business and the evolution ofTalend that has allowed you to provide this all in one integration capability to, as you say, moreof the masses?Walsh: It came quite naturally from Talend’s perspective. Data customers were using our dataintegration tools, as well as our data quality tools. We have Talend Open Studio, which is ourpopular open source data integration technology. Customers naturally were inquiring about howthey could provide these data jobs as services, so that they could be reused by other applications,or they were inquiring how they could incorporate our technology into a SOA.This led Talend to partner with a company called Sopera. They had a very rich ESB-basedintegration platform for applications. After two years of partnership, we decided it made sense tocome together in a stronger way, and Talend acquired Sopera.So, we have seen this firsthand from our customers. It really drove us to see the convergence ofdata and application integration technology, and therefore the acquisition of Sopera’s technology,as well as the people behind that technology, has enabled us to really come in with this commonplatform that we are just now releasing.Gardner: The timing sounds very good. Theres movement in the market towardsdemocratization, more inclusive platform approach to both data and applications and servicesintegration. The driver in the market about hybrid computing is coming right at the right time in
  9. 9. terms of being able to bridge different types of computing environments and integrate acrossthem.This all is great in theory and we have certainly seen a lot of action in the open sourcecommunity that had bolstered the ability of these underlying products and projects. But, whatabout real use case scenarios. Do we have any examples of where this is being used now, perhapsearly adopters? Maybe you can name them or maybe you can only describe what they are doing.But for me, showing is always better than just telling. Can we show how this all in oneintegration capability is actually being used in the field?Walsh: We have a couple of examples that I can refer to. I think the most tangible one that maymake sense to folks is that we have an insurance company that we work with. While theyve beenworking with us for quite some time on the data side of the house, looking at how they can havetheir back office data shared amongst the different industry consortia that they work with to doratings and other checks on credit worthiness or insurance risk, that has really been aboutintegrating data on the backend.Much like any business, theyre making it more accessible to their consumers by trying to extendtheir back-office systems into systems that have more general web interface or maybe aninterface at an ATM.Opened to consumersSo, they required some application integration technology, and with that, they built this webinterface and opened it up to consumers. The expectation of their user is a much more rapidresponse time. When they had to interface with an agent in the office, they may wait 24 hours fora response, but now they expect their answer to come during their web-based session.The timeframe required has led them to have an application integration solution that can respondin sub-second response rates for their transaction. In the past, they were going with a muchlonger latency for the completion of transactions.Its just a typical example that I think folks can appreciate. As people extend their back officesystems to consumers, number one, consumer expectations raised the bar in terms of the overallperformance of the system, and thus the technology that’s supporting those systems needs tonecessarily change to support that expectation.Gardner: In listening to Pat describe that use case, Dan, it sounds as if what were trying toaccomplish here is to do what the data warehousing, data mining, and business intelligence (BI)field have done, but perhaps allow many of those values to be extracted with more agility, faster,and then with a dynamic approach.Is that fair? Are we really compressing or creating a category separate from BI, but that does alot of what BI does vis-à-vis the integration of data and activities for application services?
  10. 10. Kulp: That’s exactly what’s happening. A couple of years, data mining ended up being batchjobs that were run at midnight or overnight. Then, the data would be available to the front endpeople the next morning. Youd get your reports or youd log into your system and check theresults of these batch jobs.With extending your backend data systems to the consumer, these overnight batch systems arereally not meeting the expectations of the consumers. Theyre demanding that their informationbe available immediately. They submit a new request and they want to have things updatedimmediately, so that results are available and displayed within seconds, not overnight.That requires a whole new set of skills, a whole new set of challenges. The people that weredoing the front-end application integration that queried the data from the overnight batch jobssuddenly have to have some expertise in not just cleaning the data, but allowing or working withthe team doing the data space, to provide updates to that information in a much more dynamicform.Gardner: How is this going to become more critical? Looking to the future, particularly fororganizations that are doing more and more web-based commerce, perhaps even more mobilecommerce, whether it’s through a web interface, a HTML5 interface, native applications onmobile devices, it seems to me that the consumer activities are driving more need for this fastfeedback loop integration and data analysis function.Let’s start with you, Pat. Why in the future does what we are talking about today become evenmore important, therefore become more critical as a core competency?Becoming more relevantWalsh: You can see that, as the consumerization of technology increases. Were already seeingthe pressure that IT feels from becoming more relevant to the business, that just expands.As I said before about the consumerization of devices in the workplace, it really does come downto the interfaces and the expectations that it doesn’t require a specialist in an IT field to be able tomanipulate and analyze the information that they need or even to create a service or applicationthat would enable them to do their everyday task or work function.That’s just going to expand it. It has been happening, and we are just going to see that at a morerapid pace. It’s going to require that vendors and technology companies like Talend respond inkind and build products that are more accessible to a broader audience of users.I think it’s analogous to what we saw in the early days of the Internet. Early on you would docommand-line interfaces to send files back and forth. Once there was a web-based interface, itopened it to the masses. Nowadays, we think nothing of using a web browser to do all kinds ofactivity that 20 years ago was reserved to just people that had a technical know how tomanipulate those systems.
  11. 11. We are seeing the same across these aspects of the business that up until now had really been thebastions of IT teams.Gardner: I would also wonder if data services become additional revenue sources forcompanies. If they can expose just the right amount of data safely and securely and give peoplesome tools to work with that, not only do they provide services, but the fact that they were in aposition to gather data about certain markets, certain activities, be it B2B or B2C, they can thenin a sense monetize that data back out into a field of partners and/or end-users.Is there an opportunity for enterprises to start looking at data, not just as an asset, but as actuallya product or service to sell?Walsh: Absolutely. Today, we see that they are really addressing data services as an efficiencywithin their organization. How can I leverage the investment that I have made in this initial dataanalysis or data job across the entirety of my organization? But it’s not a big step to take beyondthat to say, if it’s beneficial to my organization, why wouldn’t it be beneficial to others in myindustry or to an even broader audience?So we absolutely see that as a level of commerce that will be enabled by more sophisticated dataservices, technology, with a more accessible interface to that technology.Gardner: Dan Kulp, same perspective of the future of what’s going on in the future to you. Howdo you see the trends around mobile and even localization services and mobile commerce? Howdo these shape up, so that we will require more of the types of services we have been talkingabout today, that all in one integration, rapid iterative development around it?Comes down to consumersKulp: It really comes down to the consumers of these services and data. As the markets haveexpanded and the consumers are demanding things to get their information faster or get moreinformation or advertisers need to figure out, where are these consumers going and just the wholevariety of information sources expand out as well, the architecture of the applications and theinteractions between the front end and backend systems kind of get blurred.Things are changing, and companies like Talend that are involved in the space need to adapt aswell and provide better solutions that make these blurring lines occur a lot quicker. That’s whatwe are trying to target today.Gardner: We will have to wrap up now. Were about out of time. Pat, for those folks interestedin learning more, do you have some resources, some white papers, reports? Where would I go if Iwanted to learn more about this integration across data and applications function for the masses?What do you have available?Walsh: The easiest place to go would be our website at www.talend.com.
  12. 12. Gardner: Dan Kulp, what about in the open source community? Can you point folks to a placewhere they can learn more about some of these underlying and supporting projects?Kulp: Each of the projects have their own website with information. So CXF is cxf.apache.org;Camel is camel.apache.org; Karaf is karaf.apache.org. However, if you just go to the Apachewebsite, at www.apache.org, there are links to all of them, as well as a lot of valuableinformation about how Apache works and how these Apache communities work and how you getinvolved?A lot of that is just as important as what the technology projects themselves are trying to solve,but the community behind those projects is as much of an asset to the projects as the code itself. Iencourage people to poke around there and see all the exciting things that are going on atApache.Gardner: Youve been listening to a sponsored BriefingsDirect podcast discussion on how therole and impact of integration has shifted and how a more comprehensive and managed approachto integration is helping enterprises produce and leverage more data driven business processes.Id like to thank our guests. Weve been here today with Dan Kulp. He is Vice President of OpenSource Development at Talend’s Application Integration Unit. Thanks so much, Dan.Kulp: Thank you.Gardner: And also Pat Walsh, Vice President of Marketing at Talend in their ApplicationIntegration Division. Thank you, sir.Walsh: Thanks, Dana.Gardner: This is Dana Gardner, Principal Analyst at Interarbor Solutions. Thanks for listening,and come back next time.Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:TalendTranscript of a sponsored BriefingsDirect podcast on enterprise integration and new tools to putcontrol in the hands of "the masses." Copyright Interarbor Solutions, LLC, 2005-2011. All rightsreserved.You may also be interested in: • Talend Delivers Converged Platform for Better Daata Services Integration Across Multiple Applications • Rapidly evolving IT Trends Make Open, Agile App Integration More Important than Ever • Case Study: How Fairchild Semiconductor Has Leveraged the Workday Integration Cloud
  13. 13. • Open Source and Cloud: A Curse or Blessing During Recession? BriefingsDirect Analysts Weigh In