Dear Open Source, Please Help. Love, Manufacturing
•Download as PPTX, PDF•
0 likes•630 views
My talk from All Things Open 2015 about how the manufacturing worlds needs the energy and passion of the open source world to help ignite the most impactful segment of the US economy.
Recommended
More Related Content
Viewers also liked
Viewers also liked (14)
Similar to Dear Open Source, Please Help. Love, Manufacturing
Similar to Dear Open Source, Please Help. Love, Manufacturing (20)
Recently uploaded
Recently uploaded (20)
Dear Open Source, Please Help. Love, Manufacturing
- 1. DEAR OPEN SOURCE: PLEASE HELP. LOVE, MANUFACTURING Andy Robinson
- 2. Introductions • Andy Robinson • Information Solutions Consultant with Avid Solutions in Raleigh • 2 years as Principal at Phase 2 Automation in Taiwan • 10 years with Avid Solutions • 5 years with Monsanto/Solutia in Decatur, AL and Houston, TX • Co-Founder of aaOpenSource • Failed Founder of DeployCamp 10/26/2015 2
- 3. Introductions/A Little More Chemical Engineer by Degree Automation Engineer by Occupation 10/26/2015 3
- 4. Introductions/A Little More Typical tasks involve Design and Code Control Schemes 10/26/2015 4 By Con-struct (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
- 5. Introductions/A Little More Design and Build Hardware Control Panels 10/26/2015 5
- 6. Introductions/A Little More And Lots of Documentation 10/26/2015 6
- 9. The Why/Stats and Facts 10/26/2015 9 http://www.epi.org/publication/the-manufacturing-footprint-and-the-importance-of-u-s-manufacturing-jobs/ http://www.bea.gov/industry/gdpbyind_data.htm $5.9T $2.1T
- 10. The Why/Stats and Facts 10/26/2015 10 http://www.epi.org/publication/the-manufacturing-footprint-and-the-importance-of-u-s-manufacturing-jobs/
- 11. The Why/Stats and Facts 10/26/2015 11 Transparency Market Research Industrial Controls System Market - Global Industry Analysis Size Share Growth Trends and Forecast 2015 - 2021, 50 55 60 65 70 75 80 85 2014 2021 Industrial Control System Market Size ($B) $58B $81B
- 12. The Why/My Purpose Today Engage the open source community at large to invite you in to our world and help move the needle on the most impactful segment of our economy. 10/26/2015 12
- 17. The Problems/Tools 10/26/2015 17 Dr. Richard Mark Soley Executive Director Industrial Internet Consortium Smart Industry 2015 Conference, Chicago
- 25. The Problems/Experience 10/26/2015 25 It’s garage- ware from a couple hackers It can’t be secure if you can just read the source I can’t get production support if it’s free 99 Advisories from ICS-CERT
- 26. The Problems/Experience 10/26/2015 26 I’m not sure if legal will let us use it I’m not sure if legal will let us share it Can I sell a commercial product with it? How will we make money if we give it away?
- 29. The Problems/Communities 10/26/2015 29 I love what you are doing with aaOpenSource Great, have you had a chance to use the tools? Yeah, we’ve used your log reader and made a few improvements Wonderful, can you contribute those back?
- 30. The Problems/Communities 10/26/2015 30 $$ = Valuable Stuff
- 32. A New Hope 10/26/2015 32 http://farfarawayradio.com/2015/08/10/far-far-away-radio-presents-a-new-hope-audio-drama/
- 33. A New Hope/What are We Doing Well? 10/26/2015 33 Opensource.com
- 34. A New Hope/What are We Doing Well? 10/26/2015 34 Adam Crain/Automatak/@jadamcrain
- 35. A New Hope/What are We Doing Well? 10/26/2015 35 https://pixabay.com Proprietary Proprietary OPC Proprietary Modbus API ODATA SQL Server/T-SQL
- 37. Helping Out/Start Local with Software 10/26/2015 37
- 38. Helping Out/Build with Local Communities 10/26/2015 38
- 39. Helping Out/Join Existing Efforts 10/26/2015 39 DIGITAL MANUFACTURING COMMONS http://www.projectdmc.org/ OPEN [SOURCE] + SCADA DCS CONTROL HMI Jump in and Contribute
- 40. Helping Out/Testing Frameworks 10/26/2015 40 http://www.bigboxx.in/software-testing-training-chandigarh/
Editor's Notes
- First things first… who am I. Well I’m nobody special… just a run of the mill engineer trying to do good work every day.
- A little more about me. I am a chemical engineer by degree but once I started working in the plant I was very quickly drawn to the automation side of things and have been there ever since.
- So what does an automation engineer actually do? Well one of the first things we do is design control schemes to manage equipment. We’ll look at something and say ok, to keep the level steady we will open and close this valve and to keep the temperature on target we can open and close a steam valve or cooling water valve on the vessel or reactor. Once we’ve decided exactly how it’s going to work we lay down code that usually looks something similar to this. And yes, it’s usually that ugly.
- Another big piece of what we do is make lots of electrical drawings and then build these really large cabinets full of expensive electrical components and wiring.
- Finally, depending on the industry there can be a huge amount of documentation. I do most of my work in life sciences, building pharmaceutical plants, but we’ve got some other folks who work in the nuclear industry and it would truly blow your mind the amount of documentation and testing involved.. But to be honest it probably shouldn’t
- Ok, let’s get into the meat of why I’m here.
- The first question you should really be asking me right now is why a guy from the manufacturing world is speaking at a software conference. I’ll be honest I’m still asking myself the same question but I’ll give it a shot.
- According to the economic policy institute the gross output of the manufacturing sector was around 5.9 trillion dollar in 2013. And if you look at the net impact of manufacturing you are still talking about almost 2.1 trillion dollars. So I tried to find a roughly equivalent number for the software sector and according to the bureau of economic analysis I come up with about 300 billion dollars in 2014 for total gross output. So really smart people can disagree on if this is really apples to apples but the overarching theme is that on a relative basis, manufacturing has a dramatically larger impact on the economic health of the country than software.
- Another view is to look at jobs and you have a similar story here. Manufacturing really is the 900 lb gorilla when you look at jobs and more importantly net impact on overall economic health.
- When you drill down a little closer into the industry I work in the, industrial control system the estimates and projects says right now it’s about a 60 billion dollar industry but expected to grow to about 81 billion over the next 7 or so years. Based on my years of experience this is probably about half software, half hardware but the takeaway is that this is a pretty large industry and it’s growing.
- So that’s great Andy you’ve thrown a bunch of facts and figures but again, why are you here? I’m here to engage the open source community to ask you to please come over and engage your neighbors and friends in manufacturing and try to move the needle on the most impactful segment of our economy. I know a lot of people might say, oh it’s hopeless all of our manufacturing is moving offseas. While there is some truth to that all you need to do is look at folks like Elon Musk and what he’s doing with Tesla to understand that we can use innovation and creativity to bring jobs and companies back to the US.
- It doesn’t take a genius like Jay-Z to understand that manufacturing has a lot of problems in this country but I’m going to focus on the field that I know the most about and that’s automation. And in automation I’ll keep it to 3 core issues.
- The problems I’m going to talk about are our tools, our experience, and our communities.. Or lack thereof
- First the tools
- Where in the world do I start. I think the most obvious place is that our tools in general are really old and really creaky when they are compared to what you have available at your disposal in the general software space. Yes there are always some hot new vendors with sexy new things but 99%+ of what you see in day to day work at serious manufacturing facilities is pretty bad.
- A few weeks ago I attended the Smart Industry 2015 conference in Chicago and came away really inspired by some awesome work being done by some great people. But what really makes their work important is that they recognize how slow our progress to date has been. Dr Soley, the executive director of the Industrial Internet Consortium put up this slide. This is a Modicon controller, which is generally recognized as one of if not the earliest programmable logic controller in our industry. On the left 1980, on the right, today. Not much different other than a plastic chassis and the speeds and memory sizes are a little better. But don’t be fooled, you still program this in an amazingly similar manner to the way you did back in 1980. There is another logic controller that we use where the 64 MB version costs around $15,000… today. And I would speculate.. Without having doing any actual testing that one of the new raspberry Pi’s would probably run circles around it on the performance front. Yeah the old codgers will skoff and say that’s a toy… but the old codgers also scoffed at smart phones too.
- What’s the first thing you think of when you think of quality software tools. Source control. How in the world can you build quality software if you can’t seamlessly collaborate, rollback, and compare versions. Unfortunately the story here for our industry is pretty bad. There are some source control tools but they are usually proprietary and also locked to one specific platform. There are a few cross platform tools but unfortunately they have plenty of warts too, the integrations are a bit cludgey the repository is typically closed so the only tool that can work with the source is the one source control tool. It really amazes me we have as much code working today as we do considering the really awful state of source control for our jobs.
- Our tools and skills aren’t portable. If we code for one particular hardware platform only the thought processes and algorithms are portable. The code itself almost always has to be re-written from scratch. Obviously this is a fairly well solved problem in the general software space but not so much in the industrial automation space.
- At first glance it seems like this is a problem that has been solved with a standard titled IEC 61131-3 which defines a series of functional language and design elements that in theory are portable across platforms. All of these seem like a good thing… until this bumps up with reality which is that in 15 years in this industry I have yet to come across a single application where I could actually take my code into some form of IEC61131-3 compliant library and use it across platforms.
- And to be honest I think a bigger theme at a play here is the idea of sense of being old and decrepid. I obscured the exact brand just so I don’t crucify them in an open forum but here is the web site and really the singular brand touting the benefits of this portable language set. Any hip young creative types with new ideas want to jump in and participate, push the boundaries, explore what’s possible? Seriously folks, squarespace and wordpress are really cheap and really easy.
- There is a very real issue with the tools we use that the price point of most of them puts them out of reach of the casual party just looking to explore and experiment, especially students and young programmers. The purchase price for most with decent features start in the thousands of dollars. And the make matter worse most of them have very restrictive grace periods, like 7 days or no model at all for just exploring and trying out the tools. While there are a small handful that might be free or very low cost to download and use the simple fact is that in my 15 years working in serious manufacturing like chemical plants and pharmaceutical facilities you rarely if ever see these products used or even considered. It’s a very different world from the enterprise and web software space where the MO seems to be having to justify why you wouldn’t use open source software as a first option.
- Ultimately the real issue with our tools is that they aren’t packaged and delivered in a way that lets anyone but the most hard core professional practitioner jump in and start learning. Contrast this with the more general open source world where I can very easily download and play with Nginx, Cloud Foundry, or one of the countless applications or tool suites released under the Apache umbrella. To be fair a decent handful of the vendors might be willing to get you a 30 or 90 day license if you are willing to call them up and chat with them or maybe if you are a school you can get a special educational discount but at the end of the day those delivery methods are definitely an exception. I mean for goodness sake you can download and run Oracle for free in a dev environment. That’s pretty bad when your general posture on licensing is worse than Oracle.
- So the next issue with deal with is the one of experience. Did you know that there are people in this world who use and program software for critical functions that don’t know who Richard Stallman is? Yeah, I know it’s a total shocker but it’s true. On a more serious note however, the vast majority of people working in the Industrial Automation space have only a vague concept of what open source software is all about.
- When you talk to most of them these are some of the responses you will get. The most damning one is usually the one where they ask “who would possibly support me”. I try to walk them through the concept that you don’t have to get support from the people who sold you the software… and to be honest that might be the most powerful reason to go open source. I know most customers are pretty turned off by the mandatory 15-20% license fees for the privilege of getting upgrades every 12-18 months. Even better when you deal with highly regulated environments like I do there is very little appetite for upgrading the software just so you can stay current on support. I would speculate the model where you are explicitly paying for support and even better getting to choose who provides your support would go over much better with many of my customers. Taking a step back, most of our customers actually just call us up directly for support anyway because they feel much more comfortable discussing issues with the people who developed their applications and more intimately understand the consequences of restarting this or redeploying that. Another fun snippit about is how it can’t be secure if you can read the source and figure out how something works. To be honest this one is actually pretty comical if you take a look at the ICS-CERT feed. ICS-CERT is the computer emergency response team for Industrial Control systems. They coordinate vulnerability releases for the industry. The ill informed fallacy that somehow closed source software is obviously more secure is really getting shredded daily. There is a campaign by a guy named Dale Peterson with Digital Bond where is on a warpath about the idea of our systems being insecure by design. While I might disagree with him on the path forward I do agree that the core communication methods we use in our modern platforms are pretty much wide open.. You can’t even say it’s insecure because that would infer they tried to make it secure to start…. Which really isn’t the case.
- So while I usually don’t have to much trouble with the discussions on the previous page I don’t really have a good answer for questions like this. At this stage I would like to have some kind of reference material I can send worried engineers about legal ramifications of using open source software as well as releasing libraries and code as open source. I’ve had many conversations with product managers encouraging them to explore the concepts around open source software, libraries, or at a minimum open API’s. While they might generally be responsive to the concept of open API’s the idea of actually open sources pieces of a product are just totally foreign to this industry. Right now there is simply no roadmap, no pioneer for them to follow like the Apache foundation or countless other examples that are out there. I actually had this exact conversation with the global marketing head for a multi-billion dollar company in this space. He freely admitted they needed to do a better job engaging with their community through being more open but they had no clue how to get started.
- And that… is Communities.
- Everyone here knows that the collective is almost always better than the individual… no need to waste my time preaching to the choir..
- So here is a really common conversation I’ve had with multiple others in my industry…..what’s funny is that most of the time the excuses revolve around not being sure if managers or legal will let them share (sound familiar) or that they are just too busy. At which point I ask myself the question…so I can spend countless hours of personal time working on this for your benefit but you are too busy at work to pull your changes together to contribute back??
- I think the problem is that we are so conditioned that to get anything of value you have to pay money. This concept that we can use something provided by the community and extract serious value, not just a toy utility, is just so incredibly foreign I think people just can’t make that mental breakthrough
- So I think I can work on the previous two items but here is the one that I think ends up killing us. So many people in our space have this impression that what they are doing is so unique… that they are a beautiful snowflake. What’s comical is that I have conversations with a lot of these folks, both suppliers and end users and they are all talking about the same problems… even better.. They have mostly all arrived at the same solution.. Independently. You know what you call it when 3 people solve the same problem in isolation. A half baked solution that too way too long to develop… and we’re no better off as an industry for it.
- So enough of the griping and whining. All of that was necessary to setup the next part where I talk about what we actually do pretty well and how that opens up opportunities for this community to join the fight
- So one of the things we do at an ok level is open protocols. While we have our fair share of closed and proprietary protocols we also have a decent number of open ones.
- Modbus and OPC are the two most likely ones you are bound to come across when you start to wander over into our world. Modbus is a very old protocol that actually started off in the serial days and has been wrapped by a lot of people in a TCP stack so we can use it over our IP networks. Even better the bits and bytes of the protocol are very widely published and easily accessed for study. OPC, for lack of a better comparison is kinda like the ODBC of the process control data world. When you’ve got a software package that needs to communicate with a piece of hardware typically the first question you ask is “is there an OPC server for that device”. If the answer is yes then you have substantially solved the communication equation to this piece of hardware. There was always an issue though, OPC relied heavily on DCOM and therefore could only be deployed on Windows machines. But a handful of years ago we got OPC-UA from the OPC foundation. What was great about UA was that it was truly platform independent, no more artificial dependencies on windows. What’s even better is that the spec is truly open for all to review and implement their own stack. There have been a number of open source implementations so far and one of the more active ones I’ve seen is actually an OPC-UA Client/Server project in Node.JS. I’ve also seen another one in Java, C, and C++. Now we just need Erlang, Go, Haskell, Rust…. Finally, and probably one of the most important pieces of work going on right now is open dnp3. DNP3 is the primary protocol used by most a large majority of power distribution systems installed today. I’m not an expert in the protocol but from my understanding the protocol at a fundamental level is pretty good and pretty powerful but much of it was designed in the days prior to considering if bad guys might want to monkey with our power systems…and as such really has very little if any security. After all, we’ve got firewalls on our networks and locks on the equipment so how could someone get in? Well, a guy named Adam Crain started poking around at it and pretty much shredded most of the current implementations because they were written so poorly. In the meantime he was working on an open source implementation that dramatically improves on the poor code quality, robustness, and soon to be security of most of the commercial releases available today. If you are worried about keeping the power on and you want to work on something seriously meaningful this is the one right now. I know Adam was going to try to make it over to the conference.. He’s actually located right here in Raleigh… so if you see him please chat him up to find out what you can do to help.
- Another thing we are decent at is storing our data in a way that is generally accessible by outside tools. The term you will find in our industry is Historian. It’s our term for Time Series Databases. These are some serious pieces of software. The class leaders in this space can scale to millions of unique tags. Not millions of data points, millions of tags…each with data streaming in as fast a a ¼ second or maybe every few seconds. Those tags have metadata, they do advanced compression algorithms so you don’t lose fidelity. The data is typically stored in binary log formats that are really efficient. All the things that tools like InfluxDB, Graphite, and others are working on but they’ve been doing it for many years. By the way, if I was a low end historian provider I’d be really worried about projects like Influx. You can think of these guys as the original big data repositories long before the term became popular. Wonderfully interesting datasets from real physical systems. So the thing that most of these packages actually do pretty well is making that stored data accessible to external software packages. Most of them will let you query via some kind of SQL Server interface so you can get the data and do all of your manipulations in the comfort of T-SQL. Or, some of the newer ones are now coming out with ODATA interfaces so the toolset that can interact with the data is dramatically expanded. The moral of the story here is that this is a pretty nice entry point into our world for outside interest and software packages for doing things like dashboards or data mining.
- So at this point your probably asking yourself, why on god’s earth do I have a picture of Tom Cruise in a presentation about open source and communities. Well, in Jerry Macguire he really said what I hope you are saying right now .. Andy, you need to help me help you.
- The first thing I would do is to start local, don’t try to boil the oceans just yet. A fair number of the people in this room might have some connection either directly or indirectly to a manufacturing company. Go chat with the engineering teams. Heck, go chat with the operations people on the floor. Talk to them about their pain points, what they wish they could do that their current system doesn’t do. Then look at that and think about your skills or community tools and software packages that might solve that one. What you will most likely come up with first is dashboards for data. We are always in need of good visibility for our data. We do a nice job at the operator level but bubbling that data up to dashboards and such for front office and supervisors has always been an issue. Yes there are tools in our space that do it.. But frankly they aren’t great and they tend to be on the pricey side. The best part about attacking something from this angle is that you are just layering something on top, not actually replacing any critical functions that can cause issues in the production system.
- The next step I think is to start building communities. When I look at the open source world what I see isn’t nearly so much open code and free software, it’s all about the community around these projects. It’s the admission by multiple people that everyone, even competitors, pulling in the same direction can lift all boats and help everyone get better.. And more importantly let each individual company focus on differentiating value for them and their company.. not low level mechanics. Reflecting on something from earlier in the presentation.. We are really terrible at doing this in the manufacturing world I think.. At least compared to the open source software world. One example from very close to home. Earlier this year I tried to put on a totally user driven conference. We were going to call it DeployCamp. Really small, maybe 40-50 people, here in Raleigh. I talked to countless people that I knew. All of them really excited, rah rah, yeah I’ll be there. But when it came down to it for folks to actually get the funding to travel for a couple day conference it totally fell apart. Why? I think it’s because we don’t have a real sense of community or sense of commitment to support each other. We see each other at vendor sponsored shows watching vendor presentations. We don’t have any identity outside of the products we use. And this, again, is where I think having you, the experts at building communities, can really show us how it’s done. So how do you actually go about building these communities.. Heck I don’t know.. You guys and girls and the experts.
- So the last step, or maybe this can be the first step if you choose is to go out and find existing work that is being done and jump in to help supercharge it. Back to my experience in Chicago a couple weeks ago, I heard an incredible talk from Dr. Bill King from the Digital Manufacturing and Design Innovation Instituate. They are part of the National Network for Manufacturing Innovation. The DMDII is a public-private partnership focused on improving manufacturing in America. An offshoot or component of this is Project DMC, or the Digital Manufacturing Commons. Briefly this is an open source toolkit for design, analytic models, simulations, and so on.. All typical functions you might encounter during the development lifecycle. The key though is that this is a totally open source software package and the big goal is to have people not only contribute code but also simulation models and other key functions. I haven’t had a chance to engage with these folks yet but it looked like some really exciting work. The other avenue you can go down is to get out there and Google Bing for Open Source SCADA, DCS, Control, HMI. These are all common terms we’ll use for the piece parts of software that we use on a daily basis. Unfortunately what you might find today is projects that are either not being maintained, look and feel like garageware, or is just some little experiment someone is trying out. I seriously applaud these folks for putting in a lot of time and effort in these project but until one gets really serious and is seriously maintained with a possible commercial entity backing it up with support none of these will ever get any traction.
- Just added this slide after the talk from Hans from LogiGear this morning and reminded me of something my company is actually trying to work on.. And that’s testing frameworks. I mentioned a few slides ago that we’ve got decent connectivity to our data not only real time but also historical. What about using all these amazing testing frameworks you’ve got and tweaking them to work with our connectivity to test our logic.. Even our graphics. Just to give you a sense of scale of what this really means, my company does work in the nuclear industry. We recently completed a project where we built 40 graphics screens to monitor the operations of a really critical piece of equipment. Without giving away too much I will tell you that the cost to the customer was over $1 million dollar and the vast vast majority of that was in documentation and testing.,
- So I can’t really stand up here and ask you for your help if I haven’t been willing to put in the work myself. The best reflection of my efforts is on GitHub at aaOpenSource. This is a collection of tools and utilities for working with Wonderware System Platform – Also known as Archestra.. Which happens to be the software platform I know best. Right now probably the most complete work and the one getting the most feedback and interaction is aaLog. We’re taking the closed binary log file they write and exposing that as a series of strongly typed records that you can then do whatever you like.. Things like sending to SysLog or a csv, or a database. I’ve been heavily involved with Brian Gilmore from Splunk to build a really nice modular input for this and we’ve got plans to build a nice application around this project in Splunk. We’ve got a long list of projects that are in different conditions, some small and complete… some kinda decent but probably need some love. As I’ve talked about this work with so many people I tell them the same thing over and over. My goal is to get to a point where nobody even realizes I’m one of the primary folks who started it. I just want to be one of many contributing some code and just making things better. If it’s obvious I’m the only one doing work here then that means we don’t have enough people participating. And, with any luck we’re going to revive DeployCamp and see if we can hang it off the end of another larger conference so the travel burden is taken away and then it’s just spending an extra day in a nice location like Orlando or Vegas.