Frederick Johnson Paper1

277 views

Published on

test

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
277
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Frederick Johnson Paper1

  1. 1. Part III of III “Thinking inside the Box” Using VLANs & ACLs on Your Manufacturing Shop Floor Network Frederick K. Johnson Project Controls EngineerAbstractAs Manufacturing Execution Systems (MES) become commonplace in the UnitedStates, the shop floor infrastructure design and security will play a bigger role than ithas ever played in the past. With dedicated domain, an excellent network design andproperly segmenting a shop floor network using Virtual Local Area Networks andAccess Control Lists, high network availability and performance is defiantly achievablefor any size manufacturing operation.March 23, 2011 • Revision 1.0
  2. 2. Using VLANs & ACLs on Your Manufacturing Shop Floor Network About the Author Frederick K. Johnson has more than 10 years ofAbout the Author 2 experience as an MES Systems Engineer,Intended Audience 2 implementing andIntroduction 3 supporting Manufacturing Systems. Frederick, whoNetwork Congestion by Design 4 holds a BA in EconomicsWhat is a LAN? 5 from Marquette University, a MS Ed in Instructional Technology from Cardinal StricthUse switches and not Hubs 5 University, has worked for some of the topWhat is a VLAN? 5 Fortune 50 companies in the United States, with names like DaimlerChrysler, Hyundai MotorWhy use VLANs anyway? 5 Manufacturing Alabama LLC, The Coca-ColaVLANs & ACLs 7 Company, and Tyson Foods Inc. With both domestic and international industry experienceStatic vs. Dynamic Routing 7 across Automotive, Food & Beverages, andSummary 7 Pharmaceuticals, Frederick currently holds a position at B|Braun Medical, Inc., the world’sReferences 9 12th-largest medical device manufacturer, as a Project Controls Engineer and is the principle MES Engineer. Intended Audience The goal of this article is to: Provide a non-technical and general understanding to those interested in good examples of how General IT and 1 Manufacturing IT use some of the same 1 IT tools, concepts and principles but for different reasons. Convey a basic understanding of MES while discussing specifically about commonly used IT tools, concepts and 2 methods. Then explain how and for what reasons MES Systems Engineers use these tools on shop floor networks. The idea here is to create a Mental Forum to provide an easy to understand 3 discussion where the reader can build a basis of understanding. All rights reserved. © 2010 Frederick K. Johnson
  3. 3. Using VLANs & ACLs on Your Manufacturing Shop Floor Network Preface and communication problems may not be that “Thinking Inside the Box” is a three part series big of an issue, especially given the small that focuses, in general, on the topic of number of communicating nodes. However, Manufacturing IT (MIT) and Manufacturing with the current state of new switching Execution Systems (MES) but, specifically, on the technology, high performance and low cost of subsequent Shop Floor network the MES mid range network equipment can definitely requires as its foundation for communication. provide a small shop floor the availability and Taking a closer look at how the topics are the performance demanded by a MES, even if divided up: the network design has a few faults. On the other hand, large-scale manufacturing environments are not as forgiving, because Part I speaks about the importance of when collisions and packet losses occur, this can reap havoc by cascading faults causing serious ! creating a distinct and separate communication problems across the entire shop Manufacturing Domain off an organization’s primary domain. floor infrastructure. Part II moves to the next level and takes on Here lies one of the most pervasive problems network topology or infrastructure design facing a shop floor network and the MES relating the hardware components to their Systems Engineer. PLCs are shop floor devices ! respective locations within the Main that communicate to both machines and to data Distribution Frame (MDF) and the collection servers. The data collection servers Independent Distribution Frame(s) (IDF). act as intermediary for MES, this is one of the primary ways MES receives the data it needs to Part III dials in on the Distribution & Access manage the shop floor orders. On the shop levels within the infrastructure discussing floor, PLCs are typically the intelligence holding! how Virtual Local Area Networks (VLAN)s and Access Control Lists (ACL)s are used to A Collision: is something that is very common segment or secure communication across on networks communicating over Ethernet. the entire shop floor network. This occurs when two or more clients are transmitting packets or messages filled with data, at the same exact time. When this happens, either the packets are dropped; In addition, these articles break down the shop causing a Packet Loss or the request is sent floor environment into very basic areas of back to the sender. Each sender then, importance. At that point, the reader is free retransmits their original request, via packets, and hopefully equipped to conduct a deeper but this time at different time intervals to dive into each area, where Manufacturing IT and prevent another collision. MES are the topics and Domains, Infrastructures and VLANs & ACLs becomes the framework where all the action “literally” goes down. devices. These devices house and run programs in their internal logic. Programs running in the PCL’s logic instruct machines and controls, Introduction located on the shop floor, what to do. Over time, Manufacturing Execution Systems (MES) have become more common place in As long as PLCs are communicating to their mainstream manufacturing, the design of the respective controls, process equipment and OPC shop floor infrastructure is beginning to take on servers, the world is great! When a network a more important role than its has ever played experiences a significant increase in packet loss, in the past. Sometimes the design of these we have a serious problem. networks can make managing the MES a difficult or an extremely easy task. Yet, in small manufacturing operations, bandwidth, latency All rights reserved. © 2011 Frederick K. Johnson
  4. 4. Using VLANs & ACLs on Your Manufacturing Shop Floor NetworkProgrammable Logic Controllers (PLCs) are verysensitive to packet loss, because PLCs can “fault” It is the horsepower of the OPC server, whichor stop communicating, by “timing out” when allows the SCADA system to communicate withthe PLC fails to receive a response after a set large groups of PLCs. This special softwarenumber of reties. allows the SCADA system to grab snap-shots of those states and conditions we talked about earlier.If a PCL “times out”, machines, controlled bythat PLC, may stop functioning, thereby haltingthe manufacturing process and causing an POLLING: Is the action where the OPC serverunscheduled outage. Taking all of this into takes snapshot of the states and conditions fromconsideration, managing traffic across the shop the program running within the PLC. Typically,floor network becomes serious business. polling is done at a predetermined rate orNevertheless, there are still a few ways that an interval in time where the OPC server takes itsMES Systems Engineer can tackle this potentially snapshots at the same rate (e.g. 1/sec or 1/min)crippling communication problem. To get a better understanding of this, imagine aNetwork Congestion by Design strobe light flickering on and off as people moveBy design, a MES tends to have a hierarchical around in a large room. This is the samecentralized architecture. Almost all production concept only instead of people moving arounddata is collected at the lowest level starting in a large room, we are taking note of states andfrom the process equipment (Machines) and conditions of controls and process equipment inthen going to the Controls that drive the process a shop floor.equipment. Remember, the PLCs give both thecontrols and the machines their instructions. Of course, SCADA is only polling of states andTherefore, the PLC will know the “State” or a conditions from specific controls and machines“Condition” for any entity under its control. via each PLC, not all the states and conditions that are available. The MES is only interested inNext, generating the actual raw data, used by the data it needs to provide its end-user thethe MES, occurs by grabbing “Snapshots” of functionality it has been configured to offers asthose “States” and “Conditions” of specific a system.controls and machines that the MES requires.Again, all of this is occurring within the PLC’s Mass amounts of snap-shots are thenlogic as groups of programs run concurrently. aggregated into a database residing on theGrabbing the snap-shots is done by instructing SCADA systems. The newly acquired data isthe PLCs to not only communicate with certain then past up, by all SCADA systems, to the MEScontrols and process equipment, but to level and is aggregated into one centralized datacommunicate to data collection servers, repository called the MES Historian. What thisgenerally known as SCADA systems. means, in terms of the shop floor infrastructure, is that; communication between devices are also centralized as the data travels higher andSCADA (Supervisory Control And Data higher to each level. Large groups of machineryAcquisition). The SCADA system is a physical and controls both receive their instructions fromdevice. SCADA is a server that generally runs on a small groups of PLCs scattered throughout thea server class operating system. It also has a shop floor. These PLCs are broken up intohigh-end database system and a very special groups or production areas, where one specificsoftware package that gives the SCADA system production area will only communicate to thean OPC server function. A SCADA system can be one SCADA system assigned to that area. Eachconsolidate in one single device or distributed SCADA systems will then communicate to oneacross multiple devices. MES Historian. All rights reserved. © 2011 Frederick K. Johnson
  5. 5. Using VLANs & ACLs on Your Manufacturing Shop Floor NetworkWith this in mind, potential communication number of retries, after that; the device goescongestion is obvious. One of the way MES into communication fault.Systems Engineers get around this problem is bysegmenting or isolating network traffic into A switch, on the other hand, routes a requestwhat is called Virtual Local Area Networks or through internal logic and by a type of routingVLANs for short. protocol. It also knows about other switches that are on the network and will “route” requests to other switches that routes a requestWhat is a LAN? to the correct client or device it has connectionBefore we get into what a VLAN is, we first need to. A routing protocol is the language or theto understand what a LAN or a Local Area algorithm that switches use to communicate toNetwork is. A LAN is a group of computing one another. A switching device will also knowdevices that communicate over some type of who is connected to it and on which port thatcommunication medium and by a device is communicating through as well ascommunication protocol that is known by all the communication speed and a communicationconnected devices. Typically, a LAN consists of a protocol.group of computers that have network interfacecards that are then connected to a port on hub In the example I used above, a switch wouldor switch by an Ethernet cable. These types of essentially only call off the names of the peopledevices generally use the most common that it knows are physically sitting in the room itcommunication protocol, which is TCP/IP. A hub manages. It would skip the names of peoplewill broadcast a request to any devices willing to sitting in other rooms and then would pass thelisten it because it has no knowledge of what list to another switch when it is finished. At thatport a device is communicating on. Therefore, a point, the two people that were having thatHub simply passes all requests to everyone. This original conversation are only “slightly”is the very reason why Hubs are not ideal interrupted. The conversation can continuedevices to be placed on a shop floor network or without a problem. It is this type of theon a Distributed Communication Systems (DCS). functionality and deep logic of a switch thatHubs can cause broadcast storms on any makes it a more suitable device for any network.network by broadcasting all requests to anydevice connected to it. What is a VLAN? A VLAN uses that deep logic within a switch.Use switches and not Hubs The VLAN is no more than a logical LAN thatA broadcast storm is like you and another does not allow communication outside itsperson sitting in a room full of people having a boundaries. This logical LAN is an organizedconversation. All of a sudden, a loud speaker group of devices communicating on a singlecomes on calling out endless names, providing network that is programmed and logicallyinformation about where the people should go. grouped as one inside the logic of a switch; thus,You and the person that you are talking to making the LAN a “Virtual” one. A device, thatcannot hear one another. Then one of you just is a part of a VLAN, can physically sit anywherestops talking and motions the other to leave on the tangible LAN. However, a MES Systemsthat area. You both enter the room where the Engineer will create a VLAN by instructing thedoor closes behind you. There, you two switch to logically organize or group devicescontinue your conversation in private. together or by a range of IP addresses into one or more VLANs. We call this segmenting orEssentially, this is what happens when you place breaking up the network.hubs on your shop floor network, the onlydifferences is, PLCs don’t take theirconversations to other locations, because Why use VLANs anyway?devices like these are programmed to send a set The primary reason that VLANs are used on a shop floor network is: All rights reserved. © 2011 Frederick K. Johnson
  6. 6. Using VLANs & ACLs on Your Manufacturing Shop Floor Network A Subnet: is a sub set, in a range of, IP address To reduce and or to contain network traffic or internet protocol addresses that a company in order to prevent an unscheduled has legal rights to use on their internal local! “production outage” resulting from area network. A single class-c subnet has 255 communication faults by devices, like PLCs, available addresses that can be used. that are very sensitive to packet losses. Depending on the organization, some of those addresses are reserved for specific use, reducing the available number to less than 255.MES Systems Engineers do not want devicesbroadcasting over the entire network causing Through my career, I have seen VLANsdevices to resend requests or maybe even to constantly missed used. It happens a lot whentime out. By segmenting or breaking up a large Front office IT Professionals, fills in as MITphysical area of the LAN into smaller pieces, we Professionals, and do not have the experienceminimize and control the network traffic. to know when, why and how VLANs should be used on the shop floor. The general rule ofHowever, with advancements in controls thumb is, if there are more that 200 devicestechnology communicating via Ethernet, PLCs in communicating on a single local area network,regards to this discussion, almost act and VLANs would not be a bad idea. In acommunicate like a PC. Therefore, timing out manufacturing environment, this is about themay be outdated as an issue, because a PC will same. On the shop floor, what a MES Systemsretry to send a message many times, before Engineer is interested in is the amount of trafficactually giving up - network Latency maybe the that is being generated and how many collisionsnew culprit these days. are occurring at each level. This relates to network monitoring and analysis. In large-scaleLatency: Is a delay, it is the duration of time that manufacturing, this is almost a given.it takes a packet to get from the sender to theintended receiver. Take for an example, the auto industry, where there may exist a Stamping Shop, a Body Shop, a Paint Shop, and Finally an Assembly Shop. InWe measure Latency by sending out a packet of operations like this, there may be over 1000data and then receiving that same packet back. devices communicating on one VLAN. If thoseThe time it takes for the packet to return to the individual areas were communication on a singleoriginal address is the Latency. LAN, the MES Systems Engineers would be chasing down phantom network andOther general reasons VLANs are used, is for communication problems for a very long time.more efficient network management, to prevent What we do in this case is create four differentcross communication between devices that VLANs; one for each shop and then assign awere never meant to communicate and for different subnet(s) to each VLAN. Even thoughsecurity purposes. By reducing and containing the overall LAN is one physical entity and thelocal traffic, the numbers of collisions, occurring buildings are separate shops, we segment theat the access switching level, will significantly network into four smaller and more manageablereduce the probability of PLC level faults. This is parts with different subnets.exactly what we want. MES Systems Engineerstend to create VLANs based on large but It is a good idea to break up a shop floor LANseparate production areas. If a manufacturing into the smallest number of VLANs as possible.process has three physically different but major In the above example, four VLANs are perfect.areas, a Systems Engineer may break up the Within each, VLAN we work with the productionareas into three different VLANs running on teams to determine what future delineationsthree separate subnets. are needed. At that point, we look at fine tuning access by using ACLs. All rights reserved. © 2011 Frederick K. Johnson
  7. 7. Using VLANs & ACLs on Your Manufacturing Shop Floor NetworkVLANs & ACLs only because; it requires very littleAccess Control Lists are very useful tools, this maintenance. My recommendation is to let thetool is use to make exceptions to the rules for switches do what you bought them to do –the lines that VLANs draw across the network. manage the traffic.These lists are manually programmed into aswitch at the Access Layer or Distribution Layer, Besides, today’s switching technology is at thethey instruct the switches on which client is the state of doing much more than theirexception to the rule. Sometimes, there are predecessors ever could. When we staticallydevices that need to sit in a specific VLAN but dictate which way traffic routes, we take awaymust be able to communicate to other devices the efficient management and balancing ofacross another VLAN. The MES Historian is one traffic that the device was design to do. Insuch device and the SCADA system is another. theory, static routing will require much more management when things change on the shopStill, the SCADA device collects data from the floor network.PCLs and then passes that data up to a MESHistorian. Therefore, the MES Historian must be Now, ask yourself, when has anything everable to communicate across all VLANS. In changed on the shop floor? I am sure you getaddition, as system redundancy to reduce the picture, when you coupled that withdowntime in the event of hardware failure, the managing VLANs and access level security. LikeMES architecture may assign SCADA anything else, frequently logging in and out of afunctionality from one SCADA node to fail-over switch, making changes, means that making ato another SCADA node located on an entirely mistake is bound to happen. The biggestdifferent VLAN. differences is that shutting down your business network means that people have to use theIn all of these cases, the MES Historian and the phone instead of their e-mail. (Unless your shopSCADA nodes will be included on Access Control is VOIP, then you are hosed there too). ShuttingLists at the distribution level and at the access down a manufacturing shop may cost theswitch level to achieve our desired results. The company big money, just in idle time fromwhole idea here is to go from general to specific. people standing around; not to mention theThink high level with your VLANs and then dial other costs associated with an outage.in or your security needs at the access layer withthe use of an Access Control List. SummaryWith that in mind, breaking the VLANs into In short, designing and laying out your shopsmaller VLANs within the physical buildings floor network is no different than designing anyshould be done so only if needed. Be mindful of other local area network. The only difference isyour MES Solution because some shop floor Manufacturing IT (MIT) uses the same IT, tools,software solution and devices do not support systems, methods and concepts, but MIT usescommunication across multiple VLANs. The best them differently and in some case, for entirelyadvice is to keep things simple as possible. The different reasons. Any production supervisorbiggest caution here is; managing VLANs can be will tell you that unscheduled outages ora nightmare if shop floor network uses static downtown costs their organization money. Thisrouting. is the point that lies at the heart of it all. Moreover, any good MES Systems EngineerStatic vs. Dynamic Routing understands this point extremely well. In fact,Static routing is where the MES Systems eliminating unscheduled outages is the #1Engineers, logically programs into the switch on requirement any Engineer should consider whenwhere to send any requests that it receives designing and building a shop floorinstead of allowing the switch to “Dynamically” infrastructure.determine where to send a requests it receives.Dynamic routing is easily the best way to route All rights reserved. © 2011 Frederick K. Johnson
  8. 8. Using VLANs & ACLs on Your Manufacturing Shop Floor NetworkTherefore, you can see as the story plays out, 2. Understand how these networking toolsthat the shop floor environment is not like your are used to achieve the availability and thedata center or your office building. The shop performance manufacturing organizationsfloor is where things are made, put together. are demanding.Clients in this environment are not peoplefaxing, sending e-mails or talking on the phone. There is an easy way to think about these toolsMajority of the shop floor clients are typically that can help further clarify there use.machines communicating to other machines. Ifthese machines stop communicating, the Try thinking of the “How?” this way. Looking atmanufacturing process will stop. There are five the diagram below, just imagine that the figuremajor points that I would want a person to take represents an organization’s entireaway from this article: manufacturing process. The gray circular figure, at the center, is the MDF (Main Distribution Frame) and is where the MES Central1. First, understand why we use VLANs and System resides. ACLs on a shop floor network. Creating a VLAN would be much like, breakingWhen creating VLANs remember that, the the entire manufacturing process up into littleprimary reasons “Why?” we use VLANs in a orange boxes, as we did above. Each box wouldmanufacturing environment are: represent an individual VLAN covering a portion of the entire manufacturing process. Since we  To reduce now know that a VLANs are virtual; this has to  To contain  To secure  To preventLocal network traffic and or unwanted networktraffic that would otherwise increase theprobabilities of DCS and machine levelcommunication faults. Shop floorcommunication failures usually result inunscheduled downtime. Unscheduled downtime due to means that each box would have a dedicated collisions, packet loss and latency, switch located at an IDF (Independent! occurring at the access and or at the Distribution Frame) in its production area. distribution switching level, is Therefore, when thinking of VLANs, “Think something that is preventable. inside the Box.” Depending on the size of a manufacturing process, some shop floor infrastructures may require less number ofOther reasons “Why?” would be to: VLANs than others. Next, take look at your own shop floor network Break up large production areas, and consider the following:! serviced by a single LAN, into more manageable pieces.  Physical geographical size  Number production areas One small area of a LAN has a higher  Total number nodes probability of being able span one or  Type of MES and its requirements! two subnets over it instead of trying to lay many subnets across one LAN. The point here is, see what you have to work with and if there are any blaring production All rights reserved. © 2011 Frederick K. Johnson
  9. 9. Using VLANs & ACLs on Your Manufacturing Shop Floor Networkrequirements jumping out at you. Then work ACLs can be programmed to block or allowwith your Controls Engineers to determine if sending, receiving of different types ofone or more VLAN is needed. communication on a range of ports. This is whyFor example: If you have a shop running 10 PLCs, an MES Systems Engineer should fully2 SCADA systems and 3 MES servers, including a understand what is communicating to what andHistorian, the odds are that you will not need on which ports (Not ports on a switch). This can getmany VLANs at all. You may have to restrict tricky, but with some planning, it can be doneaccess to specific devices, but as for multiple correctly.VLANs, probably not. One VLAN should suffice.The reasoning, in this case, this VLAN is more forsecurity and protection of the shop floor vs. 4. Think “general to specific” whencommunications. conceptualizing VLANs and ACLs. In other words, keep the number of VLANs to a3. ACLs create the exceptions to the minimal and view them as containers with walls, communication rules. Let us take a closer much like the little orange squares. Then use look at of our previous diagram. the ACL to allow exceptions to the VLAN’s exclusive communication rules.As describe before, theorange boxes representVLANs covering 5. Finally, after applying, the VLANs and thesmaller pieces of the chosen ACLs, the next order of using is toentire manufacturing monitor the traffic across each VLAN.process. The yellowtubes, connecting Monitor all traffic coming back to theeach VLAN to the distribution level and the amount of trafficcircular figure, in the coming back to the core level, where the MEScenter, would be the application system and Historian typicallyswitching communication links. It is within the reside. The key here is to monitor each switchcommunication links where the ACL comes into within the entire system, but mainly at theplay. From the figure below, you can see that access level; where you can then scrutinized anythere are three lines of communication. anomalies to tune the shop floor network to perform at its highest level and capability.However, ACL1 and ACL2 are only allowing oneline of communication through. The ACLessentially defines which client or clients are the References ACL2 Didier, Paul, Fernando Macias, James Harstad, Rick Antholine, Scott A. Johnston, Sabina Piyevsky, Mark Schillace, Gregory Wilcox, and Dan Zaniewski. Converged Plantwide Ethernet (CPwE) Design and Implementation Guide. OL-21226-01, ENET- TD001C-EN-P ed. San Jose, CA: Cisco ACL1 Systems, Inc. and Rockwell Automation, Inc., 2010. Print. McClellan, Michael. Applying Manufacturingexception to the rule, of communicating outside Execution Systems. 1 ed. (St. Lucieits respective box or VLAN. Another good point Press/Apics Series on Resourceto make here is that the ACL can be a two-way Management). Boca Raton: CRC, 1997.street. Print. All rights reserved. © 2011 Frederick K. Johnson
  10. 10. Using VLANs & ACLs on Your Manufacturing Shop Floor NetworkPart number: DN0912.MIT.P3.003. ©2011 Frederick K. Johnson. All rights reserved. Printed in the United StatesDecember 2010. Any and all material used as references or sources cited are sole property of their registered owners.For more information concerning any detailed specifications outlined within this article, please refer to the specificservice provider listed by brand. All rights reserved. © 2011 Frederick K. Johnson

×