• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Final Action Research Report

  • 4,601 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
4,601
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
13
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Identifying Process Gaps Through Evaluation Suzie Rose University of Colorado, Denver July 26, 2010
  • 2. Identifying Process Gaps Through Evaluation Introduction I am an instructional designer in the training department of Edelweiss Communications. Throughout the duration of this study, I held a leadership role in an initiative to improve the extent to which approximately 2,000 existing customer-facing employees stayed abreast of company changes. To do this, I had to analyze the extent to which employees were currently staying abreast of changes, possible causes of low performance, and implement possible solutions for process improvements. First, it is necessary to define why Edelweiss Communications’ customer-facing employees must stay abreast of company changes. Their goal is to provide exceptional customer service experiences at all times. This requires that they have extensive knowledge and skills about policies, internal and external procedures, products, services, and more. As the company changes and grows with updates and new initiatives, existing employees must increase their knowledge and skill sets. To ensure employees have the resources necessary to do this, Edelweiss Communications distributes weekly updates. Weekly updates outline the details of company changes and will be discussed in more detail in a subsequent section of this document. The training department that I am part of is responsible for supporting delivery of the information within weekly updates. It is responsible for identifying if training is needed and if so, determining the method in which it is delivered. This study will help identify the extent to which customer-facing employees stay abreast of company changes so the training department can assess possible gaps and implement process improvements.
  • 3. Problem Statement Upon commencement of this study, Edelweiss Communications did not measure the extent to which customer-facing employees stayed abreast of company changes. This lack of measurement placed the training department in a precarious position. It was unable to validate if the processes used to support the delivery of information in weekly updates was effective or valuable. Without this information, the training department struggled to show its contributions to the business organization. This situation held the potential to affect decisions regarding resources and budget allocation to the training department. Edelweiss Communications was also unable to identify employees who were not staying abreast of new information. Employees who do not stay abreast of new company information need additional coaching to prevent occurrences where they inadvertently provide misinformation to customers. Customers who receive misinformation are likely to leave a company which ultimately costs Edelweiss Communications valuable revenue. In fact, a study by Frederick Reichfeld and W. Earl Sasser Jr., showed that a 5% increase in customer loyalty can increase profits from 25% up to 85% by extending the average customer relationship lifetime (1990). This significant difference clearly indicates that customer retention and loyalty is valuable to any company.
  • 4. Purpose The purpose of this study was to help customer-facing employees stay abreast of company changes. There will be multiple phases of this study. However, this phase of the study (a) defined the problem, (b) measured the current state of employee knowledge as related to company changes, (c) conducted a focus group to collect qualitative data from key players, (d) analyzed the results of a knowledge based assessment to create a baseline of data, (e) compared and contrasted findings from the focus group, and (f) determined possible causes of low performance. It laid the foundation for Edelweiss Communications’ ultimate goals of increasing efficiencies and improving processes; possibly through methods explored in the Literature Review section of this document. To put this study into action, I evaluated the extent to which employees were currently staying abreast of company changes at the time of the study. I then analyzed the data collected to identify gaps in processes. According to Ron Kurtus, the founder of the School for Champions, “By organizing or re-organizing your operations and processes, you can make your business a more effective machine” (2002). Kurtus’ viewpoint echoes that of Edelweiss Communications and illustrates its aspiration to create an efficient and well-oiled organization. To achieve this aspiration, Edelweiss Communications chose to utilize Level Two assessments as defined by Donald L. Kirkpatrick (1959). These assessments served as the source of baseline data. Throughout this study, a synthesis of Level Two assessments and process improvements culminated into an exploration of the relationship between evaluation and process improvements in corporate settings. It helped assess possible gaps in Edelweiss Communications’ processes and propose improvements.
  • 5. Intended Audience This initiative had a variety of audiences. The primary audience for this study was Dr. Jennifer VanBerschott and fellow students in the Research in Information and Learning Technologies, ILT 6720 class. The secondary audiences included the management staff of Edelweiss Communications’ content management and training departments including managers, training managers, curriculum developers, content developers, project managers, vice presidents, and presidents of both departments. Note: Participation was not compensated beyond normal salary and benefits currently received at the time of the study. Research Questions Although future areas of focus and process improvement initiatives will follow this study, the focus of this portion of the study was identifying gaps in processes and analyzing possible causes for them. To identify and analyze these items properly, an overarching research question and guiding questions were developed to help keep this study focused. Overarching Question To what extent are customer-facing employees staying abreast of company changes? To help customer-facing employees stay abreast of company changes, it is important to establish a baseline of data. Focusing this phase of the study on an attempt to capture baseline data helps the long-term goal of helping employees stay abreast of company changes.
  • 6. Guiding Questions 1. What processes are involved in helping customer-facing employees stay abreast of company changes? To what extent are the processes involved? Processes are a possible cause and distinct antecedent of the extent to which employees stay abreast of company changes. In many cases, the processes involved have an extensive amount of influence on the transfer of knowledge. This includes those within company procedures and those involved in the delivery methods of the information. 2. What are the most effective methods of evaluation to help identify gaps in processes? For the purposes of this study, Edelweiss Communications used Level Two assessments as defined by Kirkpatrick’s model of training evaluation. This question aimed to identify if the method is effective or if another phase of study should be implemented with a different data collection method. 3. What are possible process improvement methodologies that may improve the state of the situation? Since Edelweiss Communications had not yet defined the type of process improvement methodology it wished to implement, it was decided that research regarding industry methodologies be explored throughout this phase of study. These findings are found in the Literature Review section of this document. Although the overarching question remained the same throughout the duration of the study, the guiding questions changed. The second guiding question originally read, “What correlations and patterns exist between employee assessment scores and the weekly update delivery methods?” Upon further inspection, it was clear that this was an inherent feature of data analysis and would naturally be answered throughout the analysis process.
  • 7. Instead, the following question was added, “What are possible process improvement methodologies that may improve the state of the situation?” Throughout the literature review conducted during this research study, it was found that many methodologies are currently in practice to help companies improve processes and efficiencies. Because the ultimate goal of all phases of this study is to help Edelweiss Communications’ employees stay abreast of company changes, methodologies on how to do so held a high level of importance. Although Edelweiss Communications must find its own tailored methodology and further investigation will be done in subsequent portions of this study, it is important to research alternative methodologies of process improvements while the foundation of the study is built – in phase one. Figure 1 Context of the Study Process All company changes and initiatives funnel through Edelweiss Communications’ content management department which creates a report summarizing each week’s updates. This is distributed to leadership staff at each location within the company. The leadership staffs then redistribute the information to site supervisors and trainers who deliver it through team meetings, instructor led training sessions, or eLearns. The training department supports this process by defining delivery methods and creating content if applicable. Employees have scheduled time to participate in team meetings, instructor led trainings, and eLearns. Participation in these activities is a job requirement and is enforced by employees’ supervisors. Refer to figure one for a flow chart of this process.
  • 8. Demographics To provide details about Edelweiss Communications’ demographics, a community profile was necessary. A community profile, as described by Stringer (2007), is a snapshot of the context in which the investigation is placed. The snapshot of Edelweiss Communications includes communication methods, locations, company organization, and more. Employees communicate with customers through phone conversations, emails, instant messages (IMs), door-to-door interactions, and in-store interactions. Refer to Figure 2 for a visual representation of this flow. They are located in eight locations within the United States. Some employees work directly for the corporation; some are contracted through third-party agreements. Employees include men and women of all age groups and nationalities. The median age is 28. These items were considered throughout this study to help Edelweiss Communications identify trends and gaps that may be related to demographics. Figure 2 Background Weekly updates have been distributed in the process described for many years. As the company grows and responsibilities increase, Edelweiss Communications fears that attention to these updates is dropping in priority for both employees and their supervisors. Retention of weekly updates is assumed as a job responsibility, and may therefore receive little attention; it
  • 9. does not seem to have a significant priority in terms of participation or enforcement of participation. This could result in employees falling out of touch with weekly updates and could be a cause of possible performance opportunities. By reassessing the priority and accountability levels of the weekly updates, Edelweiss Communications may be able to stimulate change in this situation. Literature Review A vital component of this study was to research existing information about the relationship between evaluation and process improvements in corporate settings. Being familiar with existing studies and literature helps one situate the study into existing knowledge and hopefully contribute to it (Knopf, 2006). The key component of this familiarity for Edelweiss Communications was the ability to situate this study into existing knowledge. The literature review was extremely helpful in researching, identifying, and defining methodologies of process improvements that are currently in practice in corporate environments. This insight provided a course of action for a subsequent phase of the study: implementation of process improvements. This comparison of viewpoints aligns with Stringer’s stance that literature reviews set the stage for comparisons of viewpoints from existing literature with those one discovers through their own studies (2007). Literature Search Questions Throughout the process of this literature review, articles and studies similar in culture and context to those of Edelweiss Communications’ were searched for; this helped explore and substantiate the study’s core concerns. This study explored the relationship between evaluation and process improvements in corporate settings to help assess possible gaps in Edelweiss
  • 10. Communications’ processes and propose improvements. Because this study has never before been conducted within Edelweiss Communications, it was necessary to expand the focus of the literature review to a broader stage, but also contain it within its core concerns. To do so, literature search questions were created to break down and emphasize the study’s core concerns: 1. What links and correlations exist between evaluation techniques or methods and process improvements in corporate settings? 2. What are the most effective methods of evaluation to help identify gaps in processes? 3. What are the most effective methods of implementing process improvements? Search Procedures All content searches started online to allow for easy duplication and cross-referencing of resources. As a student of the University of Colorado Denver, I began my search within the Auraria Library Database search engine. From there, I navigated to the Article Databases & Indexes where I selected the “Companies” category. After sorting for databases related to corporate training, I found a niche in the EBSCO database. In my search, I used the following combinations of terms: (a) “Process Improvement + Assessment,” (b) “Process Improvement + Experimental Design,” (c) “Process Improvement + Study,” (d) “Process Improvement + Identify Gaps,” (e) “Process Improvement + Methodologies,” (f) “Process Improvement + Lean,” (g) “Process Improvement + Kaizen,” (h) “Six Sigma,” (i) “DMAIC,” (j) “Evaluate + Kirkpatrick + Knowledge.” I was not familiar with some of these terms at the beginning of the literature review, but rather discovered them as I performed research inquiries. These include Six Sigma, DMAIC, Lean, and Kaizen.
  • 11. Literature Review Findings Overview As previously noted, the literature review was helpful in researching, identifying, and defining methodologies of process improvements that are currently in practice in corporate environments. Services are bound by time in terms of the processes that lead to the delivery of an outcome that benefits a customer (Fraser & Fraser, 2009). As competition increases, customers’ expectations about products and services increase as well. Statistics show that retaining satisfied and loyal customers is more profitable than acquiring new ones (Pakdil, Özkök, Dengiz, Kara, Selvi & Kargı, 2009). To retain customers, it is important that companies, Edelweiss Communications included, evaluate their processes to improve flows and efficiencies wherever possible. Becoming truly customer-centric is essential for the success of all businesses (Naumann & Giel, 1995). As the American culture continues to shift its focus from product- centric to customer-centric, businesses must keep pace with the shift. Therefore, companies look for ways to improve their organizational efficiencies to meet customer expectations (Jones, Parast, & Adams, 2010). Although some company efficiencies are not directly customer-facing, most contribute in a large way to a customer’s experience with a company, service, or product. A part of customer-centricity can be achieved through process improvements. In fact, Continuous Process Improvement (CPI) has attracted significant attention from corporate and industry environments (Li & Rajagopalan, 2008). Improving processes increases company efficiencies, employee job satisfaction, and the delivery of products and services to customers. Eliminating errors that result in customer dissatisfaction improves processes (Pakdil et al., 2009). But just how do companies eliminate errors and improve processes?
  • 12. Kaizen The goal of CPIs is to evaluate and improve a current state (Tarantino, 2009). This ties directly to Edelweiss Communications’ goal. Edelweiss Communications wants to evaluate and improve the current state of the extent to which employees stay abreast of company changes. This type of evaluation and improvement takes a great deal of focus and attention. A successful corporate cultural movement that helps support CPI is Kaizen. Carter & Carmichael (2009) explain that Kaizen is the Japanese word for good change (see Figure 3 Figure 3) and it encompasses a commitment to continual and steady improvement for management and employees. Weaknesses are actively sought out, documented, and remedied via dedicated discussion and actions that are part of the normal workday, rather than being occasional or external. Ortiz (2006) explains that the philosophy of Kaizen is the foundation and driver for process improvement initiatives. It helps build a foundation for change that will enable a company to sustain process improvements for its lifetime. A successful Kaizen program has a governing committee, communication boards and newsletters, regular meetings, and event tracking. The following sections of this review outline the industry leaders in process improvement methodologies that are possible because of Kaizen: Six Sigma, Lean, and Lean Six Sigma. Any or all of these methodologies have the potential to help Edelweiss Communications improve the extent to which employees stay abreast of company changes.
  • 13. Six Sigma Six Sigma is a data-driven philosophy used to drive management decisions and actions across an organization (Caulcutt, 2001). Data is used here as a subjective term and can include various observable information of both a quantitative and qualitative nature. Motorola generated the concept in 1981 and pioneers using Six Sigma helped refine and develop it (Bendell, 2005). As the first US organization to implement Six Sigma, General Electric contributed to broadening its use across manufacturing, design and transactional areas (Grant, 2008). These companies are examples of pioneers which implemented the ideas and concepts of the Six Sigma methodology to establish concrete financial advantages. Six Sigma uses a structured method to solve business problems (Jones et al., 2010) through the use of a standard framework known as Define-Measure-Analyze-Improve-Control [DMAIC] (Gygi, DeCarlo, & Williams, 2005). Refer to Figure 4 for a visual representation of this flow. Although this is the most recognized framework, and some literature refers to it alone, Six Sigma uses another framework known as Define-Measure-Analyze-Design-Verify [DMADV] (Jones et al., 2010) – Refer to Figure 5 for a visual representation of this flow. Figure 4. DMAIC Framework Figure 5. DMADV Framework
  • 14. The DMAIC framework is designed to help eliminate waste and defects to improve on existing processes, whereas the DMADV framework is designed to develop new products, services, or tools (Jones et al., 2010). Each framework follows the phases within it in a linear manner. For detailed definitions of each phase, refer to Appendix A. Although Six Sigma is the most recognized method of process improvement, it receives mixed reviews from critics. Motorola and General Electric report significant savings as a result of their Six Sigma initiatives (Pande, Neuman, & Cavanagh, 2000). Yet others fear it will fail because of intense business competitiveness (Jones et al., 2010). In fact, one study found that companies that adopted Six Sigma underperformed in the stock market and achieved only incremental gains (Bernett & Nentl, 2010). However, of all the literature included in this review, that is the only reference to less than favourable results. Bernett and Nentl (2010) also contradict others by stating that in 1998 Motorola’s second quarter profits were minimal and the company eliminated 15,000 jobs. Despite this unfavourable review, Six Sigma is generally respected and considered successful in most corporate environments. Six Sigma is appealing to corporate organizations because it is built upon a foundation of statistical methods to improve quality, and it can define success in a quantifiable manner (Tarantino, 2009). This is of great importance to organizations such as Edelweiss Communications that aim to quantify return on investment and track improvement rates.
  • 15. Lean Another prevalent CPI method is Lean. John Krafcik coined the term ‘Lean’ to describe the new production techniques at Toyota after World War II (Bendell, 2005). Lean is defined as, “The systematic pursuit of perfect value through the elimination of waste in all aspects of the organization’s business processes” (Bendell, 2005). It incorporates five basic principles: (a) the elimination of waste, (b) the identification of the Value Stream, (c) the achievement of Flow through the process, (d) pacing by a pull (rather than push) signal, and (e) the continuous pursuit of perfection (Womack, Jones, & Ross, 1990). Refer to Appendix B for further explanation of Lean definitions. Both Six Sigma and Lean are built upon the foundation of Kaizen and business process improvement (Bendell, 2005). Unlike Six Sigma, Lean focuses on concepts and values as its structure rather than phases of a process. Another difference is that Lean is heavily centered on process analysis and flowcharting/mapping techniques, rather than data (Grant, 2008). Like Six Sigma, Lean focuses on increasing efficiencies and decreasing waste. Also like Six Sigma, Lean receives mixed reviews from critics. Bernett and Nentl (2010) report that Lean has been well received by corporate organizations since Womack and Jones captured organizations’ attention in 1996 with the release of their book, Lean Thinking. However, other critics claim that Lean has a limited application range; there is general agreement that it is well suited to companies with similar characteristics to the automotive industry rather than corporate organizations (Lasa, de Castro, & Laburu, 2009). However a recent contribution to the Lean model includes Value Stream Mapping (VSM) which focuses on the analysis and improvement of discontinuous flow lines. It includes five phases: (1) selection of product family, (2) current state mapping, (3) future state mapping, (4)
  • 16. definition of working plan, and (5) achievement of working plan (Lasa et al., 2009). This adds a more concrete plan of implementation to the concept and allows more organizations to implement the Lean model. By comparing the current state and the identified future state of a process, organizations are better able to define and achieve a working plan. Lean Six Sigma In a quest for a truly holistic approach to business and process improvement, many corporate organizations attempt to combine Six Sigma and Lean into practice simultaneously. Lean practices and techniques focus on streamlining processes, whereas Six Sigma practices and techniques help identify and eliminate root causes of problems (Evans & Lindsay, 2005). An effective combination of these methods would create an invaluable process improvement solution for many organizations. A case study presented by Fraser and Fraser (2009) discusses the successful implementation of a Lean Six Sigma model by the National Grid on a project concerning renewal prompts for vehicles. In this case, the organization used both models concurrently and matched the phases side by side. This is one variation of combining Lean and Six Sigma. In an attempt to uncover patterns of implementation between Lean Six Sigma, Shah, Chandrasekaran, and Linderman (2008) conducted a research study using implementation and performance data from a sample of 2511 industrial plants. They found that implementation of Lean practices improved the likelihood of implementing Six Sigma and indicated a significant difference in the performance levels between those who did and those who did not implement these models. However, there was no report of how the organizations combined the two models.
  • 17. Since there was not a structured method of how to combine the models, there could be great variation between these organizations. Tarantino (2009) supports the combination and explains that combining the two provides a powerful best practice for process and customer service improvements if done so in the hands of focused experts. However, the reviews are mixed. Bendall (2005) states, “It would be clearly desirable if a single process-improvement-based approach was available that effectively combined the approaches” (p. 975). He continued to explain that most attempts of combination resulted in a situation where one of the two methods was taken as a dominate, and the other as a subordinate. He pointed out that this is not a particularly good approach, since both methods have unique features and benefits that only more effective combinations would retain. Kirkpatrick Because Edelweiss Communications began its process improvement initiative by evaluating its current state using Kirkpatrick’s learning evaluation theory, it is necessary to include literature regarding this facet as well. The goal of this inclusion is to benchmark Edelweiss Communications’ choice of evaluation with other organizations. This provides the opportunity to compare Edelweiss Communications with those considered to be best-in-class. Benchmarking helps draw comparisons and determine which processes are adaptable to other organizations (Carter & Carmichael, 2009). Kirkpatrick’s learning evaluation theory consists of four levels of training evaluation: (a) reaction, (b) learning, (c) behaviour, and (d) results (1959). Rowden (1998) writes, “These often-cited levels describe a rubric that is often used to evaluate programs. It is the last two levels—behaviour change and results—that ROI is concerned with” (p. 32). Level One
  • 18. evaluation measures employee satisfaction with training. Level Two evaluation assesses the acquisition and retention of declarative knowledge, procedural knowledge, and attitude changes (Lupton, Weiss, & Peterson, 1999). Levels Three and Four, behaviour change and results, are the measurable changes we can see in employees and the benefits a company enjoys from employees implementing what they have learned (Rowden, 1998). Based upon these definitions, Kirkpatrick’s model may be effective in helping execute Six Sigma’s Measure phase. Quality of Literature Overall, the quality of the literature used in this review was very strong. All articles were peer reviewed and many were from popular notable publications such as the Journal of Applied Statistics and International Journal of Production Research. The literature provided excellent information about what is currently happening in corporate organizations. For each CPI method, information about its origin, its core structure or framework, and how well it works or is being received by corporate organizations was available. A broad variety of angles or points of view were also used in various articles. For example, Grant’s article (2008), provided overviews of Six Sigma, Lean, and a few less recognized methodologies. Jones et al. (2010) defined and described Six Sigma and then explored best practices for implementation. Bernett and Nentl (2010) conducted a survey about continual improvement and explored the findings and opinions of those surveyed. Nuran and Frasier (2009) as well as Matson and Stauffer (2009) each explored findings of case studies. This variety allowed for an exceptional sample of articles. Combining these articles into a whole helped build a multi-faceted view of the existing relationships between evaluations and process improvement efforts.
  • 19. However, some intrinsic limitations were present due to the lack of time allotted to perform the literature review. If more time were available, the opportunity would have been available to expand data collection methods to include visits to libraries and searches through tangible periodicals. Although the online search allows for easy duplication of the search process, it also creates a limitation of data to be collected. The literature also showed a limit in participants represented. All authors were journalists or researchers rather than facilitators, business leaders, or other participants that may have been able to share first-hand experiences with the success or failure of a specific CPI method. The inclusion of these points of views would have provided a greater diversity to the study and would have possibly brought about different results. Yet, the largest limit of this literature review was the lack of evaluation techniques explored. This is caused by both a lack of time to conduct research on what those techniques may be, and a lack of founding knowledge about evaluation techniques. This created a situation in which key search terms were not as strong as they could have been. This could have been resolved if more time were allotted for the literature review. However, the combination of both situations (lack of time and knowledge) results in the inability to research evaluation techniques to a greater extent. In literature reviews that will be conducted in future phases of this research study, a section devoted to evaluation techniques will be implemented.
  • 20. Gap in Literature Most of the literature in this review focused on describing and explaining the concepts, methodologies, and tools of each CPI method. They also discussed strengths, weaknesses, and success factors. However, individual company needs, goals, and context were excluded. Further information on organizational specifics is found in strategy literature which forces business leaders to complete in-depth analysis and cross referencing between the two areas. Because this is a timely and arduous task, most business leaders select a methodology for their company based on suggestions from friends, colleagues, gurus, and practitioner publications (Charlesworth, 2000). This indicates that there is a gap in the literature that would help business leaders assess their circumstances and match it with an appropriate process improvement methodology. In fact, Thawesaengskulthai and Tannock note that without an explicit decision support, organizations may base their selection on subjective judgments. A process improvement methodology selected through subjective judgements may not provide a company with all the features and benefits it needs. The literature also failed to explain how a company would implement a process improvement methodology. For example, the Define-Measure-Analyze-Improve-Control framework of Six Sigma provides a detailed breakdown of what to do, but fails to explain how to do so. Edelweiss Communications’ study is filling this gap by providing an explicit example of a study.
  • 21. Literature Review Summary This literature review explored existing content about how evaluation of training initiatives helps corporate organizations identify and implement process improvements. Models and concepts explored include Kaizen, Six Sigma, Lean, and Lean Six Sigma. The literature review allowed for study of each methodology in terms of origin, structure or framework, how they work, and the extent of their success. The review also showed findings on Kirkpatrick’s model of training evaluation in an attempt to find correlations between Edelweiss Communications’ current method of evaluation and how that may fit into the context of process improvements. Although no articles combined these issues within their own context, uniting these articles through this review helped build a multi-faceted view of the existing relationships between evaluations and process improvement efforts. The following items break down the literature review as compared to the literature search questions introduced at the beginning of this section. 1. What links and correlations exist between evaluation techniques or methods and process improvements in corporate settings? As noted in the Literature Gap section of this document, the articles failed to explain exactly how a process improvement methodology is put into action. Because of this gap, the links and correlations between evaluation techniques or methods and process improvements are also undocumented. However, the literature on Kirkpatrick’s learning evaluation theory suggest that Level Two assessments are a successful method for capturing data regarding the current state of a situation. But ultimately, links and
  • 22. correlations cannot be established based on the literature review alone. 2. What are the most effective methods of evaluation to help identify gaps in processes? As previously noted, methods of evaluation and process improvements were not identified in the literature. Therefore links between these two concepts were not explored. However, the literature on Kirkpatrick’s learning evaluation theory suggest that Level Two assessments are a successful method for capturing data regarding the current state of a situation. 3. What are the most effective methods of implementing process improvements? The literature review outlined the most prevalent methods of process improvements found. This includes Six Sigma, Lean, and Lean Six Sigma. The literature suggests that all three of these methods are effective, however, a delicately balanced combination of Six Sigma and Lean seem to hold the most promise as a complete and holistic approach to process improvements. Methods My primary data gathering method followed the protocol of statistical-observational study. To adhere to the statistical approach of the study, I collected quantitative data from a specific population (Roberts, 2010). I created and distributed a 30-question electronic assessment. The questions were based on factual knowledge and molded to comply with Level Two evaluations as defined by Kirkpatrick. This fulfills the statistical requirements of this type
  • 23. of study. The analysis of this data focused on the observational aspect of my statistical- observational study. As defined by Roberts, (2010) “In an observational study, the sample population being studied is measured, or surveyed, as it is…Data is simply gathered and correlations are investigated” (p. 1). I did not manipulate any data, I simply collected the information, restructured it to ensure proper analysis, and identified patterns or correlations. My secondary data gathering method was conducted in a focus group where I asked interview questions to collect viewpoints and thoughts from key players in the study. This exemplifies an action research aspect because it incorporates members of the community to help identify and solve problems. The data collected from this initiative contributes to Edelweiss Communications’ ability to assess and identify gaps in additional areas such as the effectiveness of weekly updates, training content, and delivery methods. By reassessing the priority and accountability levels of the weekly updates, Edelweiss Communications is now better able to stimulate change in this situation. However, the study may cause a conflict of goals for customer-facing employees. If we increase the accountability of their responsibilities regarding staying abreast of company changes, they may become overwhelmed. Because they have restrictions placed upon them in terms of time management and performance, enforcement of responsibilities (that may not have been enforced in the past) may hinder their performance in other areas, thus causing frustration and possible employee attrition.
  • 24. Participants The primary participants of this study were 1,918 customer-facing employees employed by Edelweiss Communications. There was not direct communication with them because their feedback was collected via a 30-question electronic assessment through a local learning management system (LMS). It also included two training managers employed by Edelweiss Communications with whom I conducted a focus group. I led the project, created the assessment, collected and analyzed data, and compiled this report. The content management department created the weekly updates mentioned throughout this study and delivered weekly conference calls to communicate its content to site leaders. Trainers and supervisors delivered the information in the weekly updates to customer-facing employees. The training department leaders sponsored the project and communicated data results and analysis to their superiors and the content management department up to and including presidents and vice presidents.
  • 25. Data Gathering Quantitative As previously mentioned, my primary data gathering method followed the protocol of statistical-observational study. To adhere to the statistical approach of the study, I collected quantitative data from approximately 1,918 existing customer-facing employees. I created and distributed a 30-question electronic assessment through an LMS. Employees had 30 minutes scheduled in their shift to complete the assessment. The 30-minute allocation may have been scheduled anywhere within a two week span. Participation in this assessment is a job requirement with enforcement from employees’ supervisors. Software capabilities prevented participants from skipping questions or returning to previous questions. Participants could however, exit the assessment and re-enter at the location where they left. The questions focused on factual information, were directly related to the information within the weekly updates, and covered information about company changes spanning Quarter One 2010 (January – March). Qualitative I conducted one focus group with two training managers. These training managers are located in different sites and were selected because they oversee the largest variety of site locations. I facilitated this over the phone on a group conference call as defined by the training managers’ schedules for the duration of one hour. I recited disclosures to them before the focus group began so they could understand the impact of their participation as listed in Appendix C.
  • 26. I already had an established rapport with these individuals through previous working relationships and was therefore able to help participants feel that they could say or express what they really felt or thought (Stringer, 2007). Refer to Appendix D for focus group questions. Ethical Procedures For the quantitative aspect of this study, participants were notified that data from the assessment would be collected for company informational purposes. They did not have the option to refuse participation because the assessment scores are also used for other purposes such as individualized coaching. However, they did have the option to exclude their assessment results from the data used in this study. Any and all data was and will be distributed solely within Edelweiss Communications on a need-to-know basis only. Data distributed throughout Edelweiss Communications was provided to leadership staff only and is regulated by internal records management policies and procedures. Data distributed to Dr. Jennifer VanBerschott and fellow students in the class was altered with pseudonyms or placeholders to protect the identity of the participants, the locations of the sites, the company name and profile, etc. Due to the vast amount of raw data (2,000 participants x 30 questions = 60,000 lines of data) and the fact that nearly all information within it was identifying, it was excluded from the final submission for this class. Instead, I used source data, which is the restructured data that served as my source for analysis. As previously noted, for the qualitative aspect of this study, I provided full disclosures as outlined in Appendix C. The disclosures listed (a) purpose of the study, (b) why the members of the focus group were selected for participation, (c) the members of the study who would be privy to their responses, (d) their options for participation, and (e) procedures for withdrawal.
  • 27. Checks for Rigor To ensure the outcomes of my research remain trustworthy, I incorporated multiple checks and remained balances (Stringer, 2007). Stringer (2007) states, “The credibility of a study is enhanced when multiple sources of information are incorporated” (p. 58). This is the concept of triangulation. By using both quantitative and qualitative data, I was able to implement this concept. qualitative In the statistical-observational portion of my study, I incorporate the concept of member observational incorporated checking (Stringer, 2007). Participants were, and are, able to view their scores based on their level of involvement. Customer-facing employees can view the results of their own assessment. -facing Supervisors can review the results of their own assessment and the results of their direct reports’ assessments. The training and content management departments and above can review all results. The project lead and sponsor (and above if applicable) in the training department can review all raw data, analysis, and reports. Figure 6 clarifies this concept visually. Project Lead and Sponsor Training / Content Mgmnt Supervisors Participants Figure 6
  • 28. I kept all records of raw data, reports, analysis, communications, and focus group notes to provide and show documentation of the procedures followed during this study to create to fulfill the concept of Dependability (Stringer, 2007). Data Analysis Procedures Quantitative I used Crystal Reports to capture data from all the employees who completed the assessment. Crystal Reports generated data about (a) employee job titles (supervisor or subordinate), (b) employee locations, (c) scores, and (d) which questions employees passed or failed. By utilizing the functions in Microsoft Excel such as sort, import, and pivot tables, I restructured the data from Crystal Reports to emphasize the most relevant information. The restructured data served as my source data for analysis and included (a) average scores of the assessment (by location and company-wide), (b) pass/fail percentages of the assessment (by location and company-wide), and (c) pass/fail percentages of each question (by location and company-wide). Please refer to Appendices E-G for complete details. In addition to this information, I also collected further data in various ways. I (a) searched archival scheduling records to determine the delivery method as related to each question, (b) collected the wording of each question as it appeared in the assessment from the development source files, and (c) collected the correct answer for each question from the development source files. Items b and c are directly related to identifiable company information; therefore they are listed in Appendix G as placeholders. The analysis of this data focused on the observational aspect of my statistical- observational study. As previously noted, I did not manipulate any data, I simply collected the
  • 29. information, restructured it to ensure proper analysis, and identified patterns or correlations. The data I analyzed included (a) average scores of the assessment (by location and company-wide - listed in Appendix E), (b) pass/fail percentages of the assessment (by location and company-wide - listed in Appendix E and F), and (c) pass/fail percentages of each question (by location and company-wide - listed in Appendix G). To analyze the data, I identified the aspects of the information that would help my audience understand the meaning of the data (Stringer, 2007). This falls in line with the method of analyzing key experiences as defined by Stringer. I focused on events and trends that impact my stakeholders. I began by categorizing and coding my data to identify events or experiences that may have some significance. While doing this, I searched for relationships between the following items: (a) the pass/fail percentage of the assessment and location (see E and F), (c) the pass/fail percentage of each question and location (see Appendix G), and (d) the pass/fail percentage of each question and delivery method (see Appendix G). This helped me find themes in the variety of relationships listed, which in turn, helped me identify elements that compose the experience for participants of the study. Qualitative To identify significant features and elements of my participants’ experiences, I analyzed my qualitative data through means of categorizing and coding as defined by Stringer (2007). To do this, I used the verbatim principle: using terms and concepts from the participants themselves (Stringer, 2007). I then (a) reviewed the collected data, (b) unitized the data, (c) categorized the data, (d) identified themes, (e) organized a category system, and (d) developed a report framework (Stringer, 2007).
  • 30. Schedule To ensure timely completion and organization throughout the duration of this project, I implemented and followed the schedule listed in the below table (Table 1 – Schedule). Table 1 - Schedule Item Task Type Completion Submit Action Research Proposal - Milestone June 21, 2010 Draft Revise Action Research Proposal Development June 22, 2010 – June 26, 2010 Submit Action Research Proposal Milestone June 26, 2010 Write Interview Questions Data Collection and Analysis June 21, 2010 Focus Group to collect qualitative Data Collection and Analysis June 29, 2010 data Run reports to collect quantitative Data Collection and Analysis June 28, 2010 – July 2, 2010 data Restructure reports/create source data Data Collection and Analysis June 28, 2010 – July 2, 2010 Analyze quantitative data Data Collection and Analysis July 3, 2010 – July 9, 2010 Analyze qualitative data Data Collection and Analysis July 3, 2010 – July 9, 2010 Write Literature Review Development June 27, 2010 – July 4, 2010 Submit Literature Review – Draft Milestone July 5, 2010 Revise Literature Review Development July 6, 2010 – July 10, 2010 Submit Literature Review Milestone July 10, 2010 Write Data Analysis Report Development July 11, 2010 – July 18, 2010 Submit Data Analysis Report – Draft Milestone July 19, 2010 Revise Data Analysis Report Development July 20, 2010 – July 26, 2010 Submit Data Analysis Report Milestone July 26, 2010 Write Final Research Report Development July 20, 2010 – July 25, 2010 Submit Final Research Report - Draft Milestone July 26, 2010 Revise Final Research Report Development July 27, 2010 – July 31, 2010 Submit Final Research Report Milestone July 31, 2020 No changes to the schedule were required throughout the duration of this study.
  • 31. Findings Quantitative Throughout the trends listed in this section, please note that Edelweiss Communications considers a score of 80% or higher to be a passing score. The following sections citing exceeds and fail percentages or vernacular refer to this designation. Notable Exceeds and Fail Trends Edelweiss Communications expects that supervisors ensure at least 80% of their teams pass the assessment. Only 81 of 229 supervisors accomplished this goal. In fact, of 229 supervisors, 2 had 100% of their employees fail. Refer to Appendix F for additional details. Of 229 supervisors, only 3 had a team larger than 10 and 100% of their employees passed the assessment. They were able to communicate and reinforce the content contained in the weekly updates despite their high employee count and were therefore recognized for their abilities to exceed expectations. Refer to Appendix F for additional details. These results signify that more employees are failing than exceeding and hint that customer-facing employees may be struggling to stay abreast of company changes.
  • 32. Notable Center Trends In general, pass percentages were low. The highest center (Snapdragon) had 80% of its employees pass. The lowest center (Rhododendron) had 50% of its employees pass. All centers had a median average of 67% passed. This data is particularly significant in the sense that it implies that employees are likely to be struggling to stay abreast of company changes. In fact, further details reveal that Snapdragon had the highest average scores at 86%. Rhododendron had the lowest average scores at 79%. These numbers reflect totals, which includes both customer- facing employees and supervisors. It is also important to note that Snapdragon had the highest percent passed and the highest scores. Rhododendron had the lowest percent passed and the lowest scores. Refer to Appendix E for additional details. Question Level Analysis – Below 80% Of 30 questions, 8 had less than 80% answered correctly. Refer to Appendix H for an analysis of causes listed in the order of lowest to highest average score. To protect company confidentiality, the actual questions are not listed, but only the question number and an analysis of possible reasons for the low score. This data should show the variety of possible causes and themes between them. To analyze the reasons these questions received low success rates, I used the analysis method as defined by Stringer (2007) called Antecedents and Consequences. To complete this process effectively, Stringer suggests that facilitators or participants identify (a) the core problem, (b) major antecedents to the problem, (c) significant factors related to the antecedents, (d) major negative consequences, and (e) other significant consequences. The remaining data in this section and in Appendix H was analyzed using these methods.
  • 33. A more global analysis revealed that more than 25% of the questions were answered incorrectly. This signified that employees were not staying abreast of company changes. It also indicated that the procedure in which weekly updates is distributed may not be effective. Major antecedents of this problem were identified through finding patterns in the possible causes of low scores for each question as listed in Appendix H. This includes employees (a) may not have used the Intranet properly, (b) may have expected information to be in different locations, (c) may not have been using the Intranet at all (d) were not receiving the information as required in team meetings, (e) were not receiving the information in full, (f) were not receiving it correctly (g) or may have missed some of the questions due to reading comprehension founded on (a) improperly written questions, (b) convoluted policies and procedures, or (c) insufficient reading comprehension skills. This analysis indicates that Level Two evaluation may be an effective method of evaluation to help identify gaps in processes since processes are prevalent in some of these findings. It provides numerical data which is vital to stimulating change in a corporate environment. Qualitative The results of the focus group provided great insight into the inner workings of Edelweiss Communications’ processes. My participants were selected because they have the largest amount of visibility into possible issues. Refer to Appendix D for focus group questions. Employees may not receive the information Throughout the focus group, participants clarified the process of distributing information in the weekly updates. Participant A explained that supervisors attend weekly update calls, and
  • 34. are then asked to review the information in team meetings. However, supervisors (a) may not follow the scheduled training times, (b) may not deliver the content correctly, or (c) may not cover the information at all. She stated, “Our trainers will show up to train during a team meeting, and no one is there.” This shows that team meetings are not happening at scheduled times and signifies that they may not be happening at all. As a result, employees are forced to use the Intranet as their only resource for the information, rather than learning the information in team meetings. Participant B agreed that supervisors may not cover the information because there is a lack of consistency in following protocol. She was only 70% confident that the messages were communicated. This data clearly indicated that the process of releasing information from the weekly updates was not effective. Lack of Priority After explaining that supervisors attend weekly update calls, and are then asked to review the information in team meetings, Participant A voiced her concern that the weekly calls are little more than someone reading the documents on the phone. She explained that as a result, audiences don’t feel a sense of urgency. She added that there are disconnects between the delivery of the call and the information reaching customer-facing employees. In fact, people give out codes for weekly update call attendance; so many supervisors may not even listen to the calls first-hand. She continued to explain that there is a lift in participation if the director is enforcing the priority of the weekly updates; but it must continue to be a priority. When the director enforces it, it becomes a priority. But they key is that this be driven from executive leaders. Participant B added that it really depends on the priority of the moment. Priorities of the moment make
  • 35. weekly updates seem like something that can be pushed aside. She agreed that it needs to be a consistent message from supervisors, managers, and directors. This sparked a conversation about ownership. Both participants explained that the training department acts as a messenger for certain information in weekly updates, but even then, Training does not own the initiative. Participant A stated, “It is important to the corporate training teams and operational teams, but it lacks priority and accountability on the front line.” Not only did this portion of the focus group identify gaps in the current process, but it also led to ideas to improve the situation. Leadership support is a vital component of making any process work and seemed to be a gap in the existing process. Delivery Methods In regard to the methods of training delivery, Participant B stated that instructor led training (ILT) has a stronger take-away if done by a trainer (rather than a supervisor) because trainers have the ability to make sure the transfer of knowledge occurs. To illustrate this point, she stated, “It’s more focused and intense.” Employees know it has an impact and importance because it was chosen for “training;” in team meetings, the message doesn’t seem as important. Since one method of delivery not only has a stronger take-away, but it sends an unspoken message to employees about the importance, urgency, and priority of the content, a possible gap in the process may be linked to the unspoken connotations that are tied to each method of delivery.
  • 36. Suggestions for Improvements Participant B continued to explain that to improve the extent to which customer-facing employees stay abreast of company changes, there must be an increase in consistency in following protocol. Participant A agreed and stated that Edelweiss Communications as a whole needs to focus on the weekly updates rather than the weekly update call. She recommended creating a check and balance system in team meeting environments – a sign-off process to help develop tracking methods for team meetings. She also mentioned the idea of an online survey where employees could select topics that they did or did not receive training on. Participant B agreed that there needs to be a consistent message from supervisors, managers, and directors, “There needs to be accountability, monitoring, and measurement.” Each of these ideas isolate and identify key components that could contribute to process improvements and are directly linked to the research question, “What are possible processes that may improve then state of the situation?” as found in Appendix D. Reaction to the Assessment In reaction to the assessment Edelweiss Communications distributed as a method of evaluation to help identify gaps in processes, Participant A stated that she hasn’t heard a lot of noise and that may be because she had not received the results or scores yet. Both participants agreed that an assessment is not the best way to evaluate if customer-facing employees are staying abreast of company changes or to find gaps in processes. Rather, they stated that we can get the best snapshot of gaps in the process by conducting focus groups with employees.
  • 37. Although my participants had the largest amount of visibility into possible issues, their feedback is only a part of the first cycle of action research. They are considered one of the primary stakeholders, but employees and their supervisors (secondary stakeholders) can provide valuable feedback about the causes and effects of employees failing to stay abreast of company changes as well. And like the training managers I interviewed, employees and their supervisors can provide valuable insight into possible solutions and the most effective methods of evaluation to help identify gaps in processes. The next cycle of action research will include focus groups with them to help further clarify and add information from a larger pool of stakeholders. Limitations By using both quantitative and qualitative data, I was able to implement the concept of triangulation, which states that, “The credibility of a study is enhanced when multiple sources of information are incorporated” (Stringer, 2007, p. 58). However, complete triangulation is not fully achieved at this time. It’s imperative to include qualitative analysis that typifies the points of views of both primary and secondary stakeholders. At this time, I have only accounted for a sample of primary stakeholders. There are many additional parties with whom I plan to conduct focus groups such as employees and their supervisors, the content management team that develops the weekly updates, and trainers. Once these items are complete and taken into account, the triangulation of this research project will be much stronger.
  • 38. Comparison of the Research to Literature Review The literature review that I conducted explored existing content about how evaluation of training initiatives helps corporate organizations identify and implement process improvements. Models and concepts explored included Kaizen, Six Sigma, Lean, and Lean Six Sigma. The literature review allowed for study of each methodology in terms of origin, structure or framework, how they work, and the extent of their success. The review also studied findings on Kirkpatrick’s model of training evaluation in an attempt to find correlations between Edelweiss Communications’ current method of evaluation and how that may fit into the context of process improvements. My study takes into account methods of evaluation, whereas many articles within my literature review did not. However, the method used in the study thus far, Kirkpatrick’s Level Two assessments, is not sufficient evidence on which to identify gaps in the process properly. This issue will be resolved by incorporating more qualitative analysis into further cycles of this study. My study failed to examine methods in which Edelweiss Communication can improve the processes that may not be working in its culture. The literature I studied covered methods such as Kaizen, Six Sigma, Lean, and Lean Six Sigma effectively and provided a valuable resource for me to tap into once I reach that phase of the research process. This cycle of research has clearly shown that improvements to Edelweiss Communications’ current processes are necessary. The following cycles will gather additional data from various points of view to reinforce this idea and explore methods of improvement. Once data is collected effectively, Edelweiss Communications will be in a position to select a process improvement method that meets the needs of Edelweiss Communications effectively.
  • 39. Summary of Findings Quantitative data analysis showed that the success rate for over 25% of the questions was below expectations. This global observation signified that employees were not staying abreast of company changes. It also indicated that the procedure in which weekly updates is distributed is not effective enough. Causes include: (a) employees may not have used the Intranet properly, (b) they may not have received the information as required in team meetings, and (c) reading comprehension may have influenced the data collected. This analysis indicated that Level Two analysis may be an effective method of evaluation to help identify gaps in processes since processes were prevalent in some of these findings. However, the qualitative analysis revealed that focus groups including all the primary and secondary stakeholders may provide different and valuable insight into how and why process improvements may be necessary. The feedback already collected qualitatively reveals a number of promising improvement ideas. Implications for Practice Based on these findings, it is evident that Edelweiss Communications must invest additional time into collecting qualitative data from a more diverse group of stakeholders. It must also (a) identify particular processes that are the causes of core problems, (b) select a methodology of process improvement, (c) design a plan to improve the process, and (d) implement the plan into action. This initiative will contribute to Edelweiss Communications’ ability to assess and identify gaps in additional areas such as the effectiveness of weekly updates, training content, and delivery methods. By reassessing the priority and accountability levels of the weekly updates, Edelweiss Communications may be able to stimulate improvements.
  • 40. Conclusion The quantitative findings of this study clearly indicate that there is a gap present. However, this data does not reveal the source of the gap. An analysis of questions breaks down the findings into more tangible possibilities. For example, the three most prevalent possibilities found in this analysis include (a) improperly written questions, (b) convoluted policies and procedures, or (c) insufficient reading comprehension skills. These hypotheses of causes are helpful, but are made more complete once the qualitative data of this study is integrated into the findings. The qualitative data gathered in this study revealed that employees may not have received the information as required in team meetings or may not have received it correctly. However, the results found within this study are applicable to Edelweiss Communications only. It is possible to replicate the study in another corporate environment, and is recommended to do so, but both quantitative and qualitative data are unique to Edelweiss Communications’ unique processes, policies, procedures, and culture. However, because of this tailored approach, the findings are incredible valuable, if only to a small group of individuals. An additional limitation is the lack of qualitative data. This is in part due to the (a) lack of time available to complete this phase of the study, (b) lack of departmental involvement in the study, and (c) lack of resources which made it difficult to identify and locate individuals who would be able to contribute to the study. Only a small number of primary stakeholders were interviewed in this phase of the process. Yet, this phase of the study revealed that other groups must be brought into the process and their thoughts and ideas must be integrated into the study. Since this is only phase one of the study, there will be ample time to put this into action, and the data that was collected provided further justification to do so.
  • 41. Ultimately, this study has proven beneficial for many parties. The information collected throughout this study will help the training department identify the strengths and weaknesses of their participation in the weekly update process. It will also help the content management department identify possible the strengths and weaknesses of their participation and ownership of the weekly update process. And finally, it will help employees explain possible low scores; it is unreasonable to expect employees to achieve high scores on an assessment about information that was not delivered, or not delivered correctly. This study has helped Edelweiss Communication take its first steps to help accomplish its ultimate goal of helping employees stay abreast of company changes.
  • 42. References Bendell, T. (2005). Structuring business process improvement methodologies. [Editorial]. [Electronic version]. Total Quality Management, Vol. 16, No. 8–9, 969-978. Bernett, R., & Nentl, N. (2010). Opinions and expectations about continuous improvement programs. [Editorial]. [Electronic version]. The Journal For Quality & Participation, January 2010, 35-38. Carter, L., & Carmichael , P. (2009). Best practices go beyond benchmarking. [Editorial]. [Electronic version]. Leadership Excellence, November 2009, 16-17. Caulcutt, R. (2001). Why is Six Sigma so successful? Journal of Applied Statistics, 28, 301 306. Charlesworth, K., A question of quality? A survey of Performance improvement initiatives, 2000 (British Quality Foundation, Institute of Management: London). Dictionary.com. (2010). Retrieved June 21, 2010 from Correlation: http://dictionary.reference.com/browse/correlation. Dictionary.com. (2010). Retrieved June 21, 2010 from Pattern: http://dictionary.reference.com/ browse/pattern. Evans, J.R. and Lindsay, W.M., The management and control of quality, 6th ed., 2005 (South Western College Publishers: Ohio). Fraser, N., & Fraser, J. (2009). Lean Six Sigma applied to a customer service process within a commercial finance organization. [Editorial]. [Electronic version]. Management Services, Autumn 2009, 25-31. Grant, V. (2008). Lean Six Sigma. [Editorial]. [Electronic version]. Management Services, Spring 2008, 20-23. Gygi, C., N. DeCarlo, and B. Williams, Six Sigma for Dummies, Wiley Publishing, Inc. (2005). Jones, E. C., Parast, M. M., & Adams, S. G. (2010). A framework for effective Six Sigma implementation. [Editorial]. [Electronic version]. Total Quality Management, Vol. 21, No. 4, 415-424. Knopf, J. W. (2006). Doing a literature review. [Editorial]. [Electronic version]. PS: Political Science & Politics, January 2006, 127-132.
  • 43. Kirkpatrick, D. L. (1959) Evaluating training programs, 2nd ed., Berrett Koehler, San Francisco. Kurtus, R. (2002). Steps to improve or grow your business. Retrieved June 21, 2010 from schoolforchampions.com: http://schoolforchampions.com/business/improvement_steps.htm. Lasa, I. S., De Castro, R., & Laburu, C. O. (2009). Extent of the use of Lean concepts proposed for a value stream mapping application. [Editorial]. [Electronic version]. Production Planning & Control, Vol. 20, No. 1, 82-98. Lean Terms & Definitions . (2010). Retrieved April 10, 2010 from The Northwest Lean Networks: http://www.nwlean.net/leandefs.htm#7wastes. Li, G., and Rajagopalan, S. (2008). Process improvement, learning, and real options. [Editorial]. [Electronic version]. Production and Operations Management, Vol. 17, No. 1, 61-74. Lupton, Robert A., John E. Weiss, and Robin T. Peterson (1999), “Sales Training Evaluation Model (STEM): A Conceptual Framework,” Industrial Marketing Management, 28, 1 (January), 73-86. Matson, E., and Stauffer, L. (2009). Developing an assessment tool for two organizations using Six Sigma principles. [Editorial]. [Electronic version]. Engineering Management Journal, Vol. 21 No. 4, 7-15. Naumann, E., and Giel, K. (1995). Customer satisfaction measurement and management. Wisconsin: ASQ Quality Press. Ortiz, C. (2006). All-out kaizen. [Editorial]. [Electronic version]. Industrial Engineer, April 2006, 30-34. Pakdil, F. Özkök, O. Dengiz, B. Kara, I. Selvi, N. Kargi, A. (2009). A systematic approach to reduce human and system-related errors causing customer dissatisfaction in a production environment. [Editorial]. [Electronic version].Total Quality Management, Vol. 20, No. 1, 129-137. Pande, P.S., Neuman, R.P., & Cavanagh, R.R. (2000). The Six Sigma way: How GE, Motorola and other top companies are honing their performance. New York: McGraw-Hill. Reichfeld, F. and Sasser, W.E. (1990). “Zero Defections: Quality Comes to Service”. Harvard Business Review, September – October, 1-8. Roberts, D. (2010). Statistical Studies. Retrieved June 21, 2010 from Statistical Studies: http://www.regentsprep.org/regents/math/algtrig/ats1/statsurveylesson.htm.
  • 44. Rowden, R. W. (1998). The return-on-investment from training. [Editorial]. [Electronic version]. Business Forum, Volume 27 Number 1, 31-36. Shah, R., Chandrasekaran, A., & Linderman, K. (2008). In pursuit of implementation patterns: the context of Lean and Six Sigma. [Editorial]. [Electronic version]. International Journal of Production Research, Vol. 46, No. 23, 6679-6699. Stringer, E. T. (2007). Action Research. Thousand Oaks, California: Sage Publications. Tarantino, A. (2009). Risk management for the next generation. [Editorial]. [Electronic version]. Industrial Management , january/february 2009, 14-17. Thawesaengskulthai, N., & Tannock, J. (2008). A decision aid for selecting improvement methodologies. [Editorial]. [Electronic version]. International Journal of Production Research, Vol. 46, No. 23, 6721-6737. Toronto. (2010). Retrieved June 21, 2010 from Glossary of Terms: http://www.toronto.ca/ lobbying/glossary.htm. Wayne State University. (2010). Retrieved June 21, 2010 from eLearning Glossary: http://www.eng.wayne.edu/page.php?id=1263. Wikipedia.org. (2010). Retrieved June 21, 2010 from Customer Experience: http://en.wikipedia.org/wiki/customer_experience. Wikipedia.org. (2010). Retrieved June 21, 2010 from Instructor-led training: http://en.wikipedia.org/wiki/instructor-led_training. Wikipedia.org. (2010). Retrieved June 21, 2010 from Procedures: http://en.wikipedia.org/wiki/ procedures. Womack, J.P., Jones D.T., and Ross D., The Machine that Changed the World, Rawson Associates, New York, 1990, 323 PP.
  • 45. Appendix A – Six Sigma Flows and Definitions Shared Define During the Define phase, organizations establish project parameters, review and define goals, as well as identify customer needs and processes/products to be improved. Measure Measuring consists of evaluating the performance of the core business process and collect data needed for future comparisons. Analyze The purpose of the Analyze phase is to identify the causes of poor performance and to determine how the process can be improved. DMAIC DMADV Improve The purpose of the Improve Design In the Design phase, phase is to address the root organizations design, develop, causes of poor performance. and implement new procedures or tools. Control In the Control phase, Verify In the Verify phase, organizations organizations ensure that the ensure that the new procedures or improved procedures and tools tools are consistently applied. are consistently applied. (Matson & Stauffer, 2009)
  • 46. Appendix B – Lean Definitions Principle Term Definition Elimination of waste Waste Anything a paying customer is not willing to pay for. 7 categories of waste: • Over-production • Waiting • Transportation • Inventory • Motion • Over-processing • Defects Identification of the Value Value Anything a paying customer is willing to pay for Stream Achievement of Flow Flow Continuous evaluation of inventories, batch sizes, through the process and work methods & distribution Pacing by a pull signal Pull Production is restricted to actual demand rather than capacity Continuous pursuit of n/a perfection (NWLEAN Inc, 2010)
  • 47. Appendix C – Focus Group Disclosures I’m asking you to participate as a subject in a research project entitled "Weekly Updates and Process Improvements.” You’ve been selected for this focus group because you oversee the largest variety of site locations. The purpose of this study is to help customer-facing employees stay abreast of company changes. By identifying gaps and analyzing possible causes for them, the training department is equipped to assess the strengths and weaknesses of our current processes. If you choose to participate in this focus group your responses will be recorded and shared only with myself and the project sponsor. If you have additional information that you would like to share but not record, that is an option. We ask that you are as objective and open as possible. There are no reasonably foreseeable risks associated with participating in this study. There may not be direct benefits to you from participating in this study. Participation is voluntary. You have the right to refuse to be in this study. If you decide to be in the study and change your mind, you have the right to discontinue participation at any time. However, any information already used up to the point when you withdraw consent, will not be removed from the study’s data. Whatever you decide, you will not lose any benefits to which you are otherwise entitled. This report will not contain personal identifying information such as your name or department. Your participation will not be identified and information you provide will be kept confidential and not reported to others outside the research project in a way that personally identifies you. You may ask any questions about this project of the researcher or project sponsor at any time.
  • 48. Appendix D – Focus Group Questions 1. To what extent do you feel customer-facing employees are currently staying abreast of company changes that are outlined in weekly updates? 2. How are processes involved in the retention of weekly updates? 3. How would you describe the current situations surrounding how we release weekly updates? a. What is working? b. What is not working? 4. Are things different in different locations? a. If so, how are they different? 5. Do you see any trends between delivery methods and how well employees are retaining the information in the weekly updates? 6. What is the current priority of the involvement of employees and supervisors as related to the weekly updates? 7. Is there consistency among how supervisors reinforce the importance of the weekly updates? 8. What motivates employees to care about the weekly updates? 9. Are there consequences for employees who do not stay abreast of weekly updates? 10. How did employees receive the assessment we released for Q1? 11. Do you think the assessment is an accurate way to see if employees are retaining information from the weekly updates a. What worked? b. What didn’t?
  • 49. Appendix E – Average Pass Percentages and Average Scores Site Type Total Passed % Passed Ave Count Score % Forsythia Employee 265 184 69% 58% Supervisor 11 11 100% 96% Total 276 195 71% 83% Gloxinia Employee 227 159 70% 82% Supervisor 14 10 71% 87% Total 241 169 70% 82% Laburnum Employee 344 196 57% 80% Supervisor 11 9 82% 90% Total 355 205 58% 80% Wisteria Employee 101 69 68% 84% Supervisor 3 3 100% 96% Total 104 72 69% 84% Whortleberry Employee 181 116 64% 84% Supervisor 8 8 100% 90% Total 189 124 66% 84% Rhododendron Employee 197 95 48% 70% Supervisor 11 8 73% 85% Total 208 103 50% 79% Snapdragon Employee 227 185 81% 85% Supervisor 7 7 100% 98% Total 234 192 82% 86% Valerian Employee 397 284 72% 85% Supervisor 24 22 92% 91% Total 421 306 73% 85% Across All Sites 2028 1366 67% 83%
  • 50. Appendix F - Percent Passed Detail – Listed by Site Location Site Manager Percentage Employees Employees Passed Passed Grand Failed Total Forsythia Supervisor Name 12 2 14 14% Supervisor Name 6 5 11 45% Supervisor Name 5 6 11 55% Supervisor Name 4 5 9 56% Supervisor Name 6 8 14 57% Supervisor Name 7 11 18 61% Supervisor Name 5 8 13 62% Supervisor Name 6 10 16 63% Supervisor Name 5 11 16 69% Supervisor Name 5 11 16 69% Supervisor Name 3 8 11 73% Supervisor Name 3 8 11 73% Supervisor Name 3 9 12 75% Supervisor Name 3 9 12 75% Supervisor Name 3 10 13 77% Supervisor Name 2 15 17 88% Supervisor Name 1 11 12 92% Supervisor Name 13 13 100% Supervisor Name 2 2 100% Supervisor Name 7 7 100% Supervisor Name 9 9 100% Supervisor Name 13 13 100% Supervisor Name 3 3 100% Forsythia Totals 79 194 273 71% Gloxinia Supervisor Name 4 4 0% Supervisor Name 1 1 0% Supervisor Name 5 3 8 38% Supervisor Name 3 2 5 40% Supervisor Name 7 7 14 50% Supervisor Name 2 2 4 50% Supervisor Name 1 1 2 50% Supervisor Name 2 2 4 50% Supervisor Name 2 2 4 50% Supervisor Name 5 7 12 58% Supervisor Name 4 6 10 60%
  • 51. Supervisor Name 5 9 14 64% Supervisor Name 4 8 12 67% Supervisor Name 2 4 6 67% Supervisor Name 1 2 3 67% Supervisor Name 1 2 3 67% Supervisor Name 3 9 12 75% Supervisor Name 4 13 17 76% Supervisor Name 3 10 13 77% Supervisor Name 2 8 10 80% Supervisor Name 1 5 6 83% Supervisor Name 1 9 10 90% Supervisor Name 9 9 100% Supervisor Name 1 1 100% Supervisor Name 3 3 100% Supervisor Name 16 16 100% Supervisor Name 3 3 100% Supervisor Name 8 8 100% Supervisor Name 3 3 100% Gloxinia Totals 63 154 217 71% Laburnum Supervisor Name 4 4 0% Supervisor Name 1 1 0% Supervisor Name 7 2 9 22% Supervisor Name 7 2 9 22% Supervisor Name 3 1 4 25% Supervisor Name 3 1 4 25% Supervisor Name 2 1 3 33% Supervisor Name 6 3 9 33% Supervisor Name 4 2 6 33% Supervisor Name 4 2 6 33% Supervisor Name 6 3 9 33% Supervisor Name 4 2 6 33% Supervisor Name 14 9 23 39% Supervisor Name 3 2 5 40% Supervisor Name 3 2 5 40% Supervisor Name 3 2 5 40% Supervisor Name 3 2 5 40% Supervisor Name 6 4 10 40% Supervisor Name 5 4 9 44% Supervisor Name 4 4 8 50% Supervisor Name 1 1 2 50% Supervisor Name 5 5 10 50%
  • 52. Supervisor Name 2 2 4 50% Supervisor Name 4 5 9 56% Supervisor Name 3 4 7 57% Supervisor Name 2 3 5 60% Supervisor Name 4 6 10 60% Supervisor Name 3 5 8 63% Supervisor Name 3 5 8 63% Supervisor Name 4 7 11 64% Supervisor Name 1 2 3 67% Supervisor Name 2 4 6 67% Supervisor Name 2 4 6 67% Supervisor Name 1 2 3 67% Supervisor Name 3 6 9 67% Supervisor Name 2 5 7 71% Supervisor Name 2 5 7 71% Supervisor Name 2 7 9 78% Supervisor Name 2 7 9 78% Supervisor Name 1 8 9 89% Supervisor Name 1 8 9 89% Supervisor Name 2 2 100% Supervisor Name 2 2 100% Supervisor Name 9 9 100% Supervisor Name 1 1 100% Supervisor Name 3 3 100% Supervisor Name 4 4 100% Supervisor Name 3 3 100% Supervisor Name 8 8 100% Supervisor Name 3 3 100% Supervisor Name 6 6 100% Supervisor Name 1 1 100% Laburnum 142 191 333 57% Totals Wisteria Supervisor Name 1 1 0% Supervisor Name 7 3 10 30% Supervisor Name 5 5 10 50% Supervisor Name 4 7 11 64% Supervisor Name 4 7 11 64% Supervisor Name 3 8 11 73% Supervisor Name 2 6 8 75% Supervisor Name 2 6 8 75% Supervisor Name 2 6 8 75% Supervisor Name 1 3 4 75%
  • 53. Supervisor Name 1 5 6 83% Supervisor Name 1 9 10 90% Supervisor Name 3 3 100% Supervisor Name 1 1 100% Wisteria Totals 33 69 102 68% Whortleberry Supervisor Name 9 2 11 18% Supervisor Name 4 3 7 43% Supervisor Name 1 1 2 50% Supervisor Name 4 4 8 50% Supervisor Name 6 6 12 50% Supervisor Name 4 5 9 56% Supervisor Name 4 5 9 56% Supervisor Name 7 11 18 61% Supervisor Name 3 5 8 63% Supervisor Name 1 2 3 67% Supervisor Name 2 5 7 71% Supervisor Name 1 3 4 75% Supervisor Name 1 3 4 75% Supervisor Name 3 10 13 77% Supervisor Name 2 7 9 78% Supervisor Name 2 8 10 80% Supervisor Name 2 8 10 80% Supervisor Name 1 8 9 89% Supervisor Name 3 3 100% Supervisor Name 4 4 100% Supervisor Name 4 4 100% Supervisor Name 2 2 100% Supervisor Name 3 3 100% Supervisor Name 4 4 100% Whortleberry 57 116 173 67% Totals Rhododendron Supervisor Name 1 1 0% Supervisor Name 3 3 0% Supervisor Name 3 3 0% Supervisor Name 6 2 8 25% Supervisor Name 8 3 11 27% Supervisor Name 8 4 12 33% Supervisor Name 10 5 15 33% Supervisor Name 4 2 6 33% Supervisor Name 7 4 11 36% Supervisor Name 3 2 5 40% Supervisor Name 4 3 7 43%
  • 54. Supervisor Name 6 5 11 45% Supervisor Name 6 6 12 50% Supervisor Name 5 5 10 50% Supervisor Name 3 3 6 50% Supervisor Name 2 2 4 50% Supervisor Name 4 4 8 50% Supervisor Name 4 6 10 60% Supervisor Name 5 9 14 64% Supervisor Name 3 7 10 70% Supervisor Name 3 8 11 73% Supervisor Name 3 10 13 77% Supervisor Name 1 4 5 80% Supervisor Name 1 1 100% Supervisor Name 2 2 100% Supervisor Name 4 4 100% Supervisor Name 2 2 100% Rhododendron 102 103 205 50% Totals Snapdragon Supervisor Name 4 4 8 50% Supervisor Name 4 5 9 56% Supervisor Name 4 7 11 64% Supervisor Name 4 10 14 71% Supervisor Name 4 10 14 71% Supervisor Name 2 5 7 71% Supervisor Name 3 8 11 73% Supervisor Name 4 12 16 75% Supervisor Name 2 8 10 80% Supervisor Name 2 8 10 80% Supervisor Name 1 5 6 83% Supervisor Name 2 10 12 83% Supervisor Name 1 6 7 86% Supervisor Name 1 7 8 88% Supervisor Name 1 11 12 92% Supervisor Name 2 2 100% Supervisor Name 9 9 100% Supervisor Name 1 1 100% Supervisor Name 13 13 100% Supervisor Name 1 1 100% Supervisor Name 2 2 100% Supervisor Name 1 1 100% Supervisor Name 1 1 100% Supervisor Name 4 4 100%
  • 55. Supervisor Name 1 1 100% Supervisor Name 4 4 100% Supervisor Name 2 2 100% Snapdragon 39 157 196 80% Totals Valerian Supervisor Name 11 6 17 35% Supervisor Name 9 6 15 40% Supervisor Name 7 9 16 56% Supervisor Name 6 9 15 60% Supervisor Name 6 9 15 60% Supervisor Name 4 7 11 64% Supervisor Name 5 9 14 64% Supervisor Name 5 10 15 67% Supervisor Name 6 13 19 68% Supervisor Name 4 10 14 71% Supervisor Name 5 13 18 72% Supervisor Name 4 11 15 73% Supervisor Name 4 11 15 73% Supervisor Name 3 9 12 75% Supervisor Name 4 12 16 75% Supervisor Name 4 12 16 75% Supervisor Name 3 10 13 77% Supervisor Name 4 14 18 78% Supervisor Name 3 11 14 79% Supervisor Name 1 4 5 80% Supervisor Name 3 12 15 80% Supervisor Name 3 12 15 80% Supervisor Name 3 12 15 80% Supervisor Name 2 12 14 86% Supervisor Name 2 13 15 87% Supervisor Name 2 14 16 88% Supervisor Name 1 8 9 89% Supervisor Name 1 14 15 93% Supervisor Name 6 6 100% Supervisor Name 3 3 100% Supervisor Name 1 1 100% Supervisor Name 2 2 100% Valerian Totals 115 304 419 73% 1918
  • 56. Appendix G - Question Totals – Listed by Site Location Delivery Method Totals Gloxinia Wisteria Valerian Forsythia Laburnum Snapdragon Whortleberry Rhododendron Question Sample Question 1 94% 81% 94% 98% 98% 93% 91% 98% 97% (and correct answer) None Sample Question 2 69% 49% 62% 75% 50% 83% 74% 89% 67% (and correct answer) None Sample Question 3 96% 92% 95% 95% 98% 96% 95% 98% 95% (and correct answer) None Sample Question 4 75% 81% 65% 69% 75% 87% 64% 85% 73% (and correct answer) None Sample Question 5 86% 88% 82% 81% 86% 86% 83% 91% 87% (and correct answer) None Sample Question 6 92% 99% 95% 88% 83% 98% 83% 94% 97% (and correct answer) None Sample Question 7 81% 77% 78% 81% 89% 83% 80% 85% 78% (and correct answer) None Sample Question 8 70% 76% 72% 64% 70% 67% 65% 82% 69% (and correct answer) Team Meeting Sample Question 9 93% 96% 93% 91% 92% 99% 90% 96% 90% (and correct answer) Team Meeting Sample Question 10 97% 98% 96% 94% 98% 98% 95% 99% 97% (and correct answer) None Sample Question 11 83% 84% 84% 83% 80% 83% 79% 87% 82% (and correct answer) None Sample Question 12 86% 84% 91% 85% 90% 87% 72% 92% 88% (and correct answer) None
  • 57. Sample Question 13 77% 74% 77% 78% 73% 77% 73% 89% 77% (and correct answer) None Sample Question 14 65% 54% 72% 61% 65% 62% 55% 77% 72% (and correct answer) None Sample Question 15 81% 65% 78% 80% 96% 69% 87% 85% 89% (and correct answer) None Sample Question 16 95% 98% 96% 84% 98% 99% 92% 93% 97% (and correct answer) None Sample Question 17 73% 70% 78% 67% 71% 76% 70% 79% 75% (and correct answer) None Sample Question 18 95% 94% 96% 93% 97% 92% 98% 95% 97% 10 Minutes (and correct answer) Team Meeting Sample Question 19 54% 51% 56% 58% 55% 51% 44% 68% 51% 10 Minutes (and correct answer) Team Meeting Sample Question 20 92% 92% 93% 89% 89% 91% 91% 98% 90% 10 Minutes (and correct answer) Team Meeting Sample Question 21 94% 96% 96% 92% 96% 94% 89% 90% 95% 10 Minutes (and correct answer) Team Meeting Sample Question 22 93% 97% 96% 86% 95% 98% 85% 95% 95% (and correct answer) None Sample Question 23 90% 90% 95% 86% 87% 86% 90% 93% 93% (and correct answer) None Sample Question 24 76% 80% 75% 71% 78% 76% 71% 80% 79% (and correct answer) None Sample Question 25 72% 87% 77% 62% 64% 67% 48% 81% 89% 10 Minutes (and correct answer) Team Meeting Sample Question 26 85% 96% 93% 72% 74% 94% 66% 91% 96% (and correct answer) None Sample Question 27 92% 97% 90% 84% 94% 91% 89% 94% 95% (and correct answer) None Sample Question 28 92% 97% 95% 96% 95% 93% 97% 65% 96% Internal (and correct answer) Centers Sample Question 29 91% 94% 93% 84% 89% 91% 90% 93% 91% (and correct answer) 1 hour ILT
  • 58. Sample Question 30 84% 94% 89% 75% 77% 86% 72% 89% 91% 10 Minutes (and correct answer) Team Meeting Total 84% 84% 85% 81% 83% 85% 79% 88% 86%
  • 59. Running Head: IDENTIFYING PROCESS GAPS THROUGH EVALUATION 1 Appendix H – Question Breakdown for Low Scores Question Correct Possible Causes of Low Scores Number Percentage 19 54% Upon assessing the Policy and Procedure, this fact was stated clearly. It’s possible that this was a new procedure and was not given enough attention in team meetings. It’s also possible that this is a situation employees do not come across often. 14 65% If employees used Edelweiss’ Intranet for this question, it’s possible that they expected this information to be in the applicable Policy, but it is in the correlated Procedure. The Policy does refer to the Procedure properly and effectively. 2 69% Employees may have read the question incorrectly (“are not” rather than “are”). If employees in different queues have different procedures, this may have caused confusion. 8 70% It's possible that the product name caused confusion because it includes a written numeric value. However, the options available were not numerically close to the product name and would not cause further confusion. Also, only Wisteria and Snapdragon received team meeting time. This was not likely to be an area of focus for other centers. 17 73% There were numerous changes concerning this release with no training time scheduled. The date of the release changed and was announced the following week. The change of this related item may have caused employees to unconsciously add a sense of complication to the question. 25 72% This was a change of policy and was written effectively and accurately in the Policy and Procedure. It's possible this may not have been communicated effectively in team meetings. 4 75% There are four pieces of information in each response. This may have caused confusion. The possible answers to this question were extremely similar and lengthy. It's possible that some employees may have known the correct answer but selected the wrong option due to reading comprehension 24 76% This was not new information. The weekly update explained a deviation from this information for some areas. Therefore, employees may have chosen either of the two other distracters because they assumed the question would be focusing on a change.
  • 60. IDENTIFYING PROCESS GAPS THROUGH EVALUATION 60 Appendix I – Definitions Correlation - The simultaneous change in value of two numerically valued variables (Dictionary.com, 2010) Customer Experience – The sum of all experiences a customer has with a supplier of goods or services, over the duration of their relationship with that supplier (Wikipedia.org, 2010) Customer-Facing Employees – People who are paid for the performance of their duties by Edelweiss Communications who speak directly to customers (in part from: Toronto, 2010) eLearning – Internet-enabled learning that encompasses training, education, just-in-time information, and communication (Wayne State University, 2010) Instructor Led Training – (ILT) The practice of face-to-face delivery of training and learning material between an instructor and learners, either individuals or groups (Wikipedia.org, 2010) Pattern - A combination of qualities, acts, tendencies, etc., forming a consistent or characteristic arrangement (Dictionary.com, 2010) Policies – The rules, conventions, and protocols adopted by Edelweiss Communications, of which it requires employees to adhere Procedures – A specified series of actions executed in the same manner to always obtain the same result under the same circumstances (Wikipedia.org, 2010) Team Meetings – Meetings where employees collaborate in a group to share information facilitated by employees’ direct supervisors