How do you shadow an auditor? Did you learn anything?

If you are shadowing, you are taking notes so you can discuss your observations with the person you are shadowing later.

Somewhere in your procedure for “Quality Audits”, I’ll bet there is a section on auditor competency. Most companies require that the auditor has completed either a course for internal auditor or a lead auditor course. If the course had an exam, then you might even have evidence for training effectiveness. Demonstrating competency is much harder. One way is to review internal audit reports, but writing reports is just part of what an auditor does. How can you evaluate an auditor’s ability to interview people, take notes, follow audit trails, and manage their time? The most common solution is to require that the auditor “shadow” a more experienced auditor several times, and then the trainee will be “shadowed” by the trainer.

I can’t remember posting any music from John Mayer and the song title fits our subject for this blog.

Shadowing 1st Party Audits:

ISO 19011:2011 defines 1st party audits as internal audits. When 1st party auditors are being shadowed by trainer, or vice versa, there are many opportunities for training. The key successful training of auditors is to recognize teachable moments.

When the trainer is auditing, the trainer should look for opportunities to ask the trainee, “What should I do now?” or “What information do I need to record?” In these situations, the trainer is asking the trainee what they should do BEFORE they do it. If the trainee is not sure, the trainer should explain what, why and how at that moment with real examples.

When the trainer is shadowing, the trainer should watch and wait for a missed opportunity to gather important information. In these situations, the trainer must resist guiding the trainee until after the trainee appears to be done. When it happens sometimes the best tool is simply asking, “Are you sure you got all the information you came for?”

Here are five (5) mistakes that I have observed trainers make when they were shadowing:

1. Splitting up, instead of staying together, is one of the more common mistakes I have observed. This happens when people are more interested in completing an audit than taking every advantage of training opportunities. The trainee may be capable of auditing on their own, but this is no excuse for tag teaming the auditee. This is unfair to the trainee AND the auditee. If an audit is running behind schedule, this is the perfect time to teach a trainee how to recover some time in their schedule. Time management is after all one of the hardest skills for auditors to master.

2. Staying in the conference room, instead of going to where the work is done, is a common criticism of auditors. If the information you need to audit can be found in a conference room, then you could have completed the audit remotely. This type of audit teaches new auditors very little other than how to take notes. These are basic skills that auditors should master in a classroom prior to shadowing.

3. Choosing an administrative process is a mistake, because administrative processes limit the number of aspects of the process approach that can be practiced by an auditor-in-training. Administrative processes rarely have equipment that requires validation or calibration, and both the process inputs and outputs consist only of paperwork, forms or computer records. With raw materials and finished goods to process, the job of the auditor is more challenging because there is more to be aware of.

4. Not providing honest feedback is a huge mistake. Auditors need to be thick skinned or they don’t belong in a role where they are going to criticize others. Before you begin telling other people how to improve, you first need to self-reflect and identify your own strengths and weaknesses. Understanding your own perspective, strengths, weaknesses, and prejudices is critical to being an effective assessor. As a trainer, it is your job to help new auditors to self-reflect and accurately rate their performance against objective standards.

5. “Silent Shadowing” has no value at all. By this I mean shadowing another auditor without asking questions. If you are a trainee you should be mentally pretending you are doing the audit. Whenever the trainer does something different from the way you would do things, you should make a note so you can ask, “Why did you do that?” If you are trainer you should also be mentally pretending you are doing the audit. It is not enough to be present. You job is to identify opportunities for the trainee to improve. The better the trainee, the tougher your job becomes. This is why I training other auditors has helped me improve my own auditing skills.

Shadowing 2nd Party Audits:

If you are developing a new supplier quality engineer that is responsible for performing supplier audits, it is recommended to observe the auditor during some actual supplier audits. Supplier audits are defined as 2nd party audits in the ISO 19011:2011 Standard. The purpose of these audits is not to verify conformity to all the aspects of ISO 13485. Instead, the primary purpose of these audits is to verify that the supplier has adequate controls in place to consistently manufacture conforming product for your company. Therefore, processes such as Management Review (Clause 5.6) and Internal Auditing (Clause 8.2.2) are not typically sampled during a 2nd party audit.

The two most valuable process for a 2nd party auditor to sample are: 1) incoming inspection, and 2) production controls. Using the process approach to auditing, the 2nd party auditor will have an opportunity to verify that the supplier has adequate controls for documents and records for both of these process. Training records for personnel performing these activities can be sampled. The adequacy of raw material storage can be evaluated by following the flow of accepted raw materials leaving the incoming inspection area. Calibration records can be sampled by gathering equipment numbers from calibrated equipment in use by both processes. Even process validation procedures can be assessed by comparing the actual process parameters being used in manufacturing with the documented process parameters in the most recent validation or re-validation reports.

My recommendation is to have the trainee shadow the trainer during the process audit of the incoming inspection process and for the trainer to shadow the trainee during the process audit of production processes. In between the two process audits, the trainee should be asking questions to help them fully understand the process approach to auditing. Supplier auditors should also be coached on techniques for overcoming resistance to observe processes that may involve trade secrets or where competitor products may also be present. During the audit of production processes, the trainer may periodically prompt the trainee to gather information that will be needed for following audit trails to calibration records, document control or for comparing with the validated process parameters. The “teachable moment” is immediately after the trainee missed an opportunity, but while the trainee is still close enough to go back and capture the missing details.

Shadowing 3rd Party Audits:

Use your FDA inspections and ISO certification audits as an opportunity to shadow experienced auditors and to learn what they are looking for.

If you are going to shadow a 3rd party auditor, I recommend two specific people to “shadow” the auditor. First, the process owner should be the guide for whichever process is being audited. This is the person that will be responsible for addressing any nonconformities found in the area, and they should be present during interviews–although they should be coached on when to comment and when to remain quiet and simply observe.  Second, the person that performed an internal audit of the process being audited should be present if at all possible. This person will benefit from seeing how a professional 3rd party auditor performs a process audit, because they will know which things to look for in the future so that auditees in that area are prepared for the next external audit.

If you are an audit program manager, and you would like to learn “What Makes World Class Audit Programs Different?”, please contact me. I am co-teaching an advanced course for audit program managers in April 2013.

For other sources of information related to auditor shadowing, please check out the following links:

1. Internal Auditor Training – Shadowing external auditor? – from Elsmar Cove

2. Developing Supplier Quality Auditor Training Programs – by Seth Mailhot at NixonPeabody

Advertisement

Burn the binders and get a Wiki

Procedures can always be improved, but our goal is to make better products—not better procedures. So what could possibly be so interesting about document control that I feel compelled to write another post about “blah, blah, blah?”

          I read an article about using Wiki’s for document control.

For today’s entertainment I selected another famous Brit, but this song is the only piece I have heard him sing in French.

A Wiki is just a collaborative environment where anyone can add, delete and edit content. All changes are saved and Wiki’s can be controlled—while simultaneously being available to everyone. The most famous of all Wikis is Wikipedia.

In 2009, Francisco Castaño (a.k.a. – Pancho) began a discussion thread to explain how his company was using a Wiki to manage their documentation system. In the last month, ASQ published an update on the status of Pancho’s Wiki process for document control.

In most companies, the process owner writes procedures and other people in the company rarely comment on minor errors. In the most dysfunctional companies, the Quality Department writes the procedures for the rest of the company or outsources it to consultants. Reviewing and editing procedures should be the responsibility of everyone in the company, but I never considered the possibility of having everyone within a company edit procedures simultaneously—until I saw Pancho’s thread. Throughout the discussion others have indicated that they also tried using Wikis to optimize content. This is a genius idea that is coming of age.

Many QMS consultants, including myself, have written procedures for clients. Sometimes this is part of the consulting business model. In these cases, the consultant writes a procedure once and edits it forever—while getting paid a modest fee each time a client asks for a “new” procedure. I often think that it would make more sense to do something like Linux developers have done—use the collaboration of QMS experts around the world to create a generic procedure that is free to everyone. Using Wiki’s that are publicly available this is entirely possible.

In the very near future (hopefully 2013), the responsibilities section of our procedures will fundamentally change. Instead of reading and understanding, everyone will be responsible for writing and editing (oh no, I’ll have to create a new learning pyramid).

Quality will no longer be responsible for writing procedures. Instead, the quality function can focus on monitoring, measuring, data analysis, and improvement of processes and product. The downside is that we will need fewer people in document control.

If you want to learn more about Wiki for document control, follow this thread I found on Elsmar Cove. It rich in content and even the moderators have been forced to rethink their preconceptions.

You should also read two articles by Pancho:

1. Using a Wiki for Document Control
2. Using a Wiki to Implement a Quality Management System

3 Ways to Fix the 510(k) Process: Self-Surveys, Scorecards and Modular Submissions

Modular submissions are already used for PMA submissions. Self-surveys and scorecards are tools that most companies utilize to evaluate vendors. Why not implement these solutions to make 510(k) reviews more efficient?

For entertainment we have Pomplamoose’s cover of “Single Ladies”. My wife Lisa is a big fan of Pomplamoose, and this song is one of my favorites.

A few weeks ago a posted a blog about the Triage pilot program at the FDA. I received some great comments by email and I thought I would go a little more in depth with some specific ideas for improvement of the 510(k) process. Here’s the argument for considering these three proven methods:

Self-Surveys

In my previous posting about the Triage pilot program, I suggested using the existing FDA traditional 510(k) screening checklist and converting this into a similar “SmartForm”. Another way to think of this concept is by comparing it with a “Self-Survey.” Self-surveys are sent by companies to suppliers in order to gather information about the supplier as justification for approving the supplier; Elsmar Cove has some discussion threads specific to the supplier self-surveys if you are unfamiliar with this method of torture. The critical step in the design of surveys is to require the submitter to provide references to procedures and forms or to explain why something is not applicable. This same strategy is used by BSI for their auditor combined checklists. Instead of checking “yes/no”, the auditor must reference a page in their audit notes where the objective evidence of conformity or nonconformity can be found. A submitter should fill in the checklist, rather than an FDA reviewer, because this forces the submitter to verify that everything required is included. Canada has a similar requirement called a “submission traceability table” for Medical Device License Applications (see Appendix A). Self-surveys also replace some of the tedious searching by a reviewer with cross-referencing work by the submitter.

Scorecards

Another tool that supplier quality uses for supplier evaluations is the Scorecard; Elsmar Cove has a few discussion threads including one with an example to download. For the purpose of the 510(k) process, I suggest developing scorecards for both the reviewer AND the submitter. The primary metrics for these scorecards would be on-time delivery and completeness of the submission for a submitter. The “on-time delivery” requires advanced planning and communication of the submission with the FDA. This is important so that the FDA has adequate time prior to submission to identify the best reviewer(s) for the submission. The completeness of the submission should be 100% of a self-survey, SmartForm or checklist is used to prepare the submission. The primary metrics for the reviewer would be on-time completion of the review and accuracy of the review.  The FDA already has target turn-around timescales for decisions (i.e. – 90 days), but there are different phases of review and multiple people the are involved in the reviews. Therefore, the measurement of reviewer time should be more granular. The accuracy of the reviewers should be validated by requiring all deficiencies to be re-evaluated by a peer or superior prior to involving the company. Submission sections without any findings should also be reviewed on a sampling basis as a double check. Over time, the FDA should be able to use these scorecards to match up a reviewer with a submitter. It is critical that at least one of the parties is experienced so we don’ t have the “blind leading the blind.” For those that are offended by the concept of a required second reviewer–get over it. Radiologists are periodically graded with images that are “red herrings.”

Modular Submissions

My 3rd suggestion is to consider adopting some of the pre-market approval (PMA) processes for the 510(k) process. In particular pre-IDE meetings and modular submissions seam to be logical process improvements. There is typically one component of the submission that is a little behind the rest and holding up a submission. Under the current system, nothing is submitted or reviewed for a 510(k) unless it is complete. However, it would enable companies to get new and improved products to market faster if submissions were modular. Validation such as shelf-life and sterilization validation is rarely the cause for an “Not Substantially Equivalent” (NSE) letter, but these tests are routinely the last few reports completed for a submission. Adopting a modular submission process for 510(k) would allow companies to submit sections of the submission as they are completed. This modular approach would alleviate the time pressure on both sides, and this proposed change should result in earlier product launch dates for industry. The other component of this process is the pre-IDE Meeting. Prior to initiating a clinical study, companies will submit a plan for the study to the FDA. The intent is to obtain agreement on the validation testing that will be performed by the company–including the number of patients and the design of the Clinical. These meetings would also be valuable for 510(k) submissions where the company and the FDA need a forum to discuss what verification and validation testing will be required–especially for mixed-predicate devices and devices that are significantly different from a predicate device.

What do you think about these proposed changes to the 510(k) process?

Please share your own ideas for improving the 510(k) process–including any comments regarding the FDA‘s plans for change.

The Ultimate Design Control SOP

Disclaimer: There is no need to create the Ultimate Design Control SOP. We need medical devices that are safer and more effective.

If Adele is worthy of six Grammy Awards, she’s probably worthy of a blog link too. Rumor has it that this is my personal favorite from Adele.

In my previous blog posting, I indicated six things that medical device companies can do to improve design controls. While the last posting focused on better design team leaders (WANTED: Design Team Needs Über-Leader), this posting focuses on writing stronger procedures. I shared some of my thoughts on writing design control procedures just a few weeks ago, but my polls and LinkedIn Group discussions generated great feedback regarding design control procedures.

One of the people that responded to my poll commented that there was no option in the poll for “zero”. Design controls do not typically apply to contract manufacturers. These companies make what other companies design. Therefore, their Quality Manual will indicate that Clause 7.3 of the ISO 13485 Standard is excluded. If this describes your company, sit back and enjoy the music.

Another popular vote was “one”. If you only have one procedure for design controls, this meets the requirements. It might even be quite effective.

When I followed up to poll respondents asking how many pages their procedures were, a few people suggested “one page”. These people are subscribing to the concept of using flow charts instead of text to define the design control process. In fact, I use the following diagram to describe the design process all the time: The Waterfall Diagram!

From the US FDA Website.

I first saw this in the first AAMI course I took on Design Controls. This is on the FDA website somewhere too. To make this diagram effective as a procedure, we might need to include some references, such as: work instructions, forms, the US FDA guidance document for Design Controls, and Clause 7.3 of the ISO Standard.

The bulk of the remaining respondents indicated that their company has eight or more procedures related to design controls. If each of these procedures is short and specific to a single step in the Waterfall Diagram, this type of documentation structure works well. Unfortunately, many of these procedures are a bit longer.

If your company designs software, active implantable devices, or a variety of device types—it may be necessary to have more than one procedure just to address these more complex design challenges. If your company has eight lengthy procedures to design Class 1 devices that are all in the same device family, then the design process could lose some fat.

In a perfect world everyone on the design team would be well-trained and experienced. Unfortunately, we all have to learn somehow. Therefore, to improve the effectiveness of the team we create design procedures for the team to follow. As an auditor and consultant I have reviewed 100+ design control processes. One observation is that longer procedures are not followed consistently. Therefore, keep it short. Another observed I have made is that well-design forms help teams with compliance.

Therefore, if you want to rewrite your design control SOP try the following steps:

1. Use a flow chart or diagram to illustrate the overall process
2. Keep work instructions and procedures short
3. Spend more time revising and updating forms instead of procedures
4. Train the entire team on design controls and risk management
5. Monitor and measure team effectiveness and implement correct actions when needed

The following is a link to the guidance document on design controls from the US FDA website.

Refer to my LinkedIn polls and discussions for more ideas about design control procedures:

1. Medical Devices Group
2. Elsmar Cove Quality Forum Members Group

In addition to the comments I made in this blog, please refer back to my earlier blog on how to write a procedure.

If I had a rocket launcher…

This week’s music video selection was recommended by my friend Greg. We were eating dinner together at 1776, and he was kind enough to share this amazing musician with me. I’m not a guitarist but he pointed out that Bruce Cockburn has a very unique style. He plays three different parts simultaneously. His thumb plays base on the top string while the other fingers play two separate melodies. WOW!

                 Are you frustrated? Do you wish for a rocket launcher? Maybe you would aim it at the C-level offices and pull the trigger.

                Sometimes we hear phrases like: “Well that’s just an ISO requirement.” This obvious lack of support by top management is what frustrates every Management Representative in the world.

                There was a question posted on the Elsmar Cove website on January 10^th (see previous blog for the link). In just 10 days there have been 153 postings in response to the original question. As I read through the various postings I saw several comments about a lack of support by top management. Rocket launchers are NOT the answer, but maybe a heavy bat…

                A little over a decade ago I was still learning how to supervise people. In an effort to educate myself further, I read a book (sorry can’t be sure which book anymore). In this book, the boss gave an employee a card with a picture of a baseball bat on it. The instructions provided with this magical card were to use it only when the boss failed to pay attention and the employee had something important to tell him.

                We all wish for a magical baseball bat, but unfortunately we are M-A-N-A-G-E-R-S. Along with the awesome title comes awesome responsibility. Managers are responsible for leading others. Subordinates are not the “others” I am referring to. The “others” are peers. If you cannot persuade your peers to support you, then you will fail as a manager. The Quality Department cannot fix all the problems. In fact, my philosophy is that Quality is responsible for recommending improvements, training people, and helping to implement. We assign corrective actions, but we should be assigning them to the process owner (i.e. – Manager) that is responsible for the area where the problems were created.

                If you need help persuading the unenlightened, try picking a project that is critical to the success of the stubborn one. If you can show someone that is currently a detractor how they can apply the Quality principles to help solve their problems, then you will have a convert. Converts become strong supporters. If the stubborn one happens to be at the top, figure out what the CEO’s initiatives are. Initiatives are easy to identify; they talk about it at least twenty times a week. Try showing the CEO how their initiatives can become Quality Objectives. Show them with graphs. Show up with solutions to their problem. Use the CAPA process as a framework. Show them how the management TEAM can fix it.

                If nothing seems to be working, you can always try reviewing some FDA MedWatch reports too–just to scare the crap out of the boss.

Management Representative

The video music selection for this week was a tune I heard at a restaurant called “1776” in Crystal Lake, IL. The restaurant played Chris Isaak recordings for the entire meal. Maybe the satellite radio station was stuck on the letter “I”.

 The idea for this posting was from a thread I found on Elsmar Cove:

http://elsmar.com/Forums/showthread.php?t=45658

One person posted a question about the requirement for the Management Representative (MR) to be a member of the organization’s management (see section 5.5.2 of ISO 9001:2008). Companies that are seeking initial certification sometimes struggle with this requirement. Some struggle because they do not have anyone in-house that is sufficiently trained to be the MR. Other companies struggle, because they are very small and outsource their QA functions to a consultant. The following blog is targeted at helping these companies.

     I audit companies to the ISO 13485 (medical QMS) & 9001 (QMS) Standards. The intent of both Standards was always to have the MR be part of management, but some companies did not interpret the Standards in this way. With the 2008 revision of 9001, the possibility of misinterpreting the meaning is much less likely. The companies that receive findings during the Stage 1 or Stage 2 audit for this requirement usually fall into one of two categories. Category #1: our company is small and the only person that really knows enough about ISO requirements is not a member of management. Category #2: our company is small and we outsource QA functions.

   The good news is that any manager can be assigned the responsibility of being MR. One of my clients assigned this responsibility to the VP of Sales. Another company assigned this responsibility to the Director of R&D. Both of these individuals had to put in the time to learn about Quality Management Systems, but both have embraced the challenge and I have learned a lot from them. They have a different perspective and bring a lot of value to the MR role.

    The bad news is: whomever you assign has to learn enough to be competent in the role.

   The definition of “Management” is typically a stumbling block. Most people think of managers requiring that they have other people reporting to them. This is not an absolute. The MR should report directly to a top manager such as the President or CEO to prevent conflicts of interest. As a manager, they should not require a lot of direct supervision and the President or CEO should not be overly burdened by adding one person to their list of direct reports.

   Some auditors like to see a “deputy MR” identified. My advice is to have the CEO or President sufficiently trained that they can be the “back-up” when the MR is on vacation. Every manager should know enough about their subordinate’s job duties that they can “fill in.”

   MR’s should be involved in senior staff meetings too, but not necessarily at the same frequency as every other senior staff manager. Typically operations and sales have the most frequent meetings with the CEO–often weekly. Finance is typically monthly. HR and the MR might be bi-monthly or Quarterly. Communication of the status of Quality Objectives should be regular reports to all senior staff, but you don’t have to have a Management Review to communicate the status. If the company is small enough to have only one QA person, there probably isn’t a need for more than one or two management review meetings per year.

   If your company has a finding against clause 5.5.2, I recommend the following actions:

1. Assign a person that is already a member of your senior staff as MR

2. Document the responsibility in the person’s job description

3. Document the responsibility in the org chart

4. Assign the person’s direct supervisor (typically the CEO or President) as a “deputy MR”

5. Find a good webinar on ISO training for the new MR and their boss (ideally one with a quiz and a certificate)

6. Have the new MR develop a 45 minute presentation for the senior staff on the topic of Management Responsibilities. This training should cover all of section 5 in the Standard.

7. Give the senior staff a 15 minute multiple choice quiz to evaluate effectiveness of the training.

8. Have the new MR discuss delegation of various management review inputs (see section 5.6.2) with their boss. Quality should be a shared responsibility and Management Reviews will be more effective if everyone takes part.

Elsmar Cove – Wikipedia for QA

While you are reading, here’s a seasonal favorite from one of my favorite singers (also from North Texas State) and a country singer that everyone will recognize.

After my last blog posting I planned to write a blog on supplier evaluation and re-evaluation. After a busy December, I finally have time to write again. Before I share my own ideas, I thought I would share a great resource with you: The Elsmar Cove Forum.

                Most of the people reading my blog are probably aware of a website called Elsmar Cove, but I think most people visit this site only when they need a quick answer to a question. It’s sort of like Wikipedia for Quality Assurance. Marc Smith is the creator of Elsmar Cove, http://elsmar.com/, and the forum just had its 15-year anniversary. This is a no frills website that has fantastic content and very little advertising. People from all over the world (~18,000 active participants) are contributing daily to this forum and many of the contributors are Quality and Regulatory experts. I like to use the site to keep up on best practices and trends in Quality. It also gives me an opportunity to learn from other types of Quality Systems such as: AS9100, TS16949, and ISO14001.

                Someone I used to work with had a saying he learned from his first boss: “A picture tells a thousand words, but a demonstration is better than a thousand pictures.” Therefore, I thought I would try to demonstrate the power of Elsmar Cove by researching best practices in supplier evaluation. Step 1: The first thing you do is visit the site. Step 2: Type “supplier evaluation” into the Google™ custom search. This will produce 100’s of links within the Elsmar Cove Forum related to supplier evaluation. Step 3: Skim the search results to find the entry or entries you are looking for. These search results include a supplier evaluation survey that you can download and adapt to your own company. If you need a quick solution, this is really fast and free.

                Another approach is to limit your search to Forum discussion threads. You can do this by going to the Forum Discussion page: http://elsmar.com/Forums/index.php. If you click on the tool bar link for “search” a pop-up window will appear. If you type “supplier evaluation” in this search bar, you will see 531 results presented in reverse chronological order. Below are a couple of threads that I thought were particularly good:

Benchmarking Supplier Certification Programs

http://elsmar.com/Forums/showthread.php?t=42595&highlight=supplier+evaluation

Service Supplier Rating where objective pass/fail data is not available

http://elsmar.com/Forums/showthread.php?t=44412&highlight=supplier+evaluation

Choosing Supplier Evaluation Methods – Determining what a Critical Supplier is

http://elsmar.com/Forums/showthread.php?t=10951&highlight=supplier+evaluation

Supplier Approval for Distributors of Equipment

http://elsmar.com/Forums/showthread.php?t=43508&highlight=supplier+evaluation

Second party audits – Supplier audits or product audits?

http://elsmar.com/Forums/showthread.php?t=30966&highlight=supplier+evaluation

How you do Receiving Inspection for Chemicals?

http://elsmar.com/Forums/showthread.php?t=44951&highlight=supplier+evaluation

Supplier Evaluation Responsibility

http://elsmar.com/Forums/showthread.php?t=43756&highlight=supplier+evaluation

                I hope you find Elsmar Cove to be a useful website, and my next blog will share a few ideas for supplier evaluation of my own.