Last November a new GHTF document was released on the topic of grading non-conformities: GHTF/SG3/N19:2012. This document is available on the new IMDRF website in the documents section. The 16-page document presents a new method for Certification Bodies to grade non-conformities and to communicate these findings to regulators such as the US FDA and Health Canada (e.g. – GD211 voluntary reports).
N19 recommends the same three-part structure for writing nonconformities that is taught in Lead Auditor Classes, and there is even a table of examples provided with poorly written findings and well-written findings with more specific references to objective evidence.
Section 4.2 of the guidance document, however, introduces a new concept for grading of findings. The traditional grading of findings is: Major, Minor, and Observations. Opportunities for Improvement (OFI) are no longer allowed in regulatory reports to avoid the appearance of providing consulting advice to clients. For internal audits and supplier audits, OFIs are still used by most auditors.
The new grading process defined by the guidance document has a two-step process. The first step uses a grading matrix to quantitatively determine a grade for the finding based upon the impact upon the QMS and the frequency of occurrence.
The second step of the grading process is to review escalation rules that are defined in Section 4.2.2 of the guidance document. This section emphasizes the importance of using the word “absence” in the wording of findings if a required procedure is not present in the QMS. This type of finding should only happen during initiate certification audits where 100% of the required procedures are typically verified during the Stage 1 audit. If this occurs, then the grading is increased by 1 to a possible maximum of 5.
Another possible escalation event is the release of nonconforming devices outside the control of the manufacturer. If this occurs, then the grading is increased by 1 to a possible maximum of 5. If the required procedure is absent, and product is released that is nonconforming, the guidance states that the score should not be escalated above a 5.
In all of the Lead auditing courses I have taught, both of the above escalation events would be examples of a “Major Nonconformity.” Repeat occurrences of nonconformities would typically be escalated from a minor NC to a major NC, but in this new method the scores could be a “2” or a “4”—depending upon the impact upon the QMS.
I have had enough trouble in the past with training auditors to consistently grade findings during audits, and this is one of the most important sections of the exam for a Lead Auditing Course. Recently I suggested that a client consider using the risk analysis matrix that they were already using for process risk analysis and apply the matrix to grading of findings. An example of this type of matrix is shown below.
My client used semi-quantitative scores for severity (1-3) and occurrence (1-4). The two factors were multiplied to calculate a risk priority number (RPN) ranging from 1-12. The resulting matrix is also color coded to indicate the urgency of corrective action plans to be developed for the finding.
Has anyone implemented a grading system based upon this new guidance? If you have, please share your experiences here or on one of the LinkedIn Groups I have posted this question:
As an auditor, one of the most important (and difficult) things for you to learn is how to issue a non-conformity—especially a major. This is normally done at the closing meeting of an audit, but the closing meeting is not where the process of issuing the non-conformity begins. Issuing a non-conformity actually starts in the opening meeting.
ISO 19011:2011 is the official guidance document for auditors of Quality Management Systems. Section 6.4.2 of this Standard explains the best practices for an opening meeting. The last five items in this section are critical to preparing the client for potential non-conformities:
the method of reporting audit findings including grading, if any;
the conditions under which the audit may be terminated;
time and place of the closing meeting;
how to deal with possible findings during the audit;
the system for feedback from the auditee on the findings or conclusions of the audit,
the process for complaints and appeals.
Methods of Reporting and Grading
The auditor should be crystal clear in their description of minor and major nonconformities or any other grading that will be used. The auditor should also make it clear that they are looking for conformity rather than non-conformity. This is an audit—not an inspection. Typically, a minor nonconformity is described as “a single lapse in the fulfillment of a requirement” while a major nonconformity is described as one of the following: 1) “a total absence in the fulfillment of a requirement,” 2) “repetition of a previous nonconformity,” 3) “failure to address a previously identified minor non-conformity”, or 4) “shipment of non-conforming product.” When the auditor is in doubt, then the finding is minor and never a major. For a major non-conformity to be issued there can be no doubt.
Conditions for Termination
The option to terminate an audit is typically reserved for a certification audit where a major non-conformity is identified and there is no point in continuing. Termination is highly discouraged, because it is better to know about all the minor and major non-conformities now instead of waiting until the certification audit is rescheduled. The certification body will charge you for their time anyway.
Another reason for termination is when an auditor is being unreasonable or inappropriate. This is rare, but it happens. If the audit is terminated you should always being communicating this to upper management at the certification body and the company—regardless of which side of the table you sit. For FDA inspections this is not an option. For audits performed by Notified Bodies, there is the possibility of suspension of a certificate in response to audit termination. Therefore, I always recommend appealing after the fact instead of termination. Appealing also works for FDA inspections.
Closing Meeting
The closing meeting should be conducted as scheduled and the time/location should be clearly communicated to upper management in the audit agenda and during the opening meeting. Top management won’t be happy about non-conformities, but failure to communicate when the closing meeting will be conducted will irritate them further.
How to Deal with Findings
All guides and auditees should be made aware of possible findings at the time an issue is discovered. This is important so that an auditee has the opportunity to clarify the evidence being presented. Often non-conformities are the result of miscommunication between the auditor and auditee. This happens frequently when the auditor has a poor understanding of the process being audited. It is a tremendous waste of time for both sides when this occurs. If there is an actual non-conformity, it is also important to gather as much objective evidence as possible for the auditor to write a thorough finding and for the auditee to prepare an appropriate corrective action plan in response to the finding.
Feedback from the Auditee
I always encourage auditees to provide honest feedback to me directly and to management so that I could continue to improve. If you are giving feedback about an internal auditor or a supplier auditor, you should always give feedback directly before going to the person’s superior. You are both likely to work together in the future, and you should give the person every opportunity to hear the feedback first-hand.
When providing feedback from a 3rd party Certification Audit, you should know that there will be no negative repercussions against your company if you complain directly to the Certification Body. At most, the Certification Body will assign a new auditor for future audits and investigate the need for taking action with the auditor. In all likelihood, any action taken will be “retraining.” I never fired somebody for a single incident—unless they broke the law or did something that was unsafe. The key to providing feedback, however, is to be objective. Give specific examples in your complaint, and avoid personal feelings and opinions.
Complaints and Appeals
As the auditee, you should ask for the contact information at the certification body during the opening meeting. Ask with a smile—just-in-case you disagree and so you can provide feedback (which might be positive). As the auditor, you should always make the contact information for the certification body available. If you are conducting a supplier audit or an internal audit, you probably know the auditor’s boss and there is probably no formal complaint or appeals process. In the case of a supplier audit, the customer is always right—even when they are wrong.
During the Audit
During the audit you should always make the guide(s) and process owner(s) aware of any potential non-conformities as you find them. This is their opportunity to clarify the objective evidence for you and to explain why there is not a non-conformity. Often I will refer to the Standard that I am auditing to at this point. I will identify the specific requirement(s) and show the process owner. I will say, “This is what I am trying to verify. Do you have anything that would help address this requirement?” If the process owner is not sure of how to meet the requirement, often I will provide an example of how this requirement is addressed in other areas or at other companies.
If the audit is a multi-day audit, I will review the potential nonconformities at the end of the day and give the auditee the opportunity to provide additional objective evidence in the morning. If it is already the last day of the audit or it is a single-day audit, I will give auditees until the closing meeting to provide the objective evidence. Often I will use this opportunity to explain what would be considered a minor non-conformity and what would be a major non-conformity. Usually I can say, “This is definitely not a major non-conformity, because…”
Closing Meeting
At the closing meeting, the auditee should never be surprised. If an issue remains unfulfilled at the closing meeting, the auditee should be expecting a minor non-conformity—unless the issue clearly warrants a major non-conformity. Since a minor nonconformity is described as “a single lapse in the fulfillment of a requirement,” it is difficult for an auditee to argue that an issue does not warrant a minor non-conformity. Typically, the argument is that you are not consistent with other auditors. The most common response to that issue is, “Audits are just a sample, and previous auditors may not have seen the same objective evidence.” The more likely scenario, however, is that the previous auditor interprets the requirements instead of reviewing the requirements with the client and making sure both parties agree before a finding is issued.
A major nonconformity is usually defined as one of the following: 1) “a total absence in the fulfillment of a requirement,” 2) “repetition of a previous nonconformity,” 3) “failure to address a previously identified minor non-conformity”, or 4) “shipment of non-conforming product.” When the auditor is in doubt, then the finding is minor and never a major. For a major non-conformity to be issued there can be no doubt. If a finding is major, the auditee should have very few questions. Also, I find that often the reason for a major non-conformity is a lack of management commitment to address the root cause of a problem. Issuing a major non-conformity is sometimes necessary to get management attention.
Regardless of the grading, all audit findings will require a corrective action plan—even an FDA warning letter requires a CAPA plan. Therefore, a major non-conformity is not a disaster. You just need to create a more urgent plan for action.
My first certification audit ever didn’t go so well. The reason it didn’t go well is that the auditor wrote nonconformities that my boss and our regulatory consultant didn’t agree with. At the time, I was too inexperienced to know how to handle it. My boss and the consultant, however, totally lost it. I’ve never seen veins that big in someone’s forehead–even in cartoons.
I asked them both to leave the room, because I was afraid to “push back” on the auditor. Many Management Representatives feel the same way that I did during that initial certification audit. The best way to summarize our concerns is with the following picture:
Don’t poke the bear!
This week’s entertainment is for a friend of mine. Thank you for all your help. You’ve made my year.
Recently another LinkedIn group member emailed me to say that they have seen several auditors for registrars identifying nonconformities that represented their own personal opinions rather than specific requirements of the Standard. For example: there is a requirement to assign management responsibilities and document it, but there is no requirement to have an organization chart.
Another common mistake is when auditors insist that a company must create a turtle diagram for every single process. I support the use of turtle diagrams 100%, but the only requirement in the Standard is to use the process approach–not turtle diagrams specifically.
My favorite is my own personal mistake. I wrote a nonconformity for not having a process for implant registration cards for a company that was planning to ship a high-risk implant product to Canada. There is a requirement for implant registry cards, but I forgot that Canada defines “implants” in this case as only a very short list of implant devices–not implants in general.
Auditors are human. These are audit findings–not a jail sentence. Everyone needs to remember that the worst that can happen is that you receive a nonconformity. If the auditor finds a nonconformity, then you need to develop a CAPA plan. If the auditor finds nothing, you still need to do your own internal audits to identify nonconformities and to continuously improve processes.
The question is, what should you do when an auditor is wrong?
I recommend that you “push back”, but you need to know how. Many consultants suggest saying, “Can you show me in the Standard where it says I have to do that?” That’s just like poking a bear. If you do it once, it’s annoying. If you do it multiple times, an auditor might just eat you.
One Management Representative did that to me after I had taken the time to review the requirements with him. I responded by holding the ISO 13485 Standard in front of him and reciting clause 7.3.2. He responded by saying, “Well that’s up for interpretation.” I offered to recite the ISO 14969 guidance document for him, but his boss told him to shut up.
This certainly wasn’t the only time a client pushed back during a registration audit, but other clients have had the sense to argue about things they actually understood.
One of the clients I audited, said that he would change the topic to the auditor’s favorite sports team. That’s one approach. I’m sure that more than one client has taken the approach of asking me to explain where they can learn about best practices. I’m sure that they were somewhat successful. Another approach is to slide the lunch menu in front of them; I have only met one auditor that would not be distracted by a lunch menu.
Here’s my step-by-step approach to pushing back when you disagree with an auditor:
1. shut-up and look it up (before you open your mouth, grab the applicable external standard and look up exactly what you are looking for)
2. If you are still convinced that your auditor is wrong, then tell that you are having trouble finding the requirement. Show them where you are looking, and then ask them to help you find the requirement.
3. If the auditor can’t show you where you are wrong, or it appears that the auditor is interpreting the Standard as they see fit, then focus on asking the auditor for guidance on what they will be looking for in your CAPA plan.
4. If the CAPA plan the auditor is looking for is something you think is a good idea, then shut up and implement the improvements. If the CAPA plan is not acceptable to you, then you should ask what the process is for resolution of disputes.
5. No matter what, don’t start an argument with the registrar. They actually enjoy it. They like a challenge, and they resent people with less experience critizing them.
6. If you still disagree with your auditor, then you should ask if the auditor can explain the process for appealing findings and follow that process.
Checklists are great for making sure that all aspects of the regulations are covered, but is there a way to get more out of your audits? Imagine how nice it would feel to eliminate that “Control of Records” audit, and several other audits from your schedule that consist primarily of reviewing mountains of paperwork. Register for the audio seminar hosted by FX Conferences on February 19th and you will understand why separate audits of support processes are largely unnecessary. You will be able to complete turtle diagrams for any process in minutes, and you will learn how to strategically select auditors according to the process flow.
Audio seminar with FX Conferences on February 19th:
This seminar will demonstrate how to use turtle diagrams and the process approach to perform audits. The presentation materials include a blank turtle diagram template and a sample, completed turtle diagram for an incoming inspection process. You will also learn how to assign auditors differently in order get more value from your audits.
This audio conference covers:
What is the process approach to auditing?
Why is the process approach more efficient than audit checklists?
Passing a webinar on auditing does not make you competent.
Does your company ask incoming inspectors to update CAD drawings when there is a design change? Of course not. Your company has engineers that are trained to use SolidWorks, and it takes a new engineer a while to become proficient with the software. Auditing is a skill that you learn—just like SolidWorks.
My favorite holiday movie…I’ll be watching this later tonight!
I’ve never met a manager that wondered where the value was in having an engineer update a drawing, but many managers view internal and supplier audits as a necessary evil. Instead of asking the expert how few audit days you can get away with, ask the expert: “What is the purpose of auditing?”
The purpose of internal auditing is to confirm that the management system is effective and to identify opportunities for improvement. The purpose of supplier auditing is to confirm that a supplier is capable of meeting your needs and to identify opportunities for improvement. Therefore, if an auditor has no nonconformities and no opportunities for improvement were identified—what a waste of time!
To receive value from auditing, you need auditors that are competent. In clause 6.2.1 of the ISO 13485 Standard it says, “Personnel performing work affecting product quality shall be competent on the basis of appropriate education, training, skills and experience.” As the audit program manager, make sure you recruit people that demonstrate auditing competency.
Education
First, educational background is important for auditors. You cannot expect someone who has never taken a microbiology course in their life to be an effective auditor of sterilization validation. Likewise, someone that has never taken a course in electricity and magnetism will not be effective as an auditor for active implantable devices. Therefore, determine what types of processes the auditor will be auditing. Then make sure that the person you hire to be an auditor has the necessary education to understand the processes they will be auditing.
Training
Second, an auditor needs to be trained before they can audit. The auditor needs training in three different aspects: 1) the process they will be auditing, 2) the standard that is the basis for assessing conformity, and 3) auditing techniques. If you are going to be auditing printed circuit board (PCB) manufacturers with surface-mount technology (SMT), then you need to learn about the types of components used to make PCBs and how these components are soldered to a raw board. I know first-hand that anyone can learn how SMT works, but it took me a few months of studying.
If your company is only selling medical devices in the USA, then you will need to learn 21 CFR 820 (i.e. – the QSR). However, if your company also sells devices in Europe in Canada you will need to learn ISO 13485, the MDD (93/42/EEC as modified by 2007/47/EC), and the Canadian Medical Device Regulations (CMDR). I learned about ISO 13485 in a four-and-half day lead auditor course in Florida, I learned about the MDD in a three-day CE Marking Course in Virginia, and I learned about the CMDR in a two-day course taught by Health Canada in Ontario. A 50-minute webinar on each regulation is not sufficient for auditing.
Finally, you need training on the techniques of auditing. A two day course is typically needed. I took a 50-minute webinar and passed a quiz before conducting my first internal audit, but I was not competent.
Skills
Third, an auditor needs specific skills to be effective as an auditor. The most critical skills are: 1) communications skills, 2) organizational skills, and 3) analytic skills. Communications skills must include the ability to read and write exceptionally well and the auditor needs to be able to verbally communicate with auditees during meetings and interviews. The most difficult challenge for auditors is covering all the items in their agenda in the time available. The auditor rarely has more time than the need to audit any topic, and audit team leaders must be able to manage their own time as well as simultaneously managing the time of several other auditors.
Experience
Last, but certainly not the least important aspect of auditor competency is experience. This is why 3rd party auditors are required to act as team members under the guidance of a more experienced auditor before they are allowed to perform audits on their own. This is required regardless of how many internal or supplier audits the person may have conducted in the past. More experienced auditors are also required to observe new auditors and recommend modifications in their technique. Once a new auditor has completed a sufficient number of audits as a team member, the auditor is then allowed to practice leading audits while being observed. After six to nine months, a new auditor is finally ready to be a lead auditor on their own. An internal auditor does not need the same degree of experience as a 3rd party auditor, but being shadowed 2-3 times is not sufficient experience for an auditor (1st or 2nd party). For more information about this topic, please read my blog posting on auditor shadowing.
If you are an audit program manager, and you would like to improve your own competency, please contact me to learn about a new advanced course specifically for audit program managers. I am teaching a course with Brigid Glass. The course is designed specifically for audit program managers—not for inexperienced auditors. It will be a two-day course, and we are offering the course in three different cities: San Diego, CA (April 11/12), Orlando, FL (April 15/16) and Las Vegas, NV (April 17/18). Please Contact Me if you would like to learn more about the course.
I am also teaching a one-hour, audio seminar with FX Conferences on January 9th:
This seminar will cover the areas of supplier qualification, supplier evaluation and supplier auditing. We already have a large number of companies signed-up for the seminar, and I am looking forward to having you join us.
This blog started as a single posting, but I realized that the blog was much too long. Therefore, I split the blog into three separate postings. This is the final “Part 3 of 3”. I hope you have enjoyed it. If you have suggestions for my next posting, please let me know.
If you are an audit program manager that is training a new auditor from another department, treat them like a new hire!
Once you have identified someone that you want to “hire” as an internal auditor, your next step should be to develop an “Onboarding” plan for them with their boss. If you are hiring someone that will be a dedicated auditor, please ignore my quotation marks above. In most companies, however, the internal auditors are volunteers that report to another hiring manager. Therefore, as the audit program manager you need to get a firm commitment from the auditor’s boss with regard to the time required to train the new auditor and to actually perform audits on an on-going basis.
The Trans Siberian Orchestra is a must see–especially if you can take you family to see the performance live.
Winning Over the Boss
In my previous posting I said that, “The biggest reason why you want to be an auditor is that it will make you more valuable to the company.” The auditor’s boss may or may not agree with this statement, but the boss knows that the salary is coming out of their budget either way.
Therefore, talk with the auditor’s boss and find out what the auditor’s strengths and weaknesses are. Find out which skills the boss would like to see the auditor develop. By doing this, the two of you can develop a plan for making the auditor more valuable to their boss AND the company.
Making Re-Introductions
Ideally, auditors are extraverted and they have been with the company long enough to know the processes and process owners that they will be assigned to audit—especially if they will be auditing upstream and downstream from their own process area. In the past the auditor was a customer or a supplier, but now the relationship with a process owner will change. Auditors are required to interview process owners and this involves asking tough questions that might not be appropriate in the auditor’s normal job duties. Therefore, as the audit program manager, you should re-introduce the auditor to the process owner in their new capacity as auditor.
During this re-introduction, it is important to make three points:
the auditor is going to be trained first,
you will be shadowing the auditor during the audit, and
the auditor’s job is to help the process owner identify opportunities for improvement.
By making the first point, you are reminding the process owner of the scheduled audit—well in advance. You are also informing the process owner that this auditor will have new skills, and the process owner should have some tolerance for mistakes that new employees make. You might also mention that you would like to get the process owner’s feedback after the audit so the auditor knows what areas they need to improve to become better auditors.
The second point should put the process owner at ease—assuming the process owner has a good relationship with you as the audit program manager. It is important to be descriptive when “shadowing” is mentioned. Both the process owner and the auditor may not understand the process or the purpose of shadowing. The following blog posting might help with this: “How do you shadow an auditor? Did you learn anything?”
The third point is the most critical step in onboarding a new auditor. For an auditor to be successful, they must ADD VALUE!
As an auditor, you cannot pretend to add value.
The process owner should know their process and they probably know which areas are weakest. The audit program manager should encourage the process owner to list some specific areas in which they are having problems. Ideally, the process owner would be informed of this need prior to the re-introduction. Then the process owner can be better prepared for the meeting, and hopefully they will have a few target areas already identified. Targets with associated metrics are the best choice for a new auditor, because these targets reinforce the process approach to auditing.
Next Steps
Once your new auditor has been re-introduced to the process owners they will be auditing, you need to begin the training process. As with any new employee, it is important to document the training requirements and to assess the auditor’s qualifications against the requirements of an auditor. Every new auditor will need some training, but the training should be tailored specifically to the needs of the auditor.
The training plan for a new auditor should include the following:
a reading list of company procedures specific to auditing and external standards that are relevant;
scheduled dates for the auditor to shadow another experienced auditor;
scheduled dates for an experienced auditor to shadow the auditor during the first two process audits (upstream and downstream);
goals and objectives for the internal audit program; and
any training goals that the auditor’s boss has identified for the auditor.
If you are an audit program manager, and you would like to improve your own competency, please contact me to learn about a new advanced course specifically for audit program managers. I am teaching a course with Brigid Glass. The course is designed specifically for audit program managers—not for inexperienced auditors. It will be a two-day course, and we are offering the course in three different cities: San Diego, CA (April 11/12), Orlando, FL (April 15/16) and Las Vegas, NV (April 17/18). Please Contact Me if you would like to learn more about the course.
I am also teaching a one-hour, audio seminar with FX Conferences on January 9th:
This seminar will cover the areas of supplier qualification, supplier evaluation and supplier auditing. We already have a large number of companies signed-up for the seminar, and I am looking forward to having you join us.
This blog started as a single posting, but I realized that the blog was much too long. Therefore, I split the blog into three separate postings. This post is “Part 2 of 3”. The final part in the series will be posted tomorrow–December 24, 2012.
Stop begging people to help you audit. Learn how to recruit auditors more effectively.
Nearly 100% of the people I train as auditors were not hired specifically to be auditors. Instead, auditing is something extra that they were asked to do in addition to their regular job. This situation creates three problems for the audit program manager:
you have difficulty getting enough people to perform the audits;
most auditors will come from your department, so who is going to audit you; and
the auditors have little or no motivation to get better at auditing.
Stop begging for “volunteers” from other departments and start recruiting.
My favorite holiday song of all time! I sing this to myself in car rides during July.
When I am recruiting someone to audit, I always get asked two questions:
Who/What will I be auditing?
What will I have to do?
You need to motivate people to become auditors, because it requires extra work. The answer to #2 should be specific. I recommend creating a “sell sheet” that explains the process of performing an audit. I also like to create sell sheets that are educational. Therefore, I recommend adapting the flow chart in ISO 19011:2011 (Figure 2 on page 15). I would add time estimates for each step of the process (6.2 – 6.7). This will serve as a training tool for future auditors, and it will eliminate the fear of unknown time commitment for your potential recruit.
In order to answer #1, I recommend you assign the recruit processes that are upstream and downstream. I have recommended this concept in previous postings, but essentially you are assigning the person to audits of internal suppliers and internal customers. By doing this, utilizing the process approach will be more natural to the auditor and they will have a vested interest in doing a thorough audit. This also creates a situation where the auditor is typically assigned to at least two process audits per year.
The next question is one that your potential recruit will never ask, but they are always thinking it…
Why should I become an auditor?
The biggest reason why you want to be an auditor is that it will make you more valuable to the company.
Auditors are required to interview department managers and ask tough questions. This gives the auditor a better understanding of the organization as a whole, and it gives them insight into how other managers work. This insight is pure gold.
If you want to be effective and get promoted, you need to demonstrate value to your boss and top management. If you don’t understand what other departments need, how can you help them? No manager will promote a selfish, power-hungry hog. They promote team players that make others better. Auditing gives you the insight necessary to understand how you can do that.
Auditing other departments will also give you insider information as to where new job openings will be. Sometimes you can’t wait for your boss to get promoted. In that case, you might want to know more about other departments in your company.
Each corporate culture is different, but the audit program manager needs to “sell” the recruit on volunteering to be an auditor.
Where to find recruits
Due to the cross-functional nature of auditing, I have found that my own personal experience working in multiple departments was invaluable. I have a better understanding of how a department functions than other auditors, because I have worked in that department at another company. Operations, engineering and research experience are extremely valuable for auditing, but I think the experience that transfers the best to auditing is service.
If your company is large enough to hire full-time auditors, I recommend searching for potential auditors at your suppliers and their competitors. These people will bring unique knowledge that is critical to a successful supplier selection process, and these individuals will increase the diversity in your company—instead of duplicating knowledge and expertise.
This blog started as a single posting intended to help a Compliance Manager in the Twin Cities. Unfortunately, I ran out of time to finish the blog and it has been a couple of weeks since my last post. When I restarted the blog this weekend, I realized that the blog was much too long. Therefore, this is part 1 of 3. Part 2 will be about hiring auditors, and part 3 will be about training auditors.
For those of you that want to learn more, I am teaching a course with Brigid Glass in April. The course is designed specifically for audit program managers—not for inexperienced auditors. It will be a two-day course, and we are offering the course in three different cities: San Diego, CA (April 11/12), Orlando, FL (April 15/16) and Las Vegas, NV (April 17/18). Please Contact Me if you would like to learn more about the course.
I am also teaching a one-hour, audio seminar with FX Conferences on January 9th:
This seminar will cover the areas of supplier qualification, supplier evaluation and supplier auditing. We already have a large number of companies signed-up for the seminar, and I am looking forward to having you join us.
There are 34 days until the end of 2012. You have four supplier audits and three internal audits to complete. Of course all but two of these audits are overdue. What should you do?
Well if I were Minerva McGonagall, I would let you borrow my time turner and you could just turn back time and perform all seven audits at the same time.
If your calendar is looks more like this one, then you only have 24 days before “the end” and completing those last seven audits might not be so important.
Other options that might be readily available to you include:
1. get some help
2. perform remote audits
3. reschedule some of the audits for next year
There are some great cartoons and jokes about doing more with less, but if you intend to complete seven audits before the end of the year you might need some help. There really isn’t any time left to train someone so that they are capable of conducting an effective audit by themselves. In fact, I expect training a new auditor to take at least 6 months before I think they are ready to work solo. Even if you are less demanding than I am, you still would need time for classroom training and shadowing a couple of audits. Therefore, the best I think you could hope for is one or two solo audits of the seven you need to complete.
Realistically, your only source of help would be auditors that are already trained and consultants. The last month of the year is historically very busy for everyone–especially Quality Assurance Auditors. Therefore, consultants will not be cheap and you should commit to any qualified consultants that are available without too much delay (then again, maybe they are available because they are not very good). If you have any in-house auditors that are already trained, do everything you can to get some of their time in the next few weeks.
Option 2 is to perform remote audits. This is a viable option for you to justify for a supplier with a great quality track record or suppliers in other countries. However, a remote audit is not the same as asking a supplier to complete a survey. ISO 19011:2011 provides some guidance specific to remote auditing in table B.1 of Annex B.
For a remote audit, you should still sample just as many records—if not more. You should conduct interviews by phone, Skype or some similar technology. You should analyze any available data to help identify which processes appear to be effective and which processes need to improve. For anyone that is doing this for the first time, I recommend using an audit checklist that is design for an on-site audit and attempting to complete all the questions remotely. Just remember, “Always request data.”
Option 3 is to reschedule some audits for January 2013. I have suggested this so many times to clients, but very few follow this advice. If your company is late in conducting some audits, the important thing to do is to document this, reschedule the audits, and take corrective action(s) to prevent it from recurrence. If you wait until January, you will have additional time to train an auditor as well. Finally, consultants historically have more time available in January than December.
In parallel with your efforts to catch-up on your schedule, I also recommend the following:
Create a quality objective that measures “on-time delivery” of audits and audit reports. This is an effective metric for managing an audit program.
Investigate the reasons for audits being overdue. If the occurrence was preventable, then I recommend initiating a CAPA. This will have two affects. First, your 3rd party auditors will see that you have identified the problem yourself and taken appropriate corrective action(s). If you also discuss this during a Management Review, this information can be used effectively to change the grading of an audit finding to a “minor” or to potentially eliminate the finding altogether. Second, it will ensure that this doesn’t happen again.
If you would like to learn more about scheduling of audits and managing audit programs, please subscribe to this blog and connect with me on LinkedIn. There will be several more blogs on these topics in preparation for a FREE webinar on February 14th. You can also connect with Brigid Glass on LinkedIn. She will be my co-instructor for three courses in April 2013.
If you are shadowing, you are taking notes so you can discuss your observations with the person you are shadowing later.
Somewhere in your procedure for “Quality Audits”, I’ll bet there is a section on auditor competency. Most companies require that the auditor has completed either a course for internal auditor or a lead auditor course. If the course had an exam, then you might even have evidence for training effectiveness. Demonstrating competency is much harder. One way is to review internal audit reports, but writing reports is just part of what an auditor does. How can you evaluate an auditor’s ability to interview people, take notes, follow audit trails, and manage their time? The most common solution is to require that the auditor “shadow” a more experienced auditor several times, and then the trainee will be “shadowed” by the trainer.
I can’t remember posting any music from John Mayer and the song title fits our subject for this blog.
Shadowing 1st Party Audits:
ISO 19011:2011 defines 1st party audits as internal audits. When 1st party auditors are being shadowed by trainer, or vice versa, there are many opportunities for training. The key successful training of auditors is to recognize teachable moments.
When the trainer is auditing, the trainer should look for opportunities to ask the trainee, “What should I do now?” or “What information do I need to record?” In these situations, the trainer is asking the trainee what they should do BEFORE they do it. If the trainee is not sure, the trainer should explain what, why and how at that moment with real examples.
When the trainer is shadowing, the trainer should watch and wait for a missed opportunity to gather important information. In these situations, the trainer must resist guiding the trainee until after the trainee appears to be done. When it happens sometimes the best tool is simply asking, “Are you sure you got all the information you came for?”
Here are five (5) mistakes that I have observed trainers make when they were shadowing:
1. Splitting up, instead of staying together, is one of the more common mistakes I have observed. This happens when people are more interested in completing an audit than taking every advantage of training opportunities. The trainee may be capable of auditing on their own, but this is no excuse for tag teaming the auditee. This is unfair to the trainee AND the auditee. If an audit is running behind schedule, this is the perfect time to teach a trainee how to recover some time in their schedule. Time management is after all one of the hardest skills for auditors to master.
2. Staying in the conference room, instead of going to where the work is done, is a common criticism of auditors. If the information you need to audit can be found in a conference room, then you could have completed the audit remotely. This type of audit teaches new auditors very little other than how to take notes. These are basic skills that auditors should master in a classroom prior to shadowing.
3. Choosing an administrative process is a mistake, because administrative processes limit the number of aspects of the process approach that can be practiced by an auditor-in-training. Administrative processes rarely have equipment that requires validation or calibration, and both the process inputs and outputs consist only of paperwork, forms or computer records. With raw materials and finished goods to process, the job of the auditor is more challenging because there is more to be aware of.
4. Not providing honest feedback is a huge mistake. Auditors need to be thick skinned or they don’t belong in a role where they are going to criticize others. Before you begin telling other people how to improve, you first need to self-reflect and identify your own strengths and weaknesses. Understanding your own perspective, strengths, weaknesses, and prejudices is critical to being an effective assessor. As a trainer, it is your job to help new auditors to self-reflect and accurately rate their performance against objective standards.
5. “Silent Shadowing” has no value at all. By this I mean shadowing another auditor without asking questions. If you are a trainee you should be mentally pretending you are doing the audit. Whenever the trainer does something different from the way you would do things, you should make a note so you can ask, “Why did you do that?” If you are trainer you should also be mentally pretending you are doing the audit. It is not enough to be present. You job is to identify opportunities for the trainee to improve. The better the trainee, the tougher your job becomes. This is why I training other auditors has helped me improve my own auditing skills.
Shadowing 2nd Party Audits:
If you are developing a new supplier quality engineer that is responsible for performing supplier audits, it is recommended to observe the auditor during some actual supplier audits. Supplier audits are defined as 2nd party audits in the ISO 19011:2011 Standard. The purpose of these audits is not to verify conformity to all the aspects of ISO 13485. Instead, the primary purpose of these audits is to verify that the supplier has adequate controls in place to consistently manufacture conforming product for your company. Therefore, processes such as Management Review (Clause 5.6) and Internal Auditing (Clause 8.2.2) are not typically sampled during a 2nd party audit.
The two most valuable process for a 2nd party auditor to sample are: 1) incoming inspection, and 2) production controls. Using the process approach to auditing, the 2nd party auditor will have an opportunity to verify that the supplier has adequate controls for documents and records for both of these process. Training records for personnel performing these activities can be sampled. The adequacy of raw material storage can be evaluated by following the flow of accepted raw materials leaving the incoming inspection area. Calibration records can be sampled by gathering equipment numbers from calibrated equipment in use by both processes. Even process validation procedures can be assessed by comparing the actual process parameters being used in manufacturing with the documented process parameters in the most recent validation or re-validation reports.
My recommendation is to have the trainee shadow the trainer during the process audit of the incoming inspection process and for the trainer to shadow the trainee during the process audit of production processes. In between the two process audits, the trainee should be asking questions to help them fully understand the process approach to auditing. Supplier auditors should also be coached on techniques for overcoming resistance to observe processes that may involve trade secrets or where competitor products may also be present. During the audit of production processes, the trainer may periodically prompt the trainee to gather information that will be needed for following audit trails to calibration records, document control or for comparing with the validated process parameters. The “teachable moment” is immediately after the trainee missed an opportunity, but while the trainee is still close enough to go back and capture the missing details.
Shadowing 3rd Party Audits:
Use your FDA inspections and ISO certification audits as an opportunity to shadow experienced auditors and to learn what they are looking for.
If you are going to shadow a 3rd party auditor, I recommend two specific people to “shadow” the auditor. First, the process owner should be the guide for whichever process is being audited. This is the person that will be responsible for addressing any nonconformities found in the area, and they should be present during interviews–although they should be coached on when to comment and when to remain quiet and simply observe. Second, the person that performed an internal audit of the process being audited should be present if at all possible. This person will benefit from seeing how a professional 3rd party auditor performs a process audit, because they will know which things to look for in the future so that auditees in that area are prepared for the next external audit.
If you are an audit program manager, and you would like to learn “What Makes World Class Audit Programs Different?”, please contact me. I am co-teaching an advanced course for audit program managers in April 2013.
For other sources of information related to auditor shadowing, please check out the following links:
If you were just notified of an FDA inspection and you don’t think you are ready, using tricks to hide your problems is a huge mistake. I have heard a few recommendations over the years for “secrets” to hide those problems. In this post, I share my favorite “secrets”–and why they DON’T work.
Here are my top 10 ways to make an FDA inspection worse:
10. Stalling when the inspector makes a request– This just irritates inspectors. At best the inspector will use the waiting time to identify additional documents to sample or to review the information you have provided more closely. At worst the inspector will accuse the company of not cooperating with the inspection and the inspector may return the following week with several more team members to help them. Whenever this occurred during a 3rd party audit that I conducted, I would move on to another area and interview someone. However, before I left the person that was slow to respond, I provided the person with a list of documents and records that I expected to be waiting for me upon my return. In extreme cases, I had to bluntly tell the management representative that I needed documentation more quickly. As an instructor, I teach auditors techniques for coping with this tactic.
9. Suggesting records for the inspector to sample – This is specifically forbidden in the case of 3rd party inspections and audits. The FDA has work instructions for identifying sample sizes and samples are supposed be selected randomly. In reality, samples are rarely random and usually the inspector is following a trail for a specific lot, part number, etc. When clients offered me samples, I tried to be polite and review the record they provided. However, I also would request several other records–or follow a trail as I have indicated above. Another approach I often use is to focus on high-risk items (i.e. – a risk-based approach to sampling). In general, you can expect the FDA inspectors to sample more items than a registrar–and sample sizes are often statistically derived if the number of records is sufficiently large. When sample sizes are quite small, I recommend sampling 100% of the records since the previous inspection/audit. This is not always possible for 3rd party auditors, but internal auditors often can achieve this.
8. Outsourcing processes to subcontractors – The FDA recently re-instated the requirement for contract manufacturers and contract sterilizers to be registered with the FDA by October 1, 2012. Therefore, hiding manufacturing problems from the FDA by outsourcing manufacturing is increasingly more difficult to do. In addition, the FDA focuses heavily on supplier controls and validation of outsourced processes. Therefore, an inspector will identify high-risk processes performed by subcontractors and request documentation of process validation by that supplier. If the company does not have the validation reports–this could quickly escalate to a 483–and possibly a visit to the subcontractor.
7. Trying to correct problems during the inspection– This is what I like to call the document creation department. At one company I worked for, we noticed a mistake across several of the procedures and made a change overnight between the 1st and 2nd day of the audit. When the auditor asked for the procedures in the morning, he asked “Is the ink dry yet?” The auditor then proceeded to request records that demonstrated compliance with the newly minted procedures. As you might have guessed, this resulted in several nonconformities. When clients attempt to correct problems found by an inspector, the inspector typically will respond with the following statement, “I applaud you for taking immediate action to contain and correct the problem. However, you still need to perform an investigation of root cause and develop a corrective action plan to prevent recurrence. To do this investigation properly may take several days.” I also teach auditors to memorize this phrase.
6. Writing a letter to file – When companies make minor design changes, one of the most common approaches is to “write a letter to file.” This phrase indicates that the design team is adding a memo to the Design History File (DHF) that justifies why design validation is not required or why regulatory notification/approval is not required. The FDA used to publish a decision tree to help companies make these decisions. In fact, such a decision tree is still part of the Canadian significant change document. The FDA recently withdrew a draft document that eliminated many perceived opportunities to utilize the “letter to file” approach. However, the FDA will still issue a 483 to a company if the inspector can identify a change that required validation that was not done or a 510(k) that was not submitted for a design change. In fact, the FDA specifically looks for these types of issues when an inspector is doing a “for cause” inspection after a recall or patient death.
5. Shut it down – Not running a production line that has problems is a favorite strategy for hiding problems. However, the FDA and auditors will simply be forced to spend more time sampling and reviewing records of the problematic production line. If you need to shut a line down, make sure everything is identified as nonconforming and segregate rejected product from good product carefully. You should also use these problem lines as an opportunity to show off your investigation skills and your ability to initiate CAPAs. If you simply forgot to validate some equipment or do some maintenance, take your lumps and keep production running. If you are a contract manufacturer, never shut it down without notifying the customer. If you do not tell your customer, you will get a complaint related to on-time delivery and a 483.
4. Storing all records off-site – I first heard about this tactic during an auditor course I was co-teaching. During the course we had many reasons why the company should be able to provide the records in a timely manner. However, I have experienced this first-hand as a 3rd party auditor. When this happens, I do three things: 1) I increase my sampling of records that are available, 2) I carefully review the supplier controls and supplier evaluation of the storage facility (assuming it is outsourced), and 3) I verify that the company has a systematic means for tracking the location (i.e. – pallet and box) for every record sent to storage. FDA inspectors will simply move along to another record and follow-up on their earlier request with a second visit or a request to send a copy of the document to them after the inspection.
3. Identifying information as confidential – A company can claim information is confidential and may not be shared with the public, but very little information is “Confidential” with regard to the FDA or Notified Bodies. Therefore, this strategy almost never works. In fact, this will enrage most FDA inspectors. In training courses, I train auditors to ask the auditee to redact confidential information. For example, a CAPA log may have confidential information in the descriptions but the trend data on opening and closing dates is never confidential.
2. The FDA is not allowed to look at those records – Although this statement is technically true for internal audit reports and management reviews, the FDA always says that they can get at this information through the CAPA system. What the FDA means is that there should always be evidence of CAPAs from internal audits and management reviews. If there is not, then this will quickly become a 483. Another person I met tells the story that when they agreed to share the management review records with the inspector, the inspector rarely issued a 483. When they refused to share the management review with the FDA, the inspection went quite badly from that point forth. I don’t agree with being vindictive, but it happens.
1. Show me where that is required – This is just silly. Inspectors and auditors are trained on the regulations, while you are trained on your procedures. Spend your time and effort figuring out how your procedures meet the regulations in some way. Challenging the inspector excites the inspector. We all like a challenge–and we rarely lose. One auditee tried this approach with me in front of their CEO. This experience gave me the opportunity to show off that I had memorized the clause in question–and the corresponding guidance document sections. I think the CEO realized quickly that the Management Representative was not terribly qualified.
My final advice is to do your best to help the inspector do their job, and treat every 483 as “just an opportunity to improve.” Just make sure you submit a response in 14 days or you will receive a warning letter too!
I just wanted to thank everyone for reading my blog. It's hard to make the dry and boring worth reading about. I have also created a formal Thank You page on my personal website: http://13485cert.com/qc-is-dead/.
Blog Stats
113,237 Views All Time!
Countdown to 2-Day Course!
Best Practices in Audit Program ManagementMarch 9, 2013
The new grading process defined by the guidance document has a two-step process. The first step uses a grading matrix to quantitatively determine a grade for the finding based upon the impact upon the QMS and the frequency of occurrence.
I have had enough trouble in the past with training auditors to consistently grade findings during audits, and this is one of the most important sections of the exam for a Lead Auditing Course. Recently I suggested that a client consider using the risk analysis matrix that they were already using for process risk analysis and apply the matrix to grading of findings. An example of this type of matrix is shown below.
Here are five (5) mistakes that I have observed trainers make when they were shadowing:
13485cert
QC is Dead 