A New Way to Grade Findings

Last November a new GHTF document was released on the topic of grading non-conformities: GHTF/SG3/N19:2012. This document is available on the new IMDRF website in the documents section. The 16-page document presents a new method for Certification Bodies to grade non-conformities and to communicate these findings to regulators such as the US FDA and Health Canada (e.g. – GD211 voluntary reports).

To download the guidance document, go to http://www.imdrf.org/.

N19 recommends the same three-part structure for writing nonconformities that is taught in Lead Auditor Classes, and there is even a table of examples provided with poorly written findings and well-written findings with more specific references to objective evidence.

Section 4.2 of the guidance document, however, introduces a new concept for grading of findings. The traditional grading of findings is: Major, Minor, and Observations. Opportunities for Improvement (OFI) are no longer allowed in regulatory reports to avoid the appearance of providing consulting advice to clients. For internal audits and supplier audits, OFIs are still used by most auditors.

The new grading process defined by the guidance document has a two-step process. The first step uses a grading matrix to quantitatively determine a grade for the finding based upon the impact upon the QMS and the frequency of occurrence.

The second step of the grading process is to review escalation rules that are defined in Section 4.2.2 of the guidance document. This section emphasizes the importance of using the word “absence” in the wording of findings if a required procedure is not present in the QMS. This type of finding should only happen during initiate certification audits where 100% of the required procedures are typically verified during the Stage 1 audit. If this occurs, then the grading is increased by 1 to a possible maximum of 5.

Another possible escalation event is the release of nonconforming devices outside the control of the manufacturer. If this occurs, then the grading is increased by 1 to a possible maximum of 5. If the required procedure is absent, and product is released that is nonconforming, the guidance states that the score should not be escalated above a 5.

In all of the Lead auditing courses I have taught, both of the above escalation events would be examples of a “Major Nonconformity.” Repeat occurrences of nonconformities would typically be escalated from a minor NC to a major NC, but in this new method the scores could be a “2” or a “4”—depending upon the impact upon the QMS.

I have had enough trouble in the past with training auditors to consistently grade findings during audits, and this is one of the most important sections of the exam for a Lead Auditing Course. Recently I suggested that a client consider using the risk analysis matrix that they were already using for process risk analysis and apply the matrix to grading of findings. An example of this type of matrix is shown below.

My client used semi-quantitative scores for severity (1-3) and occurrence (1-4). The two factors were multiplied to calculate a risk priority number (RPN) ranging from 1-12. The resulting matrix is also color coded to indicate the urgency of corrective action plans to be developed for the finding.

Has anyone implemented a grading system based upon this new guidance? If you have, please share your experiences here or on one of the LinkedIn Groups I have posted this question:

Medical Devices: QA/RA – http://bit.ly/SG3N19-QARA

ASQ – http://bit.ly/SG3N19-ASQ

Please share you own methods for grading findings?

How to Issue a Major Non-Conformity with a Smile

As an auditor, one of the most important (and difficult) things for you to learn is how to issue a non-conformity—especially a major. This is normally done at the closing meeting of an audit, but the closing meeting is not where the process of issuing the non-conformity begins. Issuing a non-conformity actually starts in the opening meeting.

ISO 19011:2011 is the official guidance document for auditors of Quality Management Systems. Section 6.4.2 of this Standard explains the best practices for an opening meeting. The last five items in this section are critical to preparing the client for potential non-conformities:

1. the method of reporting audit findings including grading, if any;
2. the conditions under which the audit may be terminated;
3. time and place of the closing meeting;
4. how to deal with possible findings during the audit;
5. the system for feedback from the auditee on the findings or conclusions of the audit,
6. the process for complaints and appeals.

Methods of Reporting and Grading

The auditor should be crystal clear in their description of minor and major nonconformities or any other grading that will be used. The auditor should also make it clear that they are looking for conformity rather than non-conformity. This is an audit—not an inspection. Typically, a minor nonconformity is described as “a single lapse in the fulfillment of a requirement” while a major nonconformity is described as one of the following: 1) “a total absence in the fulfillment of a requirement,” 2) “repetition of a previous nonconformity,” 3) “failure to address a previously identified minor non-conformity”, or 4) “shipment of non-conforming product.” When the auditor is in doubt, then the finding is minor and never a major. For a major non-conformity to be issued there can be no doubt.

Conditions for Termination

The option to terminate an audit is typically reserved for a certification audit where a major non-conformity is identified and there is no point in continuing. Termination is highly discouraged, because it is better to know about all the minor and major non-conformities now instead of waiting until the certification audit is rescheduled. The certification body will charge you for their time anyway.

Another reason for termination is when an auditor is being unreasonable or inappropriate. This is rare, but it happens. If the audit is terminated you should always being communicating this to upper management at the certification body and the company—regardless of which side of the table you sit. For FDA inspections this is not an option. For audits performed by Notified Bodies, there is the possibility of suspension of a certificate in response to audit termination. Therefore, I always recommend appealing after the fact instead of termination. Appealing also works for FDA inspections.

Closing Meeting

The closing meeting should be conducted as scheduled and the time/location should be clearly communicated to upper management in the audit agenda and during the opening meeting. Top management won’t be happy about non-conformities, but failure to communicate when the closing meeting will be conducted will irritate them further.

How to Deal with Findings

All guides and auditees should be made aware of possible findings at the time an issue is discovered. This is important so that an auditee has the opportunity to clarify the evidence being presented. Often non-conformities are the result of miscommunication between the auditor and auditee. This happens frequently when the auditor has a poor understanding of the process being audited. It is a tremendous waste of time for both sides when this occurs. If there is an actual non-conformity, it is also important to gather as much objective evidence as possible for the auditor to write a thorough finding and for the auditee to prepare an appropriate corrective action plan in response to the finding.

Feedback from the Auditee

I always encourage auditees to provide honest feedback to me directly and to management so that I could continue to improve. If you are giving feedback about an internal auditor or a supplier auditor, you should always give feedback directly before going to the person’s superior. You are both likely to work together in the future, and you should give the person every opportunity to hear the feedback first-hand.

When providing feedback from a 3^rd party Certification Audit, you should know that there will be no negative repercussions against your company if you complain directly to the Certification Body. At most, the Certification Body will assign a new auditor for future audits and investigate the need for taking action with the auditor. In all likelihood, any action taken will be “retraining.” I never fired somebody for a single incident—unless they broke the law or did something that was unsafe. The key to providing feedback, however, is to be objective. Give specific examples in your complaint, and avoid personal feelings and opinions.

Complaints and Appeals

As the auditee, you should ask for the contact information at the certification body during the opening meeting. Ask with a smile—just-in-case you disagree and so you can provide feedback (which might be positive). As the auditor, you should always make the contact information for the certification body available. If you are conducting a supplier audit or an internal audit, you probably know the auditor’s boss and there is probably no formal complaint or appeals process. In the case of a supplier audit, the customer is always right—even when they are wrong.

During the Audit

During the audit you should always make the guide(s) and process owner(s) aware of any potential non-conformities as you find them. This is their opportunity to clarify the objective evidence for you and to explain why there is not a non-conformity. Often I will refer to the Standard that I am auditing to at this point. I will identify the specific requirement(s) and show the process owner. I will say, “This is what I am trying to verify. Do you have anything that would help address this requirement?” If the process owner is not sure of how to meet the requirement, often I will provide an example of how this requirement is addressed in other areas or at other companies.

If the audit is a multi-day audit, I will review the potential nonconformities at the end of the day and give the auditee the opportunity to provide additional objective evidence in the morning. If it is already the last day of the audit or it is a single-day audit, I will give auditees until the closing meeting to provide the objective evidence. Often I will use this opportunity to explain what would be considered a minor non-conformity and what would be a major non-conformity. Usually I can say, “This is definitely not a major non-conformity, because…”

Closing Meeting

At the closing meeting, the auditee should never be surprised. If an issue remains unfulfilled at the closing meeting, the auditee should be expecting a minor non-conformity—unless the issue clearly warrants a major non-conformity. Since a minor nonconformity is described as “a single lapse in the fulfillment of a requirement,” it is difficult for an auditee to argue that an issue does not warrant a minor non-conformity. Typically, the argument is that you are not consistent with other auditors. The most common response to that issue is, “Audits are just a sample, and previous auditors may not have seen the same objective evidence.” The more likely scenario, however, is that the previous auditor interprets the requirements instead of reviewing the requirements with the client and making sure both parties agree before a finding is issued.

A major nonconformity is usually defined as one of the following: 1) “a total absence in the fulfillment of a requirement,” 2) “repetition of a previous nonconformity,” 3) “failure to address a previously identified minor non-conformity”, or 4) “shipment of non-conforming product.” When the auditor is in doubt, then the finding is minor and never a major. For a major non-conformity to be issued there can be no doubt. If a finding is major, the auditee should have very few questions. Also, I find that often the reason for a major non-conformity is a lack of management commitment to address the root cause of a problem. Issuing a major non-conformity is sometimes necessary to get management attention.

Regardless of the grading, all audit findings will require a corrective action plan—even an FDA warning letter requires a CAPA plan. Therefore, a major non-conformity is not a disaster. You just need to create a more urgent plan for action.

What are the 6 New Essential Requirements?

Click HERE if you want to receive future “European Regulatory Updates” by email. Just provide your Name, Company, Phone Number, and the email address where you would like to receive updates.

Annex I of the European Medical Device Directive (http://bit.ly/M5MDD) is titled “Essential Requirements.” Most companies demonstrate that their device meets the 13 Essential Requirements (ERs) by creating an Essential Requirements Checklist (ERC). I have no idea what the origin of the ERC is, but you know that regulators love tables and checklists. This particular checklist is so commonly used that the Global Harmonization Task Force (GHTF) included an example of an ERC, called an “Essential Principles Checklist” (EPC) at the end of a guidance document on how to create Summary Technical Documentation (STED) for In Vitro Diagnostic devices (http://bit.ly/STEDIVD)—which is now maintained on the IMDRF.org website.

On September 26, 2012, the European Commission released a proposal for new EU Medical Device Regulations (http://bit.ly/EUProposal). This proposal still includes ERs in Annex I, but there are 19 ERs in the proposal. One regulatory professional recently sent me a follow-up question in response to an audio seminar I conducted in November (). Her question was, “What are the six new ERs?”

A few of the early reviews of the proposal indicated that there were no significant changes, but I have learned the hard way that you should always go to the source and verify the information for yourself (i.e. – Genchi Genbutsu). Here’s what I found:

General Requirements (ER 1-6a)

1. No real change to this requirement.
2. This requirement was reworded to clarify the intent (see Annex ZA of EN 14971:2012 for more info @ http://bit.ly/ISO14971-2012changes).
3. It appears as though the Commission thought the current ER 3 was redundant and the requirement was addressed by ER 1 and ER 5 already.
4. This is now the new ER 3, and the requirement now clarifies how Notified Bodies shall apply this requirement in cases where a lifetime of the device is not stated.
5. This is now the new ER 4, and there is no real change.
6. This is now the new ER 5, and the wording has been clarified.

ER6a is conspicuously missing from the proposed ERs, but don’t get excited. Clinical Evaluations are still required as part of the Technical Documentation in Annex II, Section 6.1c: “the report on the clinical evaluation in accordance with Article 49(5) and Part A of Annex XIII.”

Chemical, Physical & Biological Properties (ER 7)

ER 7.1 has one new requirement: “d) the choice of materials used, reflecting, where appropriate, matters such as hardness, wear and fatigue strength.” ER 7.2 and 7.3 remain unchanged. ER 7.4 has been simplified to what is proposed as the new, shorter ER 9. ER 7.5 is now the new ER 7.4, and the changes reflect the current status of phthalate regulations and similar issues. ER 7.6 is now the new ER 7.5, but there is no change to the content. The new ER 7.6 requires that manufacturers address the risks associated with the size and properties of particles—especially nanomaterials. The changes associated with this section will impact certain device types more than others—such as orthopedic implants.

Infection & Microbial Contamination (ER 8)

ER 8 is still ER 8, but ER 8.1 is now prescriptive regarding design solutions and the current ER 8.2 is now the new ER 10. The new ER 10 is expanded and references the new EU Regulations regarding devices manufactured utilizing tissues or cells of animal origin: Commission Regulation (EU) No 722/2012 of 8 August 2012 (http://bit.ly/AnimalTissueReg). The new ER 8.2 is a new requirement that was an oversight of the MDD, and the new ER 8.7 now clarifies that the labeling must differentiate sterile and non-sterile versions of the product; packaging is no longer an acceptable mechanism for differentiation. The balance of ER 8 remains unchanged.

Construction & Environmental Properties (ER 9)

This ER is now identified as the new ER 11, and this section is expanded. This reflects the emphasis on the need to evaluate the safety of devices with accessories, compatibility with other devices, and the affects of the use environment.

Devices with a Measuring Function (ER 10)

This ER is now identified as the new ER 12, but ER 10.2 from the current Directive appears to be missing. What’s up?

Take a look at the new ER 11. ER 10.2 is now the new ER 11.6.

Protection Against Radiation (ER 11)

This ER is now identified as the new ER 13, but there is nothing new.

Requirements for Devices Connected to or Equipped with an Energy Source (ER 12)

ER 12.1 and 12.1a are now ER 14. This section is specific to software requirements and has more detail than the current Directive. IEC 62304:2006, “Medical device software – Software life cycle processes,” is the Standard that will be expected by Notified Bodies as a reference for ER 14. ER 12.2 through ER 12.6 are now ER 15, but there is nothing new. This Section ER 12.7 and its sub-parts are now addressed by ER 16. ER 12.8 and its sub-parts are now addressed by ER 17.

Information Supplied by the Manufacturer (ER 13)

This is now identified as ER 19: “Label and Instructions for Use.” This section is simplified from ER 13 (i.e. – there are fewer sections), but this ER does not seem to be any shorter. ER 19.1 has sub-parts a-g, and this ER section incorporates the concepts previously addressed by ER 13.1, 13.2, 13.4 and 13.5. ER 19.2 is a new and improved version of the previous ER 13.3 specific to labeling requirements. This labeling section is expanded from sub-parts “a” through “n” to “a” through “q”. The UDI requirement is sub-part “h”. ER 13.6 is now ER 19.3 specific to the instructions for use (IFU). This section is expanded from sub-parts “a” through “q” to “a” through “t”.

The number of sub-parts to ER 19.3 doesn’t reflect the additional requirements for IFUs that are proposed by the Commission. The sub-sections of this part warrant special attention. Items that frequently are found missing from IFUs on the market today include:

1. ER 19.3c – performance intended by the manufacturer
2. ER 19.3h – installation and calibration instructions
3. ER 19.3k – how to determine if a re-usable device should be repaired/replaced
4. ER 19.3m – restrictions on combinations with other devices
5. ER 19.3o – detailed warning information
6. ER 19.3p – information about safe disposal of the device
7. ER 19.3t – notice to user/patient to report adverse events

ER 18 – Use by Lay Persons

This is a short section, but the requirement is new. There are now additional requirements for products intended for use by a lay person. The Risk Management Report, Design Validation, and Clinical Evaluation Report will need to include specific evidence to demonstrate conformity with this ER. The Post-Market Surveillance Plan for these products should carefully verify the accuracy of risk estimates. Post-Market Clinical Follow-up (PMCF) Studies would be challenging in the past, but the prevalence of social media and product registration databases may facilitate conducting PMCF Studies for these products in the future.

Australia & Canada

There is also an EPC that is required by the Therapeutic Goods Administration (TGA) in Australia (http://bit.ly/EPCTGA) and by the Therapeutics Product Directorate (TPD) in Canada (http://bit.ly/CanadianSTED). If you would like to learn more about the Essential Principles of Safety and Performance you should also review the GHTF guidance document on this topic (http://bit.ly/EPSafetyPerf) on the IMDRF.org website. This 2012 version of the document supersedes GHTF/SG1/N041:2005.

I have observed approval of products where the European ERC was submitted in lieu of an EPC for Australia and Canada. I guess they are a little more rationale than some other regulators, but if you have experienced any “push back” regarding this approach please share this by posting a comment or emailing me: rob@13485cert.com.