A New Way to Grade Findings

Last November a new GHTF document was released on the topic of grading non-conformities: GHTF/SG3/N19:2012. This document is available on the new IMDRF website in the documents section. The 16-page document presents a new method for Certification Bodies to grade non-conformities and to communicate these findings to regulators such as the US FDA and Health Canada (e.g. – GD211 voluntary reports).

To download the guidance document, go to http://www.imdrf.org/.

N19 recommends the same three-part structure for writing nonconformities that is taught in Lead Auditor Classes, and there is even a table of examples provided with poorly written findings and well-written findings with more specific references to objective evidence.

Section 4.2 of the guidance document, however, introduces a new concept for grading of findings. The traditional grading of findings is: Major, Minor, and Observations. Opportunities for Improvement (OFI) are no longer allowed in regulatory reports to avoid the appearance of providing consulting advice to clients. For internal audits and supplier audits, OFIs are still used by most auditors.

The new grading process defined by the guidance document has a two-step process. The first step uses a grading matrix to quantitatively determine a grade for the finding based upon the impact upon the QMS and the frequency of occurrence.

The second step of the grading process is to review escalation rules that are defined in Section 4.2.2 of the guidance document. This section emphasizes the importance of using the word “absence” in the wording of findings if a required procedure is not present in the QMS. This type of finding should only happen during initiate certification audits where 100% of the required procedures are typically verified during the Stage 1 audit. If this occurs, then the grading is increased by 1 to a possible maximum of 5.

Another possible escalation event is the release of nonconforming devices outside the control of the manufacturer. If this occurs, then the grading is increased by 1 to a possible maximum of 5. If the required procedure is absent, and product is released that is nonconforming, the guidance states that the score should not be escalated above a 5.

In all of the Lead auditing courses I have taught, both of the above escalation events would be examples of a “Major Nonconformity.” Repeat occurrences of nonconformities would typically be escalated from a minor NC to a major NC, but in this new method the scores could be a “2” or a “4”—depending upon the impact upon the QMS.

I have had enough trouble in the past with training auditors to consistently grade findings during audits, and this is one of the most important sections of the exam for a Lead Auditing Course. Recently I suggested that a client consider using the risk analysis matrix that they were already using for process risk analysis and apply the matrix to grading of findings. An example of this type of matrix is shown below.

My client used semi-quantitative scores for severity (1-3) and occurrence (1-4). The two factors were multiplied to calculate a risk priority number (RPN) ranging from 1-12. The resulting matrix is also color coded to indicate the urgency of corrective action plans to be developed for the finding.

Has anyone implemented a grading system based upon this new guidance? If you have, please share your experiences here or on one of the LinkedIn Groups I have posted this question:

Medical Devices: QA/RA – http://bit.ly/SG3N19-QARA

ASQ – http://bit.ly/SG3N19-ASQ

Please share you own methods for grading findings?

Advertisement

How do you prepare for ISO 13485 registration?

A LinkedIn connection of mine recently asked for sources of good guidance on ISO 13485 registration. I wrote a blog recently about Quality Management Systems in General, but I had trouble finding resources specific to the ISO 13485 registration process. Therefore, I decided to write a blog to answer this question.

Here’s my favorite movie clip with a song for you.

Typically people learn the hard way by setting up a system from scratch. The better way to learn it is to take a course on it. I used to teach a 2-day course on the topic for BSI. The link for this course is: http://bit.ly/Get13485; I shortened the link to the BSI website.

Other registrars offer this course too. I suspect you can find a webinar on this through TUV SÜD, BSI, SGS, LNE/GMED, Dekra, etc. from time to time.

The only registrar I could find that described the process step-by-step was Dekra. I have copied their steps below:

Inquiry to Surveillance in 5 Steps

1. Inquiry
An initial meeting between [THE REGISTRAR] and the client can take place on site or via teleconference. At this time, the client familiarizes [THE REGISTRAR] with company specifics and its quality assurance certification requirements; [THE REGISTRAR] explains its working methods and partnering philosophy, and previews the details of the process.

Rob’s 2 Cents

As a client I have completed two initial certifications personally and three transfers, but I have only once had the sales representative actually visit my company. I think this process is typically accomplished by phone and email. If any registrars are reading this, you will close on more accounts if you visit prospective clients personally. In fact, the one that actually visited my company (Robert Dostert) has been on speed dial for almost a decade and he’s received a bit of repeat business.

2. Application Form
The client chooses to move forward by filling out an online application form. Based on the information obtained during the inquiry stage, along with the application form, [THE REGISTRAR] prepares a quote, free of charge, for the entire certification process. A client-signed quotation or purchase order leads to the first stage of the certification process.

Rob’s 2 Cents

For both of the Notified Body transfers I completed, I completed application forms and requested quotes from multiple Notified Bodies. During the quoting process, my friend Robert was more responsive and able to answer my questions better than the competition. Robert was also able to schedule earlier audit dates than the competition. To this day I am still amazed that Notified Bodies are not more responsive during this initial quoting process. All of the Notified Bodies are offering a certificate (a commodity). The customer service provided by each Notified Body, however, is not a commodity. Each Notified Body has it’s own culture, and every Notified Body has good and bad auditors. Therefore, you need to treat this selection process just like any other supplier selection decision. I have provided guidance on this specific selection process on more than one occasion, but I am definitely biased.

3. Phase One: Document Review and Planning Visit

LNE/GMED Flow Diagram for the process of ISO 13485 Certification

At this stage, [THE REGISTRAR] performs a pre-certification visit, which entails verifying the documented quality systems against the applicable standard. [THE REGISTRAR] works with the client to establish a working plan to define the [THE REGISTRAR] Quality Auditing process. If the client wishes, [THE REGISTRAR] will perform a trial audit or “dress rehearsal” at this stage. This allows the client to choose business activities for auditing and to test those activities against the applicable standard. It also allows the client to learn and experience [THE REGISTRAR] ‘s Quality Auditing methods and style. The results of the trial audit can be used toward certification. Most clients elect for one or two days of trial auditing.

Rob’s 2 Cents

Dekra’s statement that, “The results of the trial audit can be used toward certification,” is 100% opposite from BSI’s policy. BSI calls this a pre-assessment. The boilerplate wording used in BSI quotations is, “The pre-assessment is optional service that is an informal assessment activity intended to identify areas of concern where further attention would be beneficial and to assess the readiness of the quality management system for the initial formal assessment.” During these pre-assessments, BSI auditors explain that any findings during the pre-assessment will not used during the Stage 1 and Stage 2 certification audits, and the client will start with a “clean slate.” Most of the clients I conducted pre-assessments for were skeptical of this, but most auditors are ethical and make a every effort to avoid even the perception of biasing their sampling during the Stage 1 and Stage 2 audits. I highly recommend conducting a pre-assessment. You want an extremely thorough and tough pre-assessment so that the organization is well prepared for the certification audits. If the auditor that will be conducting the Stage 1 and Stage 2 audit is not available to conduct a pre-assessment, try to find a consultant that knows the auditors style and “hot buttons” well. FYI…You can almost always encourage me to do a little teaching when I’m auditing (I just can’t resist), and my “hot buttons” are CAPA,  Internal Auditing, and Design Controls.

4. Phase Two: Final Certification Audit 
Once the client’s documented systems have met the applicable standards, [THE REGISTRAR] will conduct an audit to determine its effective implementation.  [THE REGISTRAR] uses a professional auditing interview style instead of a simple checklist approach. This involves interviewing the authorized and responsible personnel as designated in the documented quality system.

Rob’s 2 Cents

For certification audits, ISO 17021 requires a Stage 1 and Stage 2 audit to be conducted. The combined duration of the certification audits must be in accordance with the IAF MD9 guidance document–which is primarily based upon the number of employees in the company. The “interview style” that Dekra is referring to is called the “Process Approach”. This is required in section 0.2 of the ISO 13485 Standard, and this is the primary method recommended by the ISO 19011 Standard for auditing–although other methods of auditing are covered as well.

5. Surveillance 
[THE REGISTRAR] arranges for surveillance audits semi-annually or annually as requested by the client.

Rob’s 2 Cents

I highly recommend annual surveillance audits, because the short duration of surveillance audits becomes unrealistically short when the auditor is asked to split their time between two semi-annual visits. A few clients have indicated that the semi-annual audits help them by maintaining pressure on the organization to be ready for audits all year-round and prevents them from procrastinating to implement corrective actions. This is really an issue of management commitment that needs to be addressed by the company. Scheduling semi-annual surveillance audits is doesn’t address the root cause. The only good argument I have for semi-annual cycles is if you have a very large facilities that would have an audit duration of at least 2 days on a semi-annual basis. The most important think to remember about scheduling surveillance audits is to make sure that you schedule the audits well before the anniversary. I recommend 11 months between audits. By doing this, you end up scheduling the re-certification audits 3 months before the certificate expires. BSI has a different policy. They want auditors to schedule the first surveillance audit 10 months after the Stage 2 audit, the second surveillance audit 12 months after the first surveillance audit, and then the re-certification audit must be scheduled at least 60 days prior to certificate expiration (i.e.  – no more than 12 months after the second surveillance audit). No matter what, schedule early.

If you have additional questions about becoming ISO 13485 registered, please post a discussion question in the following LinkedIn subgroup: Medical Device: QA/RA. For example, on Monday a new discussion question was posted asking for help with selection of a Notified Body for CE Marking. You will need to become a member of the parent group (Medical Device Group)–if you are not already one of the 140,000+ members connected with Joe Hage. George Marcel and I manage this subgroup for Joe. George is out in the Bay Area and I’m in the Green Mountains.

Where can I learn about this Quality Management System (QMS) stuff?

A blog follower from Jon Speer’s website, Creo Quality, recently sent me a message asking for sources of good guidance on this Quality Management System (QMS) Stuff. There are a bunch of links below for you to follow and some practical advice. Enjoy learning!

J’aime Pink Martini et le chant de China Forbes.

The single best guidance document on the implementation of a QMS system in accordance with ISO 13485 is “13485 Plus” (type in the words in quotes to the CSA search engine).

There are also a bunch of pocket guides you can purchase for either ISO 9001 or ISO 13485 to help you quickly look-up information you are having trouble remembering. One of my Lead Auditor students recommended one pocket guide in particular and she was kind enough to give me her copy.

There are some webinars out there that provide an overview of QMS Standards. Some are free and some have a modest fee. I’m not sure the value is there for these basic overview webinars, but if you need to train a group it’s a great solution. I know BSI has several webinars that are recorded for this purpose.

AAMI has an excellent course on the Quality System Regulations (QSR) which combines 21 CFR 820 and ISO 13485.

There are a number of blogs I recommend on my website.

You can try to identify a local mentor–either in your won company or at your local ASQ Section.

You can join the following LinkedIn subgroup: Medical Device: QA/RA. You will need to become a member of the parent group (Medical Device Group)–if you are not already one of the 140,000+ members connected with Joe Hage. George Marcel and I manage this subgroup for Joe. George is out in the Bay Area and I’m in the Green Mountains.

You can visit the Elsmar Cove website and participate in the discussions you find there. I wrote a blog about Elsmar Cove a while back (wow almost 2 years ago now).

The best way to learn this stuff is to do all of the above.

And for the encore performance…

Death by CAPA

I have no theme to relate this song with my posting, but you just can’t go wrong with blue jeans and a black t-shirt…

You might want to play this video twice…it’s a long posting.

I completed almost 100 audits in the past two years, and I review the Corrective Action and Preventive Action (CAPA) process during every single audit. Surprisingly, this seems to be a process with more variation from company to company than almost any other process I review. This also seems to be a major source of non-conformities. In the ISO 13485 Standard, clause 8.5.2 (Corrective Action) and clause 8.5.3 (Preventive Action) have almost identical requirements. Third-party auditors, however, emphasize that these are two separate clauses. We are purists. Although we acknowledge that companies may implement preventive actions as an extension to a corrective action, we also expect to see examples of actions that are strictly preventive in nature.

Many companies seem to be confused, but it doesn’t need to be. Just ask yourself one question. What is the source of this action?

If the answer is a complaint, audit nonconformity, or rejected components—then your actions are corrective.

If the answer is, a negative trend that is still within specifications or an “opportunity for improvement” (OFI) identified by an auditor—then your actions are preventive.

If you are investigating the root cause of a complaint, people will sample additional records to estimate the frequency of the quality issue. I describe this as investigating the depth of a problem. The FDA emphasizes the need to look to other product lines, or processes, to see if a similar problem exists. I describe this as investigating the breadth of a problem. Most companies describe actions taken on other product lines and/or processes as “preventive actions.” This is not always accurate. If a problem is found elsewhere, actions taken are corrective. If potential problems are found elsewhere, actions taken are preventive. You could have both types of actions, but most people incorrectly identify corrective actions as preventive actions.

Another common mistake is to characterize corrections as corrective actions.

The most striking difference between companies seems to be the number of CAPAs they initiate. There are many reasons, but the primary reason is failure to use a risk-based approach to CAPAs. Not every quality issue should result in the initiation of a formal CAPA. The first step is to investigate the root cause of a quality issue. The FDA requires that the root cause investigation is documented, but if you already have an open CAPA for the same root cause…

DO NOT OPEN A NEW CAPA!!!

If you do not have a CAPA open for the root cause that you identify, then what should you do?

I know this will shock everyone, but…it depends.

The image below gives you my basic philosophy.

 

 

 

 

 

 

 

 

Most investigations document the estimated probability of occurrence for a quality issue. This is only half of the necessary risk analysis I describe below. Another aspect of an investigation is to document the severity of potential harm resulting from the quality issue. If customer satisfaction, safety or efficacy are affected by a quality issue—the severity is big. Risk is the product of severity and probability of occurrence.

If the estimated risk is low and probability of occurrence is known, then alert limits and action limits can be statistically derived. These quality issues are candidates for continued trend analysis—although the alert limit or action limit may be modified in response to an investigation. If the trend analysis results in identifying events that require action, then that is the time when a formal CAPA should be opened. If the trend remains below your alert limit, then no formal CAPA is needed.

If the estimated risk is moderate or the probability of occurrence is unknown, then a formal CAPA should be considered. Ideally, you will be able to establish a base-line for occurrence and demonstrate that frequency decreases upon implementation of corrective actions. If you can demonstrate a significant drop in frequency, this verifies effectiveness of actions taken. If you need statistics to show a difference, then your actions are not effective.

If estimated risk is high or there are multiple causes that require multiple corrective actions, a quality improvement plan may be more appropriate. There are two clauses in the Standard that apply. Clause 5.4.2 addresses planning of changes to the Quality Management System. For example, if you correct problems with your incoming inspection process—this addresses 5.4.2. Clause 7.1 addresses planning of product realization. For example, if you correct problems with a component specification where the incoming inspection process is not effective—this addresses 7.1. Depending upon the number of contributing causes and the complexity of implementing solutions, the plan could be longer or shorter. If it will take more than 90 days to implement a corrective action, you might consider the following approach.

Step 1 – open a CAPA

Step 2 – identify the initiation of a quality plan as one of your corrective actions

Step 3 – close the CAPA when your quality plan is initiated (i.e. – documented and approved)

Step 4 –verify effectiveness by reviewing the progress of the quality plan in management reviews and other meeting forums…you can cross-reference the CAPA with the appropriate management review meeting minutes in your effectiveness section

If the corrective action required is installation of new equipment and validating that equipment, the CAPA can be closed as soon as a validation plan is created. The effectiveness of the CAPA is verified when the validation protocol is successfully implemented and a positive conclusion is reached. The same approach also works for implementing software solutions to better manage processes. The basic strategy is to get the long-term improvement projects started with the CAPA system, but monitor the status of these projects outside the CAPA system.

Best practices would be the implementation of Six-sigma projects with formal charters for each long-term improvement project.

NOTE: I believe in closing CAPAs when actions are implemented, and tracking the effectiveness checks for CAPAs as a separate quality system metric. If closure takes more than 90 days, the CAPA should probably be converted to a Quality Plan. This is NOT intended to be a “work around” to give companies a way to extend CAPAs that are not making progress in a timely manner.