How do you audit for compliance with ISO 14971:2012?

Let’s say that you went ahead and purchased ISO 14971:2012, read Annex ZA, and you identified a couple of gaps in your procedure. After you revised your Risk Management Procedure to be compliant with the revised Standard, then what are you supposed to do?

For the next few weeks I plan to torture all of you with holiday music. If you don’t like it, buy a satellite radio for Christmas sake.

Most QA Managers struggle over whether they should purchase ISO 14971:2012 or not. I wrote a couple of blog postings about this, but my point was not to debate this question. My point was that companies need to be compliant with the MDD and the ISO 14971 Standard. The “changes” from the 2009 to the 2012 version are simply the European Commission reminding manufacturers that there are 7 aspects of the ISO 14791 Standard that  do not meet the requirements of the MDD. Therefore, if your company has already verified that your Risk Management Process is compliant with the MDD–then you have nothing to change. However, if your Risk Management Process is only compliant with ISO 14971:2009, then you need to revise your processes and procedures to address these 7 aspects.

Once you have made your revisions, how do you audit for compliance with ISO 14971:2012?

Step 1: Planning the Audit

This will be an internal audit and since you (the QA Manager) are the process owner for the Risk Management process, you cannot also audit this process. You need to assign someone that has the technical skill to perform the audit, but this person cannot be the process owner (you) or a direct report to the process owner (the rest of the QA department). Fortunately, the Director of Engineering is also trained as an internal auditor at your company. She is trained on ISO 14971:2009, but she is not trained on ISO 14971:2012. To address this gap, she must read the updated Standard to understand what’s new.

Clause 3.2 of ISO 14971 requires that top management review the Risk Management Process for Effectiveness.

She has participated in risk management activities, but each product development engineer participates in risk management activities for their own design projects. Therefore, she has several projects she can sample risk management records from without auditing her own work. You have communicated that you need this audit finished sometime in December, because you want any CAPA’s resulting from the audit to be finalized before the next Management Review at the end of January. The timing of the Management Review is important, because the Risk Management Procedure requires that top management assess the effectiveness of the Risk Management Process during Management Review meetings.

There are no previous audit findings to close from the last audit of the Risk Management Process, but the Director of Engineering has 7 specific items to emphasize from the 2012 revision of the Standard and a revised procedure for Risk Management. Therefore, she will prepare for the audit by identifying some new interview questions to specifically address these changes–as well as some more generic, open-ended questions.

Specific Questions for 7 Items in ISO 14971:2012, Annex ZA:

1. How does the risk analysis evaluate the acceptability of risks in the lowest category? (This is a leading question, but it is specifically designed to determine if negligible risks are discarded.)

2. Please provide a few examples of how risks in the lowest category were reduced. (Sections 1 and 2 of the Annex I require all risks to be reduced as far as possible, and for all risks to be evaluated for acceptability. The wording of this question also allows auditors flexibility in their sampling.)

3.  How did the design team determine when they had implemented sufficient risk controls to minimize risks? (Many companies use a color-coded matrix as a quasi-objective method for determining when risks are adequately reduced. This process is often referred to as the ALARP concept. Annex ZA specifically prohibits using economic considerations as part of this determination.)

4. How did you conduct a risk-benefit analysis? (The Standard allows for performing a risk-benefit analysis when overall residual risks exceed the acceptability criteria as outlined in the risk management plan. However, the MDD requires an overall risk-benefit analysis in Section 1 of Annex I. Section 6 also requires that a risk-benefit analysis be performed for each individual risk.)

5. How were risk control options selected? (Section 2 of the MDD implies that the manufacturer shall review All the control options and pick the most appropriate ones. Therefore, the auditor should specifically look for evidence that the team systematically reviewed all possible control options to reduce risks–rather than stopping as soon as the risks were reduced to an acceptable level.)

6. What were your team’s priorities for implementation of risk control options? (It’s possible that the previous question will be sufficient to gather evidence that risk controls were implemented with the required prioritization as specified in the MDD. However, this question would be used as a follow-up question if it is not clear that the team prioritized the risk control options in accordance with Section 2 of Annex I.)

7. How was effect of labeling and warnings in the instructions for use incorporated into the estimation of residual risks? (Almost every company remembers to include residual risks in their IFU as a warning or caution statement. However, Section 2 of Annex I does not allow for including this information given to the users as a method of reducing risks. Therefore, in a Design FMEA you would not list labeling and IFUs in your column for current risk controls when you determine the risk. This should be identified as an action to be taken–with no impact on the score for residual risk.)

The above questions are not examples of using the process approach, but each question is phrased in an open-ended manner to maximize the objective evidence gathered during the interview process. If you are doing a process audit, it’s still ok to include questions that use the element approach.

Generic Questions:

1. When was the ISO 14971:2012 version of the Standard added to the controlled list of external Standards?

2. Please provide examples of where you have updated the Essential Requirements Checklist (a Technical File document) to reference the newest revision of ISO 14971:2012, and please show at least one example of how the Risk Management Report was updated to reflect this revision.

3. How did you verify training effectiveness for the design team specific to the updated Risk Management Procedure prior to conducting a risk analysis?

These generic questions do not require reading the ISO 14971:2012 Standard. Instead, each question forces the auditee to demonstrate their knowledge of the revised Standard by asking open-ended interview questions. Each of these questions is also designed to test linkages with other support processes. This is an example of how to use the process approach.

Step 2: Conducting the Audit

The next step of the auditing process is to conduct the audit. During the audit, the Director of Engineering will gather objective evidence of both conformity and nonconformity for the risk management process. The generic interview questions that were developed allow her to evaluate the effectiveness of linkages between the Risk Management Process and other processes such as: 1) document control, 2) creating technical documentation for regulatory submissions, and 3) the training process. The specific questions verify that each of the 7 elements identified in Annex ZA of ISO 14971:2012 are adequately addressed in the revised procedure. When the audit is completed, the auditor will have a closing meeting with the process owner (you) and the auditee(s) so that everyone is clear what the findings were, and if there were any nonconformities this is the time to clarify what needs to be done in order to prevent each nonconformity from recurring.

Step 3: Writing the Report & Taking Corrective Action(s)

This is no different from any other audit, but it is critical to have the report completed soon enough so that CAPA’s can be initiated (not necessarily completed) prior to the Management Review.

Step 4: Verifying Effectiveness of Corrective Action(s)

Many people struggle with verifying effectiveness of corrective actions–regardless of the process. My advice is to identify a process metric to measure the effectiveness. Then the effectiveness check is objective. For example, monitoring the frequently of updates to the list of external standards can help verify that the process for monitoring when Standards are updated is effective. Likewise, the frequency of updates to the Essential Requirements Checklist and the Risk Management records referenced in the Essential Requirements Checklist indicates if the Risk Management process is  being maintained. Finally, monitoring the lag between the time procedures are updated and when the associated training records are updated quickly identifies if there is a systemic problem with training or if a train gap is just an example of a single lapse.

Advertisement

ISO 14971 – Buy the new 2012 version?…comment please

I’m sure that there are some that disagree with my determination that the latest revision of EN 14971, revision 2012, is unnecessary (the European Commission certainly does).

 You will have to go to my website to read my cheeky posting on this topic.

And here’s another cheeky attitude from the UK…(sorry, this is not a family channel).

Therefore, I would like to clarify why I feel this way by reviewing how risk is addressed in the MDD (93/42/EEC as modified by 2007/47/EC).

1. The term risk is mentioned only 4 times in the Articles in the MDD
2. The term risk is mentioned once in Annex II and III, twice in Annex VII, and three times in Annex VIII and X—for a total of 10 times.
3. The other 41 times risk is mentioned are in the Essential Requirements (i.e. – Annex I).

When companies submit a Design Dossier for review by a Notified Body, an Essential Requirements Checklist is included. This references, in table format, how all the requirements of Annex I are being met—including those related to risks. Throughout Annex I, a similar phrase is repeated many times. For example, in the first Essential Requirement (ER1) it states: “…any risks which may be associated with [a device’s] intended use [shall] constitute acceptable risks when weighed against the benefits to the patient and are compatible with a high level of protection of health and safety.” In ER2 it states: “the manufacturer must…eliminate or reduce risks as far as possible…”. There is no room in the MDD for consideration of cost or economic impact when the manufacturer is designing a device with regard to risks and benefits.

If a company’s Risk Management Procedure has been found to be acceptable by a Notified Body, and the company has addressed all the Essential Requirements (ERs) with regard to risk, then there should be no impact from these 7 deviations identified in EN 14971:2012. However, if your company has not addressed each of these ERs, then you might want to consider each of these areas:

1. Treatment of negligible risks
2. Discretionary power of the manufacturer as to the acceptability of risks
3. Risk reduction “as low as possible” (ALAP) verses “as low as reasonably possible” (ALARP)
4. Discretion as to whether as risk benefit analysis needs to take place
5. Discretion as to the risk control option/measures
6. Deviation as to the first risk control method
7. Information of the users influencing the residual risk

My final advice is to review Annex I and Annex X from the perspective of risk management. You may realize that you have some gaps that nobody noticed. After all, audits are just a sample.

PS – I think it’s ironic that the origins of the ALARP principle are UK case law (see link above).

Best in Class Validation Program

This is one of the early music videos I remember from the 80’s.

The original question from a former client was: “What does a best in class CNC machining process validation program look like?” Although I intend to answer this question, I know a few other clients that have done a great job of this. Hopefully they will add their own opinions as a comment. Therefore, I am expanding the scope of this question to validation in general.

The problem with validation is that you can always do a more thorough validation. Only in the cases of processes such as sterilization, do we have ISO Standards that tell us what is required. Otherwise, we are normally the experts and we have to use our own judgment as to what is necessary. In general, the best approach is a risk-based approach.

For each design specification established for a component, we also need to identify what process risks are associated with failure to meet the specification. Most companies perform a process failure modes and effects analysis (pFMEA). This risk analysis has three quantitative components: 1) severity of the failure’s effect, 2) probability of occurrence, and 3) detectability.          The first factor, severity, is based upon the intended use of the device and how that component failure impacts that use. Usually it is important to have a medical professional involved in this portion of the estimation.

The second factor, probability, is typically quantified during the process validation activities. One company I audited developed a ranking scale for probability that was linked directly to CpK of the process. Higher CpK values received lower scores, because the process was less likely to result in an out-of-specification component. Another company I worked for used a six-point logarithmic scale (i.e. – 10e-6 = 1, 10e-5 = 2, 10e-4 = 3, 10e-3 = 4, 10e-2 = 5, and 10e-1 = 6). This logarithmic scale was based on sterilization validation where a sterility assurance level of 10e-6 is considered “validated”.

The third factor, detectability, is best estimated by using a quantitative scale that is based upon a gauge R&R study or some other method of inspection method validation.

Most companies struggle with determination of what is acceptable for design risk analysis. However, for process risk analysis it is usually much easier to quantify the acceptable risk level.

Once you have determined that a process is not acceptable at the current residual risk level, then you must take corrective actions to reduce the risk. The first step to achieve this should be to review the process flow. There are critical control points that can be identified in the process flow. One of those places is at the end of the process at the inspection step in the process.

The inspection step in the process flow affects detectability of defects. For many automated processes, such as CNC machining, it is not reasonable to perform 100% inspection. Therefore, these processes require validation. Most engineers make the mistake of trying to validate every dimension that is machined. However, only some of the dimensions result in device failures. These are the dimensions that are critical to validate. Best practice is to calculate the process capability for meeting each of these critical specifications (i.e. – CpK). A minimum threshold should be established for the CpK (refer back to the process risk analysis for ideas on linking CpK to risk acceptance). Any CpK values below the threshold require a more consistent process. These are the component specifications that should be the focus of process validation efforts.

During a process validation, it is often advisable to perform a design of experiment (DOE) in order to quantify the affects of each process variable. Typically a DOE will evaluate the impact on CpK for each variable at a high, low and middle value while other variables are maintained at nominal values. Any variables that appear to have a significant impact on the CpK are candidates for performing an operational qualification (OQ). For a machining process, this could include spindle speeds, feed rates, and material hardness. If variation of the variable has little or no impact upon the CpK, then there is probably little benefit to inclusion of this variable in an OQ.

The output of an OQ validation should be high and low limits for each process variable that will result in a “good” part. Performance Qualification (PQ) validation is the final step of the process validation. In the PQ, most companies will conduct three repeat lots at nominal values for the variables. If the OQ is designed well, there is often little added value in the PQ. Therefore, the sample size is typically three lots of 10 samples each. If the OQ validation does not clearly identify safe operating limits for the variables, or the process has marginal capability (i.e. – a low CpK), then the OQ should be repeated and an additional DOE may be needed.

Here are a few resources for those of you that are in “Deviceland” and may not be aware of guidance on validation in other related industries:

1. Guidelines for the Validation of Chemical Methods for the FDA Foods Program (3/22/2012) – http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM298730.pdf
2. 2.      Process Validation: General Principles and Practices (January 2011) –  http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM070336.pdf?utm_campaign=Google2&utm_source=fdaSearch&utm_medium=website&utm_term=Process%20Validation:%20General%20Principles%20and%20Practices&utm_content=1
3. Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods (9/8/2011) –  http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf
4. 4.      CPG Sec. 490.100 Process Validation Requirements for Drug Products and Active Pharmaceutical Ingredients Subject to Pre-Market Approval (3/12/2004) –  http://www.fda.gov/ICECI/ComplianceManuals/CompliancePolicyGuidanceManual/ucm074411.htm?utm_campaign=Google2&utm_source=fdaSearch&utm_medium=website&utm_term=validation&utm_content=3
5. Q 2 (R1) Validation of analytical procedures: text and methodology (June 1995) – http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_000431.jsp&mid=WC0b01ac0580029593&jsenabled=true

The Perfect Pecan Pie – Take a “SWAG”

Here’s a riddle…

                How do you know when you’re getting old?

                When country starts sounding really gooood.

Despite what you might think, my intent is not to make fun of country. Everyone’s tastes change over time. At a young age I fell in love with the sticky sweet sound of jazz singers, but in recent years I have started to warm up to the sound of modern country singers. Why?

                In order to properly appreciate a story about love lost and disappointment you need to have experienced life’s little pimples. I think life has finally beat me up enough times that I can personally relate to the deeper melancholy lyrics of country music. Another reason for my greater appreciation is that modern country is a cousin to jazz—kind of bluesy. I think you can see how I’ve grown in this week’s music video selection. My choice is the theme song from the movie Crazy Heart, “The Weary Kind”, written and performed by Ryan Bingham.

My series on the subject of Risk Management training continues (see my most recent blogs on the same topic). In my Risk Management Training, I use the example of making “The Perfect Pecan Pie” as a practical example of applying the principles of Risk Management.

                One of the most important steps of the Risk Management process is hazard identification. I described the process I use for hazard identification in my earlier blog on Risk Management planning. Section 4 of the 14971:2007 Standard defines the requirements for Risk Analysis. Hazard identification is only the first step in Risk Analysis. If done correctly, you should be able to identify hundreds of things that can go wrong with your pecan pie (i.e. – hazard identification). The next step in Risk Analysis is prioritizing these hazards. Prioritizing hazards should focus on the “severity of effect” first. I prefer to use a 5-point scale of even numbers (2, 4, 6, 8 & 10). The reason for this is that I like to create a risk matrix that is 5×5 and I want to emphasize severity over probability of occurrence—the two factors that make up risk. I learned this strategy from an auditing client (Thank you for sharingJ).

                Any potential pit fall that could prevent on-time delivery of the perfect pecan pie should be identified in your Risk Management File, but not every potential hazard requires risk controls. Once I have identified the potential hazards, I estimate the probability of occurrence next. Probability is estimated on a 5-point scale also (1, 2, 3, 4 & 5). The product of the two estimates is the estimated risk. I like to set a threshold for risk controls at 10. Therefore, any hazard that deserves a 10 for severity of effect will automatically require implementation of risk controls. For each product, companies should establish their own criteria for risk acceptability (i.e. – the Risk Management Policy). The potential benefit of the product should also impact this policy. High risk products should have great benefits too.

                Most people struggle with estimating these two numbers. Don’t worry! Take a “SWAG” (scientific wild-ass guess). What matters is that the risk analysis is reviewed and updated. Companies seldom get the risk analysis right the first time so it is critical to review post-production data and update the risk analysis based upon this data. If people tell you that your pie is too sweet, try to estimate what percentage feel that it is too sweet and what percentage feel it is just fine.

                For example, I used to think that a toasted flavor was ok. Most of my family likes this flavor, but the rest of the world seems to hate it. Once you figure this out, you need to change your risk controls to make sure the pie doesn’t burn—even a little. You might try decreasing the temperature or increasing the monitoring frequency. Either way you will decrease the potential frequency of burning a pie.

                You have now completed Section 4 of the 14971:2007 Standard. Please remember there are eight slices to every pie:)

Risk Management – It’s Not My Job

There’s no deeper meaning to this week’s YouTube selection. I just thought I would share one of my favorite guitar soloists with you. The recording quality is only good, but just watching Tim play reveals how freakishly good he is. I highly recommend the live CD with Dave Matthews and Tim Reynolds. If someone knows of a better quality recording that I can post in my blog, please let me know.

Have you experienced a discussion similar to this?

Auditor: “How do you manage risk throughout the production process?”

Auditee: “That is the responsibility of our customers. We will prepare a risk analysis if customers pay for it, but usually customers do the risk analysis.”

Most contract manufacturers in the medical device industry exclude design from their Quality Management Systems. Unfortunately, most of the contract manufacturers also associate risk management with only the design process. Risk Management cannot be “not applicable” in an ISO 13485 Quality Management System. The requirement of section 7.1 is: “The organization shall establish documented requirements for risk management throughout product realization. Records arising from risk management shall be maintained.” The Standard also references ISO 14971 as a source of guidance on Risk Management.

                For a contract manufacturer, compliance with ISO 14971 is not my primary concern as an auditor. My primary concern is to verify that contract manufacturers analyze risks associated with the processes that they perform and do their best to minimize those risks. What I don’t understand is why more companies don’t want to have strong risk management process. Risk management is how we prevent bad things from happening. Bad things like scrap, complaints and recalls. Should we expect our suppliers to have a strong risk management process?

                Duh.

                Contract manufacturers should be doing everything they can to get better at risk management. During pre-production planning they should be asking, “What happens if…” The contract manufacturer knows best HOW things will fail in production, while the customer knows best WHAT happens when things fail in production. In order to be safe and effective, both companies need to collaborate on risk analysis.

                The reason companies avoid doing risk analysis is because it’s time consuming and tedious.

                 Too bad, so sad.

                 Balancing my checkbook is time consuming and tedious too, but I balance my checkbook to prevent an overdraft charge. Not doing risk analysis can be much more painful. Scrapping out a part can cost tens or hundreds of dollars. Complaints can cost thousands of dollars. Recalls can cost millions of dollars.

                If I owned a contract manufacturing company, I would make sure that everyone in the company is involved in risk management, because we don’t want scrap, we can’t afford mistakes that lead to complaints, and a recall will put us out of business.