2016001: Hope Creek's 1st SRV Inspection Responce To Me

Reposted from 9/30/2017

The questions I posed to the NRC to drive this article is on the NRC's docket.

May 10, 2016

SUBJECT: HOPE CREEK GENERATING STATION UNIT 1 – INTEGRATED INSPECTION REPORT 05000354/2016001

Annual Sample:  Safety Relief Valve Set Point Drift

a. Inspection Scope

The inspectors reviewed PSEG's identification, evaluation, and corrective actions associated with longstanding main steam safety relief valve (SRV) set point drift issues at HCGS.  Specifically, at HCGS, one or more SRVs have exceeded the TS allowable as-found lift set point acceptance criteria in 17 of the 19 operating cycles over the life  of the plant (See Section 4OA3.1 for a review of Licensee Event Report (LER) 05000354/2015-004-01 related to as-found test results from refueling outage 19 (RF19)).  PSEG contracted with NWS Technologies to perform SRV as-found testing, SRV pilot valve assembly inspection and repair, and SRV as-left testing at their offsite facility.

The inspectors assessed PSEG’s problem identification threshold, technical and cause analyses, operating experience (OE) and trend reviews, vendor oversight, and the prioritization and timeliness of corrective actions to evaluate whether PSEG was appropriately identifying, characterizing, and correcting problems associated with these issues and whether the planned and/or completed corrective actions were appropriate.  The inspectors compared the actions taken in accordance with the requirements of PSEG’s and NWS’ maintenance procedures, PSEG’s CAP, 10 CFR 50 Appendix B,  Hope Creek’s TSs, and the Maintenance Rule.  The inspectors interviewed Nuclear Oversight (NOS) and engineering personnel to gain an understanding of potential operational challenges, overpressure protection capability and margin management, NWS performance, planned and completed corrective actions, and SRV performance.  The inspectors also reviewed NWS pilot assembly test and inspection documentation, including quality assurance (QA) acceptance and independent verifications, to ensure that NWS performed activities in accordance with prescribed procedures and industry standards.  In addition, the inspectors performed several walkdowns of SRV related instrumentation (including the control room, the remote shutdown panel, and the alternate shutdown automatic depressurization system panel instrumentation and alarm panels) to independently assess the material condition, operating environment, SRV performance, and configuration control.  [See also NRC Inspection Report 05000354/2012004, Section 4OA2.2, NRC Inspection Report 05000354/2013005, Section 4OA2.6, and NRC Inspection Report 05000354/2015003, Section 4OA2.4 for additional NRC assessment of the Hope Creek SRV issues.]

a. Findings and Observations

No NRC or self-revealing findings were identified.  A licensee-identified violation associated with as-found set point test failures in RF19 is documented in Section 4OA7.  

The Hope Creek main steam SRVs are 6" x 10" Target Rock Model 7567F, 2-stage SRVs consisting of a pilot stage, a main stage, and an air operator for remote operation.  Hope Creek has 14 safety-related main steam SRVs that provide reactor pressure vessel overpressure protection and an automatic/manual depressurization function.  Hope Creek TS 3.4.2.1, “Safety/Relief Valves,” requires that 13 of the 14 SRVs be operable with the specified code safety valve function lift setting (+/- 3 percent).  Hope Creek TS surveillance requirement 4.4.2.2 requires that at least half of the SRV pilot stage assemblies be removed and set pressure tested in accordance with the Surveillance Frequency Control Program (currently at the refueling outage (RFO) frequency of every 18 months).  Since RF15 in April 2009, PSEG has performed as found lift tests on all 14 SRV pilot valves every outage.  PSEG conducts this surveillance testing during RFOs when the SRVs are accessible during reactor shutdown conditions.  Historically, Hope Creek has experienced numerous as-found lift pressure failures during SRV testing.  Most recently, in June 2015, PSEG identified that 10 of 14 SRVs lifted above the TS specified pressure band (see Section 4OA3.1).

The Target Rock 2-stage SRV has an industry-wide history of set point drift.  Early documentation from General Electric (GE) identified that the Target Rock 2-stage SRV design was susceptible to corrosion bonding resulting in set point drift.  The corrosion bonding failure mode occurs due to bridging oxides created between the pilot disc surface and the pilot valve body disc seating surface during service.  The corrosion bonding trend results in the valve lifting at a higher pressure, failing to meet its set point criteria during the first lift attempt, but successfully lifting during consecutive tests (after the corrosion bond is broken during the first lift).  Over the years, PSEG

May 11 2018: The multiple failures of the coatings and new valve designs indicate massive experimentation going on in the reactor. They are suppose put these changes into a comprehensive testing regime guaranteeing there will no surprises and the changes worked as advertised before they to go into the reactor. I think all these changes are engineered to buy time for the utility, not really solve the problem. It is really fraud and corruption!!!    

personnel reviewed failure mechanisms and implemented maintenance recommendations from industry OE, GE Service Information Letters (SILs), and Boiling Water Reactor Owners Group recommendations in an unsuccessful attempt to address Target Rock pilot set point drift failures.  For example, the industry and PSEG have identified and implemented numerous mitigating strategies including: different pilot disc materials/coatings, addressing critical pilot disc and seat dimensions, correcting methods of insulation installation, and increased TS as-found set point margin (from +/- 1 percent to +/- 3 percent) in an attempt to improve Target Rock 2-stage SRV reliability.  The inspectors noted that PSEG implemented a mitigation strategy to install new pilot discs in all 14 SRV pilot valves every RFO; however, based on the continued set point drift failures, this aggressive practice has not proven effective at mitigating the corrosion bonding failure mechanism. 

During the review of Hope Creek LER 05000354/2010-002-01 in September 2011, NRC inspectors questioned whether multiple SRVs exceeding the TS allowable as-found lift set point acceptance criteria represented a significant CAQ (SCAQ).  In response, on September 13, 2011, PSEG initiated corrective action NOTF 20525076 to address the inspectors’ concern.  PSEG reviewed their CAP procedure guidance and determined that the condition was not a SCAQ; however, it warranted a root cause evaluation (RCE).  In February 2012, PSEG completed the RCE, “SRV Setpoint Drift Root Cause Evaluation” (70128407-010), to evaluate the longstanding SRV set point drift issues.  PSEG’s root cause analysis reviewed station preventative maintenance practices (rigging, storage, transportation, etc.), maintenance procedures, internal maintenance history, vendor maintenance history (including testing and inspection reports, replacement parts, and practices), industry OE, and the application of this OE at Hope Creek.  The root cause team evaluated the Target Rock SRV pilot valve design, manufacturing, and application.  The root cause team also reviewed effects of Extended Power Uprate, steam line vibration, and performance of each SRV by serial number.  In February 2012, the multi-disciplined PSEG root cause team determined that the Target Rock 2-stage SRV pilot valve design was incapable of satisfying the set point drift design requirements on a consistent basis.  PSEG’s corrective actions to prevent recurrence of the above root cause included plans to replace the currently installed Target Rock 2stage SRVs with a design that eliminates set point drift events exceeding +/-3 percent and improves SRV reliability.  Based on several engineering studies (including industry OE), PSEG’s Main Steam SRV Replacement Project (H-11-0009) recommended replacing the existing 2-stage Target Rock pilot valves with a SEBIM pilot operated design or with an upgraded Target Rock 3-stage pilot.  During the first quarter of 2014, PSEG made the decision to no longer pursue the SEBIM model replacement valve due to difficulties meeting Hope Creek specifications.  PSEG developed design change package (DCP) 80107006, “Safety Relief Valve (SRV) Replacement,” and had planned to install seven Target Rock 3-stage pilots in May 2015 (RF19).  However, in the early months of 2015 (just prior to RF19), PSEG decided to defer installing the new 3-stage Target Rock valves due to significant OE at Pilgrim Nuclear Power Station (including a Target Rock Part 21 report).  At the time of this inspection, PSEG tentatively planned to install one new 3-stage Target Rock pilot valve in the Fall 2016 RFO (RF20), contingent on the satisfactory acceptance testing results.  The inspectors noted that PSEG’s decisions that resulted in delays in replacing the existing 2-stage Target Rock pilot valves were appropriate, conservative, and aligned with the principle of not moving forward in the face of uncertainty.  From a historic perspective, leading up to RF19 in May 2015, the inspectors noted that PSEG’s aggregate actions to address SRV pilot valve set point drift issues were aligned with industry initiatives, appropriate, and commensurate with the safety significance.

On June 3, 2015, based on initial post-RF19 test reports, PSEG initiated corrective action NOTF 20692390 documenting that four SRVs failed their as-found set point tests.  Upon completion of the as-found testing on June 10, 2015, PSEG updated the NOTF documenting that 10 of 14 SRVs had failed their as-found set point tests.  On July 30, PSEG submitted a LER (LER 2015-004-00) for the SRVs set point failures. 

On  August 13, engineering completed two technical evaluations assessing the safety significance of the set point failures and determined that the set point drift did 

Maybe it bounded reactor vessel overpressure and other engineering design limits...but nobody talks about the reduction in that margin for the last decade. Remember an overpressure accident leading to cracking the vessel or bursting a large pipe would be a accident like no other meltdown in the western world.

not impact or challenge the ability of the SRVs to perform their function of relieving reactor vessel overpressure (see Section 4OA3.1).  On August 26, 2015, PSEG submitted a revision to the LER (LER 2015-004-01) to include the associated technical evaluations and impact on SRV operability.  Based on a review of the LERs, technical evaluations, and associated corrective action NOTF, the NRC resident inspectors identified that PSEG did not identify and/or evaluate an apparent adverse trend in as-found set point testing results (see table below).  Specifically, the resident inspectors noted that both the number and

Update May 11 2018: On the step increase of problem, the NRC says they brought it to the attention of Hope Creek. I brought it to the NRC before they knew it.    

magnitude of the RF19 failures represented a step increase compared to the previous four operating cycles (RF15 – RF18).  The resident inspectors discussed this observation with PSEG staff on several occasions and subsequently engaged PSEG senior managers and the PSEG engineering staff on a conference call on September 16, 2015.  This conference call included NRC Region I Division of Reactor Projects and Division of Reactor Safety managers and technical staff.  On November 5, 2015, following the resident inspectors’ additional engagement on the potential adverse trend, PSEG initiated two corrective action NOTFs to: (1) evaluate a possible trend in SRV set point drift magnitude and/or number of valves affected (NOTF 20709653), and (2) evaluate a potential correlation between the number of as-found set point failures and the time interval between SRV removal and SRV testing (NOTF 20709757).  Based on a review of corrective action NOTFs and NOS reports, the inspectors found no evidence that PSEG had identified and evaluated this potential trend prior to NRC engagement. 

RF15 04/09

RF16 10/10

RF17 04/12

RF18 10/13

RF19 05/15

SRV set point failures

6 6 6 5 10

Average set point drift (average of all 14 valves)

3.77% 3.64% 3.30% 2.34% 5.34%

Highest set point  pressure (psig)

1212 1199 1202 1192 1240

Number of valves above 1200 psig

2 0 1 0 5

Approximate average delay in days between SRV removal & SRV test

N/A N/A 20 25 50

Causal analysis Note: significance level  (SL)2 RCE completed in 02/12

SL3 ACE (70096933)

SL3 ACE (70115711)

SL2 WGE (70138789)

SL2 WGE (70161353)

SL4 no evaluation

On February 17, 2016, engineering completed two evaluations (70181904-010 and 70181906-010).  Engineering concluded that no definitive trend could be established based on a review of the as-found set point failures by cycle, SRV location, and set pressure group (i.e., valves set to lift at 1108 psig, 1120 psig, or 1130 psig), with one exception.  Engineering noted that the data showed that the 1108 psig set pressure group had an increasing trend in failures after cycle 14.  Engineering initiated an action item to perform a more detailed trend analysis of the 1108 psig group by specific pilot valve serial number and critical as-found dimensions (70181904-060).  In the evaluation, engineering concluded, that although the H1R19 test results show a significant increase in failures compared to H1R18, the failure rate does not represent an adverse trend and the single H1R19 data set is not sufficient to declare a trend.  The inspectors noted that engineering’s evaluation did not fully evaluate the possible trend in SRV set point drift magnitude (note from the table above that the average set point drift more than doubled when compared to the RF18 data).

Engineering reviewed the test data for RF17 through RF19 from the table above and concluded that an extended time interval between SRV removal from the plant until as found set point testing can adversely impact the results (number of set point failures).  Engineering initiated an action item to expedite SRV as-found testing in RF20 to further evaluate and assess the potential adverse trend in the RF19 failure rate (70181904-050).  The inspectors noted that the data supported engineering’s  conclusion regarding the impact of a time delay before testing.  Based on an OE review, the inspectors also noted that a Pilgrim Nuclear Power Station LER (LER 2004-001-00) attributed three 2-stage SRV pilot valve failures due to a significant delay in performing as-found testing.  The inspectors noted that the data suggests that significant delays prior to testing may result in more failures and a higher average set pressure.  However, the data also showed that the corrosion bonding phenomenon adversely impacted SRV pilot valve set pressures during the operating cycle as some valves failed even when tested within a few days of removal.  Thus, expediting the as-found testing would not eliminate corrosion bonding induced test failures; however, it may reduce the number and magnitude of the overall failures and result in as-found test results that more accurately reflect SRV pilot valve performance during the operating cycle.

The inspectors noted that engineering’s evaluation did not assess a potential correlation between time delays on the front end of the cycle and the failure rate (including magnitude).  Specifically, potential significant time delays between completing the required as-left +/- 1 percent testing and installing the pilot valves back into the plant and the potential impact on as-found failure rate at the end of the operating cycle.  The inspectors reviewed the data for RF18 and RF19, and concluded that there was no correlation on the front end.  

The inspectors noted that the significant step-change in SRV pilot valve as-found test results from the RF19 testing represented a CAQ.  Based on interviews and document reviews, the inspectors determined that PSEG had not identified the condition until prompted by the resident inspectors.  The inspectors determined that PSEG’s not identifying and evaluating the CAQ was a performance deficiency that was reasonably within PSEG’s ability to foresee and correct.  The inspectors evaluated this PSEG performance deficiency in accordance with IMC 0612, Appendix B, “Issue Screening,” and determined that the issue was minor.  This issue was minor because the inspectors did not identify any PSEG and/or NWS deficiency that may have contributed to an increased failure rate, nor any actions that PSEG should take to preclude recurrence prior to RF20.  

Based on a historical review of PSEG’s causal evaluations initiated to evaluate SRV set point drift failures (see table above), the inspectors concluded that PSEG had high confidence in their 2012 RCE, which may have led to not questioning the RF19 as-found test results.  Specifically, the inspectors noted that, following the RCE (completed in February 2012), PSEG initiated a significance level (SL) 2 work group evaluations (WGEs) after RF17 and after RF18, but initiated no causal evaluation after RF19.  In addition, the inspectors noted that on June 5, 2015, PSEG personnel did not provide adequate documentation supporting the CAP Management Review Committee (MRC) decision to screen NOTF 20692390 as SL4 with no associated evaluation, especially considering the number and magnitude of the as-found test failures.  The inspectors noted MRC’s decision, barring any documented justification, was not aligned with PSEG procedure LS-AA-120, “Issue Identification and Screening Process,” Attachment 2 (“Significance Level Guidance”) and Attachment 3 (“Guidance for Determining Evaluation Type”).  The inspectors determined that PSEG’s not following their CAP administrative procedure was a performance deficiency that was reasonably within PSEG’s ability to foresee and correct.  The inspectors evaluated this PSEG performance deficiency in accordance with IMC 0612, Appendix B, “Issue Screening,” and determined that the issue was minor.  Notwithstanding, the inspectors viewed the issue as another missed opportunity for PSEG to self-identify this trend.

Based on the RF19 as-found test results (all second lift tests were within 3 percent of the specified set point, with the average of 1.39 percent), engineering concluded that all ten SRV test failures were also due to the corrosion bonding phenomenon.  The inspectors noted that, based on PSEG and industry OE and RF19 test results, engineering’s conclusion was reasonable.  However, at the time of this inspection,PSEG had not performed internal inspections of any of the SRV pilots removed during RF19 to confirm their theory.  PSEG plans to perform inspections (including subsequent as-left set point testing) of all 14 SRV pilot valves commencing in June 2016 to support Hope Creek’s next RFO (RF20).  

Based on a review of as-left test documentation for all 14 SRVs pilot valves installed in RF17 and RF18 and a sample of SRV pilot assembly inspection records, the inspectors noted that NWS personnel maintained high-quality records that clearly documented the as-found condition, repairs and/or replaced components, the as-left condition, QA acceptance, and procedure compliance.

 2. The station is planning the replacement of the currently installed Target Rock two-stage SRVs with three-stage SRVs that are expected to eliminate setpoint drift events exceeding +/-3% and improve SRV reliability. The replacement is expected to begin in the next planned refueling outage, H1 R21, in the spring of 2018, pending resolution of open technical items with the valve manufacturer. The replacement will take place over several outages in order to replace all fourteen SRVs.