River Bend: Why are all these component failures and human errors in the same scram?

The NRC recently verified my analyses of this event. I was shockingly spot on correct...

The long delayed Mike Mulligan River Bend special inspection came out.

 I think River Bend needs mandatory testing...giving the plant the worst scram to see how the operators, procedures and all the equipment responds. Plant scram testing if you will. One should have occurred a first startup after the 2014 Christmas scram and the other near the outage to test all the operator and equipment at full decay heat load. Simulators are a amazing inventions...but they have their limitation. A simulator just doesn't carry this.   
 Originally published on 1/28/2016...

First pump published on Jan 29. Republished. 

I am happy they are going to be looking into vessel control.  

Jan 27: The Nuclear Regulatory Commission has begun a special inspection at the River Bend Station nuclear power plant to review circumstances surrounding an unplanned reactor shutdown that occurred on Dec. 25, 2014.
The plant, operated by Entergy Operations, Inc., is located in St. Francisville, La. 

The plant was operating at 85 percent capacity Christmas morning when an unplanned shutdown with complications occurred. 

Following the shutdown some problems were experienced with the plant's feedwater system, which supplies short-term cooling water to the reactor core, as well as several electrical circuit breakers. Operators took compensatory action to ensure the plant would remain in a safe shutdown mode. 

"The purpose of this special inspection is to better understand the circumstances surrounding the event, determine if there are any generic implications, and review the licensee's corrective actions to ensure that the cause of the event, including associated equipment problems have been effectively addressed," NRC Region IV Administrator Marc Dapas said. 

Several NRC inspectors will spend about a week on site evaluating the licensee's root cause analysis, maintenance of some plant systems and adequacy of corrective actions. An inspection report documenting the team's findings will be publicly available within 45 days of the end of the inspection.

At least a maintenance problem didn't take out another feed pump...it was a plant employee taking out all the demineralizers.

The should replace the feed reg valve and the feed system control systems.

Hmm, no RFP trips on vessel water swell? That is how all scrams should go. Seems the operators killed two MFP, then a plant employee killed the other MFP. This proves it is something in the training...it is not cycling the SRVs or swell from that. It is controllable. 

It is the slow acting FRV or leaking FRV from having the MFP on or all of them on.  

River Bend: Licensee Event Report50-458 / 2014-002-00

Reactor Scram Due to Average Power Range Monitor High-flux Signal Following a Malfunction of the Main Turbine Electrohydraulic System.

During the upcoming refueling outage in February 2015, an evaluation of a potential replacement of the obsolete card will b2 completed. A complete replacement with a digital EHC system is planned for the refueling outage to occur in early 2017. These actions are being tracked in the corrective action program.

Jan 5,

The River Bend NRC senior resident inspection gave me a call today. Another submarine brother. Most surprising out of the talk, he hasn't yet got boned up on the RB historical factors that caused this Christmas event like turbine control, vessel level management or the continued problems with feed water pumps in his LERs and inspection reports.

We talked about all the troubles with River Bend, ANO, Palisades, VY and pilgrim...how Entergy is always in the NRC's news at 11 pm tv station.

I told him, you know what irks us; like when they have component failures and vessel level management over and over again, and you can't get control of these guys. I think we and the NRC's Washington management victimizes he resident inspector...we don't give you the tools and power to control these guys.  

The agency is deciding if it going to be another River Bend special inspection...can I knock them into a special inspection?

What was cool, I had this page up and running on the internet before he called. We went though this article like my talking points and it was up in his computer? 

When I talked about the poor turbine grounding and the buildup of static electricity leading to erratic instrumentation readings and RPS trips...he kinda got quiet for a second. I think we are back to turbine grounding issue.

I do have to give great credit to the NRC...they do call me back.       

River Bend always has had a lot of NRC violation. A plant is not in a very healthy condition when they get a scram, then they get a lot of components breakdowns and employee errors in the scam. 

Lets see if I can interpret what happened?  

As far as the vessel repeatedly going out high on swell in every scram at 100%...I don't buy it for one second. They got a poorly optimized feed regulation valve system or it is poorly designed. They probably got the system half ass optimized so they would rather have a high level vessel feed water pump trip and scram, than a scare the pants of everyone low vessel level scram or low low level. The feedwater regulation valves and control system don't operate quickly enough to follow a down-power transient OR scram...they over feed the vessel. The feed water valves close too slowly!

There will be hell to pay if a huge chunk of water ends up going down the main steam lines...there will be a terrific water hammer outside the design of the plant. This could lead to bigger problems with cooling the core down too fast. Excessively feeding the vessel and cooling down the core adds a lot of reactivity.   

Power Reactor Event Number: 50704

Facility: RIVER BEND
Region: 4 State: LA
Unit: [1] [ ] [ ]
RX Type: [1] GE-6
NRC Notified By: DANIEL PIPKIN
HQ OPS Officer: DANIEL MILLS Notification Date: 12/25/2014
Notification Time: 12:41 [ET]
Event Date: 12/25/2014
Event Time: 08:37 [CST]
Last Update Date: 12/25/2014
Emergency Class: NON EMERGENCY
10 CFR Section:
50.72(b)(2)(iv)(B) - RPS ACTUATION - CRITICAL
50.72(b)(3)(iv)(A) - VALID SPECIF SYS ACTUATION
Person (Organization):
THOMAS FARNHOLTZ (R4DO)
Unit SCRAM Code RX CRIT Initial PWR Initial RX Mode Current PWR Current RX Mode
1 A/R Y 85 Power Operation 0 Hot Shutdown
Event Text
AUTOMATIC REACTOR SCRAM 

"At 0837 [CST] on 12/25/14, a loss of Reactor Protection System (RPS) 'B' occurred which resulted in a Division 2 RPS half SCRAM. This occurred concurrent with a Division 1 RPS half SCRAM which had been inserted for LCO 3.3.1.1 Action 'A' due to issues with the #2 turbine control valve RPS logic on 12/23/14. This resulted in a full RPS actuation and Reactor SCRAM. During the SCRAM, a reactor water Level '8' occurred which tripped the running reactor feed pump. Reactor water level peaked at 56 inches. This Level '8' is under investigation. Once reactor water level lowered below 51 inches the Level '8' signal was reset, and the team attempted to start the 'C' reactor feed water pump. 

"The 'C' reactor feed pump failed to start upon attempt. The 'A' reactor feed pump was then started successfully. The startup feed regulating valve failed to open in automatic or manual mode, resulting in an RPV Level '3' signal (lowering to 8.1 inches). The operators manually aligned the 'C' feed water regulating valve and restored reactor water level to normal band. The plant is stable in Mode 3 pending investigation." 

1) Had some kind of warning light or maybe a half scram on the "#2 turbine control valve RPS logic on 12/23/14" indicating some instrumentation components was failing (the issues).
 

2)I&C put in a Div 1 and A RPS in a half scam...must have been doing some testing or repair. This assures on a "B" trip you will get a full scram. Usually on a half a scram, it is not enough to get a scram.
 

3) Then they got a "B" RPS trip on a #1 turbine control valve RPS logic.

4) The reactor lever went high leading to a reactor water lever "8' trip. This tripped all the feed pumps. The reactor water level was increasing out of control high. They didn't want water going down their steam line. This is unprofessional not being able to control reactor water level.
 

5) The water decreased to 51 inches thereby resetting the level "8" trip.
 

6) They tried restating the "C" feed water pump...it failed to restart.

7) They started the "A" reactor feed pump.
 

8) The start-up feed regulating valve failed to operate in auto or manual...vessel going down to a RPV level "3".
 

So something buggy was going wrong with the turbine control valve RPS logic, they got a unexpected scram a day later while trouble shooting and testing on the other side for some unknown reason, they lost control of water level, the "C" reactor feed pump wouldn't start for some unknown reason and then the start-up feed regulating valve didn't operate for some unknown reason. There is a lot of gear failing here.

A professional nuclear plant staff aims for no gear or components failing in a scram...it has the opportunity to confusing the licences operators. A professional nuclear plant staff never loses control of the reactor vessel.

I would be surprised if the I&C guys got mixed on the #1 turbine control valve RPS logic and the "B" RPS circuit. In other words, a #1 turbine trip goes on to tripping the "B" RPS side...this 1/B and 2/A wording is very confusing. 

Is this the problem? Agastats and relays not being replaced on a bad test...agastats and relays aging out?

The theme with Palisades and Pilgrim...Entergy having a philosophy of operating their regular and safety equipment to run to failure. Palisades according to the NRC, has a problem with just meeting the minimum intent of the federal rules and this got their plant and their employees in so much trouble. 

Licensee Event Report 50-458 / 2014-003-00

On June 10, 2014, with the plant operating at 100 percent power, technicians performing a scheduled surveillance test found that one instrument channel in the reactor protection system failed its time response acceptance criterion. This was the second of two such tests that failed in similar fashion. Since it is conceivable that the second tested channel was out of specifications at the time the first channel was tested, this condition caused independent redundant channels in the same trip system to be inoperable at the same time. The actions required by the applicable Limiting Condition for Operation were not taken since the operators were not aware of the latent condition at the time of the first surveillance test failure. An engineering evaluation of this condition was performed, and the RPS system was declared operable with compensatory measures. Until this issue is resolved, the frequency of the calibration tests in the channels with Agastat relays has been increased to once per year. This condition is reportable in accordance with 1OCFR50.73(a)(2)(i) (b) as operations prohibited by Technical Specifications, as well as 1OCFR50.73(a)(2)(vii), a potential common-cause inoperability of independent trip channels. Due to the design redundancy of the independent channels of the RPS system, this condition would likely have not prevented the system from performing its safety function. Had an actual full MSIV isolation occurred with the channel response times in their as-found condition, the reactor scram signal would likely have still occurred within the specified instrument response time.
 IMMEDIATE ACTIONS

In the calibrations performed in 2010, a degrading trend in the response times was noted in the four channels containing the Agastat relays (**94**). In the 2010 tests, the response time of each of the four channels was 89 milliseconds. The as-found response times found in the recent tests ranged from 90 to 102 milliseconds. In each case, the Agastat relay was replaced and the response time was then verified to be within specifications. The response times for the channels with no Agastat relays ranged from 41 to 51 milliseconds. 

Here below is another electronic protection device failure. Again two failures in a row in one event. They are not taking care of the plant. First, the huge C feed pump motor shorts and burns, then the closest breaker to the pump (feed water pump breaker) fails to protect the bus by tripping. It is only the next breaker in the line that protects the rest of the plant who trips and works.  

These kinds of failures have been known to catch on fire the whole switchroom...loaded with supplies to other important components. It is a nasty fire in a small area..

Lets get this straight...they were in a start-up with everything new or repaired.

The "C" feedwater was running.

They started the "B"...it shorted. 

The feeder breaker failed...the protection breaker disconnected the whole bus instead of just the shorted main feed pump. As the B and C came from the same bus...the C loss power.

What happened to A feedwater pump, why didn't they start that guy up? Bet you it was tagged out. So all feedwater pumps were unavailable. Hmm,they lost the main condenser and had to shut the main steam isolation valve. You see how this goes on these kinds of accidents, the choice of what system can feed the vessel gets quickly narrowed to almost nothing. They didn't use the A feedwater pump because of losing the Mcond and shutting the MSIVs.

Licensee Event Report 50-458 / 2012-003-00

At the time of the event, the "C" reactor feedwater pump was in service. When the operator started the "B" feedwater pump, an electrical fault occurred at the pump motor. The lockout relay on the pump's feeder breaker failed to trip the breaker, and the main supply breaker to the "B" 13.8kV switchgear tripped to clear the fault. This caused the loss of power to the "C" pump, as well as switchgear supplying the circulating water system and the normal service water system.

The lockout relay installed on the breaker for the "B" feedwater pump is a General Electric HEA 61. The analysis of this event found that the lockout relay failed to operate as designed due to age-related mechanical binding and a possible coil failure. This condition resulted from an inadequate preventative maintenance program for the relays and a design issue with the trip plate.

Basically the vendor was corrupt and incompetent, Entergy had astonishing poor oversite of this vendor maintenance activity. I'll bet you Entergy secretly ok'd the too large lugs.   

The inspection of the terminal box on the "B" feedwater pump determined that fault occurred due to an inadequately crimped terminal lug on one of the three current transformers. The motor (**MO**) had been rewound by a vendor in 2008. When the motor was returned, new lugs were resupplied by the vendor to be installed onsite. The lugs were installed by a local vendor. The investigation found that the lugs were too large for the application. Additionally, the crimping tool used for the installation did not fully compress the lugs, leaving an inadequately bonded connection. 

That is the "run to failure ideology...the coal plants taught them how to do this. It is unbelievable how often they allow these relays and agastates to fail by end of life through an intentionally inadequate preventative maintenance program. Usually the agastates and relays are obsolete and no longer made for the repair parts stream. They get vendors to reverse engineer these obsolete parts and they manufacturer them on their own. Because of this and it is such a inconvenienced to disrupt an outage...they chose to bet the ranch on not failing...close their eyes and wait till one fails.    

The lockout relay installed on the breaker for the "B" feedwater pump is a General Electric HEA 61. The analysis of this event found that the lockout relay failed to operate as designed due to age-related mechanical binding and a possible coil failure. This condition resulted from an inadequate preventative maintenance program for the relays and a design issue with the trip plate.

Oh, the "C" feed pump "failed to start on call" on the current LER starting this off, below (Licensee Event Report 50-458/2011-003-00) the "B" and "C" failed in some manner, in the above the "C" shorted out leading to isolating the whole bus and losing the Mcond and MSIVs. They are really bad with all these feed pump failures in such a short time.
 

Here is the Augmented Inspection below...remember rework problems. What causes rework problems?

RIVER BEND STATION - NRC AUGMENTED INSPECTION TEAM REPORT

05000458/2012009

During a reactor startup on May 24, 2012, operators at River Bend Station initiated a manual scram of the reactor from 33 percent reactor power. The reactor scram was the result of a loss of feedwater, circulating water, and nonsafety-related cooling water caused by an electrical fault associated with a main feedwater pump motor. The fault was not isolated by the motor feeder breaker due to a failed relay, resulting in the trip of the supply breaker for the 13.8 kV nonsafetyrelated electrical bus. Because of a previous cable failure and fire on May 21, 2012, all operating circulating water pumps and nonsafety-related service water pumps were powered through this supply breaker. The loss of the running pumps resulted in the loss of condenser vacuum and cooling water to turbine building and safety-related loads. Both divisions of safetyrelated standby service water started and restored cooling to the safety-related loads.

Licensee Event Report 50-458/2011-003-00

On December 23, 2011, at approximately 6:10 a.m. CST, the main turbine (**TRB**) tripped unexpectedly, resulting in a reactor scram. The plant was stable at 100 percent power at the time of the event, and no safety-related systems were out of service. Operators implemented the appropriate response procedures, and began to stabilize reactor vessel pressure and water level.

The rapid closure of the turbine control valves caused a rise in reactor pressure that actuated at least fifteen of sixteen main steam safety relief valves (SRVs). The initial shrink in reactor water level accompanying the reactor scram caused a Level 3 alarm, with water level reaching a low point of -0.1 inches approximately 15 seconds after the scram (Level 3 is 9.7 inches). The main feedwater control system responded, and the subsequent increase in reactor water level caused a Level 8 trip of all three reactor feedwater pumps (**PMP**) approximately three minutes into the event.

As reactor water level lowered back through the normal operating range, operators attempted to restart a feedwater pump, but component malfunctions were encountered on "B" and "C" pumps. The reactor core isolation cooling (RCIC) (BN) system was manually actuated approximately nine minutes after the scram and injected water into the reactor for approximately two minutes. The "A" feedwater pump was restored to service approximately one minute after RCIC was initiated.

This event is being reported in accordance with 10 CFR 50.73(a)(2)(iv)(A) as a condition that resulted in the automatic actuation of the reactor protection system (RPS) (JC).

In the immediate notification performed in accordance with 10 CFR 50.72, it was reported that the reactor vessel Level 3 condition caused the actuation of primary containment isolation valves in the suppression pool cooling system. Those valves were already closed at the time of this event due to the system being out of service.

Here below(Licensee Event Report 50-458/2011-003-00) is what caused the turbine trip below...more vendor incompetence and extremely poor oversight by energy. The theory goes, the philosophy is to reduced the expensive full time educated and experienced River Bend employees...use the cheaper contractor to replace the highly specialized and skilled Entergy employees. Maybe the petroleum well drilling and platform job are sopping up all the good employees?    

Bottom line, the installation and operation the turbine shaft ground system was repeatedly botched...they observed it not wearing properly and they failed to document this condition. This is what caused the turbine trip.  

This all from the beginning is called rework problems...a job is done and is later discovered the job was botched. A bad plant has a lot of rework problems, the rework can go to 50% to 70% of all jobs. It is extraordinarily wasteful and millions of dollars of value is stolen from the stock price and value. 

Main turbine trip 

The trip signal to the main turbine originated in the electro-hydraulic control (EHC) system, and caused a fast closure of all turbine control valves. Analysis of transient data found that both the primary and back-up turbine speed signals became erratic and indicated an overspeed condition. Within approximately one second, all four turbine control valves were commanded to go fully closed. The RPS system responded to the fast closure signals from the turbine control valves, initiating a reactor scram.

The cause of the turbine trip was a spurious backup over-speed trip. An electrical discharge from the turbine shaft to the vicinity of the EHC turbine speed pickup probes generated electromagnetic pulses that influenced the speed probes, which act on magnetic flux associated with a toothed wheel coupled to the turbine shaft. Over-speed signals in at least 2 out of 3 channels caused a turbine trip signal. The cause of the electrical discharge was due to a failure of the shaft grounding system.

The turbine shaft grounding system was modified in 2004 to add a new brush at the midstandard location (between the high-pressure turbine inboard bearing and the thrust bearing). There are four brushes assigned to the turbine shaft, three of which provide ground protection. The mid-standard brush was removed as part of troubleshooting the turbine trip and it was found to have very little wear for the time in service. An inspection conducted by both internal and external technicians concluded that the brush was not providing protection, given the level of wear observed. The brush is designed to pivot with bristle wear to maintain shaft contact. Since less wear was observed than expected, it is concluded the brush wore until the maximum range of the brush pivot was achieved, after which the brush lost contact with the shaft due to making hard contact within the indicator housing.

This investigation also found that the mounting bracket for that brush was improperly fabricated, such that the angle between the brush head and the shaft was not correct. Following this forced outage, the as-left reading on the brush wear indicator is about halfway between "replace" and "new." This allows adequate brush movement and shaft contact. Actions will be taken in the next refueling outage to correct the angle on this bracket to make it read accurately. 

There were significant contributing factors in this event:

• The grounding brush at turbine bearing no. 2 was installed as part of a modification to add a new grounding point. At that time, the preventative maintenance (PM) task for measurement of shaft voltage should have been revised to include shaft voltage measurements from either the new grounding brush or the shaft voltage monitoring. The PM was not revised. Increased shaft voltage would indicate that the shaft grounding brush was not working properly.

• The post-modification testing following installation of the new brush in 2004 was not performed properly. An improperly fabricated mounting bracket built for that modification apparently does not allow the wear indicator to accurately show the "new" indication for a new brush head.

• Maintenance and Engineering personnel recognized that the wear indicator did not accurately measure actual brush wear, but did not document the deficiency in the corrective action program.

The limit switches were out of adjustment with the failure to start the "C" feed pump...a classic rework issue. The botched limit switch job and then retesting cause the main feed pump not to start. A plant has hundreds of thousands to millions of components and relays...if they had a big problem with rework problems it could cause a lot of issue in the control room with degraded and broken components showing up in accidents or scrams. A professional plant staff bets their careers that all components operated as designed.
Remember the staff of a nuclear plant consist of 800 to a 1000 employees. They have a army of willing and high paid contractors and vendors....there really is no excuse here. It is usually plant staff and management disorganization that causes this. It is institutional disorder on a huge level.     

The operator first attempted to start the "C" feedwater pump. Part of the start sequence is the opening of the minimum flow valve, initiated by depressing the pump "start" button. When the pump start sequence was initiated, the operator observed the indication for the minimum flow valve start to travel from closed position to an intermediate position, instead of going fully open. The pump start circuitry is electrically interlocked with that valve position indication, so this failure prevented the pump from starting. Troubleshooting found that the valve was actually opening as commanded, but the limit switches were out of adjustment, preventing the fully open indication from being applied to the start circuitry.

Testing should have picked that up...not discovered in a troublesome scram or accident. You going to discover one day a accident that scares the pants off the staff and public...finding these kinds of problem in just a scram means you will find a very uncomfortably amount of degraded and broken components in a bad accident. It could end in it being more than just a embarrassment to the staff and management...the public and politicians just may lose faith in you.  

The operator then attempted to start the "B" feedwater pump. One of the actions involved in starting a feedwater pump is to verify that the auxiliary lubricating oil pumps on the pump and the gearbox are operating correctly. Upon initiation of the pump start, the operator observed that the gearbox auxiliary oil pump was cycling "on" then "off," which is abnormal. Troubleshooting found that a pressure regulator in that lube oil system was out of adjustment. This issue was corrected prior to plant restart.

This is interesting because it implicates the turbine grounding system and brushes...static electricity builds up and discharge across the HP turbine

My professional opinion is the turbine and the turbine control system is junk. You had a lot trips over this. It needs to be completely refurbished or replaced. Who does the contractor servicing of your turbine?  

Automatic Reactor Scram During Main Turbine Control Valve Testing Due to Control System Malfunction

Of great note:

Final Precursor Analysis Sequence Precursor Program -- Office of Nuclear Regulatory Research

River Bend Station Automatic Reactor Trip Due to Loss of Non-Vital 120V Instrument Bus

Event Date 12/10/2004 LER: 458/04-005-01 CCDP1 =2.7 x 10-5

May 16, 2006

This is a example with Entergy-River Bend again having a serous transient and scram based on a preventable fault on their transmission system. We see again lots of equipment failures leading to confusing the operators in the very  busy portion of dealing with the scram.

Particular here in the special inspection, the NRC has implicated River Bend with seriously mismanaging vessel level control...banging level up and down all over the place...mis-communication with coordinating operating the safety relief valve and managing the vessel control.  Basically simulator training is poor with managing reactor vessel level control and they have big troubles with simulator fidelity issues in modeling the vessel swell in a scram and the operation SRVs. They were training all the operators on a simulator model on the behavior of vessel level that didn't match what vessel level actually did in a scram. 

Again and again, why does River Bend have so many broken components creating scrams, then once in a scram, why are there so many more broken components showing up and what about all the human errors?

At least a Feed pump didn't burn up or fail upon call. That seems to happen much later.  

You can size up how competent a nuclear plant crew is and how effective simulator training is by how they manage vessel water. If is bangs around past high level trips and whips down to below the low vessel level trips...it means these guys are poorly trained and the crews  won't be effective in a riskfull accident. It is a indicator of safety culture. It test crew coordination and communication...this is a prerequisite for safety. 

If a regulator allows the crew to bang around nuclear vessel level over and over...level mis-management...then this is a indication of the ineffectiveness of a regulator to control and limit chaos in the control room of a nuclear power plant. 

Now if the NRC has control room criticism like in the below inspection report about inadequate vessel level management, then the site in a few years have another vessel level management problem of bigger proportion, has a continuing problem with vessel level management...then this in on the NRC. The agency isn't carefully observing crew vessel level management especially in the simulator for a prolonged period of time. The agency and the ROP lacks the stamina and the attention in detail to ascertain all the crews are competent in managing vessel control after finding a initial serous shortcoming in vessel level management.

March 2007 IR: (banging around uncontrollably vessel water level); "The team reviewed plant operating parameters and the associated time line elements and determined that reactor vessel water level had gone outside the established level band at least 6 times during the 53 minutes that the main steam isolation valves were closed."

This is what you call a NRC institutional breakdown or failure. The agency aren't self monitoring their actions or violations of the licensees...keeping tabs of the ROP and NRC bureaucratic structure to see if it creates a positive change in the licensee.

If the agency interaction with a licensee doesn't cause deep and positive long term behaviors and drive positive vessel level management changes in a licensee on the first swipe...if the agency isn't self aware of the effectiveness of their interactions, then they aren't in control of a bad actor licensee. Their ROP isn't effective on what we all want out of the nuclear power industry.             

First swipe at trying to fix vessel level management: February 10, 2005: SUBJECT: RIVER BEND STATION - SPECIAL INSPECTION REPORT 05000458/2004012

Reactor Coolant System Level 2 Actuation 

At approximately 7:35 a.m., the operators closed the outboard MSIVs due to lowering main condenser vacuum. At 7:58 a.m., the inboard MISVs were closed, also due to lowering main condenser vacuum. This was an anticipatory action taken with at least an 8.5-inch vacuum remaining in the main condenser. Reactor pressure was being controlled by the SRV low-low set function. At 8:04 a.m. level control was assigned to the At The Controls (ATC) operator who was working on restoring feedwater. At 8:10 a.m., SRV F0551D opened on low-low set and the control switch was taken to the OPEN position to bring pressure to the low end of the pressure band; the band was set at 500-1090 psig. At this point, through licensed operator interviews, the inspectors determined that it was not clear who had pressure control or which operator placed the SRV control switch in the OPEN position. The root cause investigation stated that the ATC operator had responsibility for both level and pressure control. This is a difference between the team’s investigation and the licensee’s root cause.

At 8:14 a.m. a Level 3 was reached. The SRV remained open until 8:16 a.m. when the ATC operator reported that the reactor pressure vessel (RPV) level was decreasing and approaching the Level 2 setpoint. An operator was instructed to close the open SRV, while another operator was directed to inject with high pressure core spray. The closure of the SRV promptly dropped level to -52 inches, which exceeded the Level 2 setpoint of -43 inches. The only remaining recirculation pump tripped. Feedwater Pump FWS-P1C was started. Within one minute, RPV level was restored above the Level 2 setpoint and restored above the Level 3 setpoint within 3 minutes. During this time, the CRS was observing the ERIS display for RPV level and did not notice any change because of the power loss to that system.

The Level 2 that was experienced by the operators was not expected, nor should it have occurred. There were at least three contributing causes for the Level 2. First, the ATC operator should not have had responsibility for both level and pressure control. Second, communication between the ATC operator and the SRV operator was not sufficient to limit unexpected RPV level fluctuations. At the time of the incident, all MSIVs were closed and the ATC operator was in the process of restarting a feedwater pump.

According to the RBS's pressure control strategy, pressure should have been controlled via the main steam line drains or through cycling SRVs. If SRV cycling is to be used, then close coordination between the ATC operator and the SRV operator should take place to limit unexpected level fluctuations. Contrary to this, an operator placed the SRV into the OPEN position and walked away from the control board without sufficient coordination with the ATC operator. The third contributing cause was that the CRS relied on the ERIS display that was not functioning properly. The CRS was cognizant that the ERIS system was suspect, but continued to rely on the system output. According to the ERIS display at the time of the Level 2, level was not changing.

...10 CFR 55.46.c states in part, “A plant-referenced simulator used for the administration of the operating test or to meet experience requirements . . . must demonstrate expected plant response to operator input and to normal, transient, and accident conditions to which the simulator has been designed to respond . . . .” RBS experienced two reactor scrams (August 15 and October 1, 2004) in which actual plant SRV manipulations caused shrink, swell, and level indications that were different than what was modeled in the simulator. After some investigation by the licensee, it was determined that level variations in the simulator were 6-8 inches different than in the actual plant. Considering that RPV level is normally maintained between Level 8 (51 inches) and Level 3 (9.7 inches), 6-8 inches constitutes approximately a 15-20 percent difference than actual plant condition. Coupled with the fact that most of the operators on shift during the events had never actually manipulated SRVs in the plant, this simulator fidelity deficiency could have an impact on operator performance. This issue was documented in the licensee’s corrective action program in Condition Report CR-RBS-2004-2334. This violation is of very low safety significance because it did not involve an exam or operating test, but did involve a simulator fidelity issue which impacted operator actions and resulted in...

Second swipe: March 1, 2007 SUBJECT: RIVER BEND STATION - NRC SPECIAL INSPECTION REPORT 05000458/2006013

3.1.3 Safety/Relief Valve Operation Discussion: On October 19, 2006, at about 5:59 pm, an inadvertent main steam isolation occurred on low reactor pressure caused by high pressure core spray injection. With the main steam isolation valves closed, reactor pressure began to increase as the large volume of cooler water injected by the high pressure core spray system expanded. Approximately 7 minutes later, with reactor pressure at 1090 psig, operators opened a safety/relief valve to control pressure below the automatic relief setpoint. The Table 3.1-2 documents the valve manipulations that occurred during the event. 

Abnormal Operating Procedure AOP-0001, OSP-0053, Attachment 1B, "Post Scram Pressure Control Strategies," Revision 5, states, in part,

"1.2 Post-Scram Pressure Control for an MSIV Isolation.

     "1.2.1 IF only the inboard MSIVs close due to a loss             of air to containment, THEN perform the                   following:

    "1. Take manual control of the inboard MSIVs by             taking the control switch of each valve to             CLOSE.

    "2. Utilize available steam drains to control               pressure.

    "3. IF required, THEN augment pressure control with         SRVs. Each SRV cycle should beclosely           coordinated with the at-the-controls operator.

    "1.2.2. For a full MSIV isolation, perform the                       following:

    "1. Verify SRVs are cycling automatically to                control pressure.

    "2. IF automatic SRV cycling is preventing the             level control operator from controlling RPV             water level in the required band, THEN perform         one of the following:

• Closely coordinate with the level control operator to manually operate SRVs as required to control pressure in the prescribe pressure band, without driving level outside the prescribed level band.

• Transition level control from the Feed and Condensate system to the RCIC system.

• Run RCIC either directly for level control, or in pressure control lineup (maximized)."

However, following the main steam line isolation, the safety/relief valves never operated in automatic. Therefore, operators did not verify that they were cycling in automatic, nor could they observe that the automatic function was preventing the level-control operator from controlling reactor pressure vessel level in the required band. In addition, manual control of the safety/relief valves drove level out of the required band on multiple occasions during the event. Licensed operators and plant management stated that operators knew that under the conditions that existed they could not properly control level if the safety/relief valves were cycling in automatic and that they had been trained to operate the safety/relief valves manually under these conditions. This expectation was supported by operations management. Additionally, plant management stated that this procedure was not a requirement and was in conflict with the bases of the emergency operating procedures. The team reviewed Section 1, “Purpose,” of Attachment 1B and noted that Step 1.3 stated:

“The “Continuous Use” designation of this procedure is intended to apply to the Hard Card attachments only. The Strategy attachments and procedure body are informational in nature and do not provide step by step procedural guidance.

Section 3, “Strategies,” Step 1.3, stated:

“Strategy attachments are provided in this procedure for those activities which do not lend themselves to step by step instructions due to the varying impact on these activities by differing plant conditions for different transients.”

Additionally, in Section 4, “Hard Cards,” Step 4.7.1 stated:

“Attachments 1A, 1B, and 1C are strategies, not Hard Cards.”

The team noted that these procedure statements should be reviewed in light of the definitions given in River Bend Nuclear Procedure RBNP-001, Revision 25, “Control and Use of RBS Procedures.” Section 3, “Definitions,” Step 3.4 defines the level of use of plant procedures, indicating that there are three categories of procedure: Continuous Use, Reference Use, and Informational Use.”

Informational use procedures were defined as procedures frequently performed or not complex in which the activity could be accomplished from memory and within the skill of qualified individuals. While these procedures are not required to be available at the work location, they are still expected to be followed. The team also reviewed the bases for EOP-0001, Step R P-3, “Stabilize RPV Pressure Below 1090 psig.” One portion of this document suggested that Safety/relief valves should generally be opened manually. However, the bases discussed many exceptions to this general statement. Additionally, the document stated:

“. . .the adequacy of steps taken to stabilize RPV pressure must be judged by the effect of any continuing pressure variations on RPV water level. . . “

The team reviewed plant operating parameters and the associated time line elements and determined that reactor vessel water level had gone outside the established level band at least 6 times during the 53 minutes that the main steam isolation valves were closed. This fact, combined with an evaluation of the data, shown in Table 3.1-2 indicated to the team that operators may have been attempting to control pressure at specific points without regard for reactor water level at the time. The team concluded that the failure of licensed operators to permit the safety/relief valves to cycle in automatic and to manually operate Safety/relief valves without driving level outside the prescribed level band, as required by abnormal operating procedures, was a violation.

Bottom line, from the 2007 vessel management problem till right up to today, River Bend has has unabated serious and poor vessel water level management issues uncontrolled by the