Monday, January 23, 2017

Junk plants Waterford and Arkansas Nuclear One: Startling Southerner Stupidity (SSS) During Plant Maintenance 

Entergy does this over and over again?
January 19, 2017
Findings Introduction.  The inspectors reviewed a self-revealing finding of preliminary low to moderate safety significance (White) and an associated apparent violation of Unit 2 Technical Specification 6.4.1.a for the failure to provide adequate lubrication to the inboard generator bearing so that the Unit 2 emergency diesel generator A would provide emergency power to safety equipment.  Specifically, two separate maintenance activities introduced errors that led to having inadequate oil to lubricate the bearing, and the bearing failed on September 16, 2016, during a 24-hour surveillance.
Description.  On September 16, 2016, 14 hours into a 24-hour surveillance at full load, the inboard generator bearing for the Unit 2 emergency diesel generator A failed, as evidenced by load swings, overheating, and sparking.  Operators secured the diesel, declared it inoperable, documented the failure in Condition Report CR-ANO-2-2016-03307, and complied with Unit 2 Technical Specification 3.8.1.1, “A.C. Sources,” Action B.  Maintenance personnel discovered significant damage to the bearing indicative of a lack of oil lubrication.  On September 28, 2016, the licensee shut Unit 2 down prior to the expiration of the technical specification action statement.  The licensee completed repairs and successfully tested the diesel on October 22, 2016, and restarted Unit 2 on October 27, 2016.
During a causal investigation for the lack of lubrication, the licensee identified that the oil level scribe mark on the sight glass was below the minimum level necessary to provide proper oil lubrication to the bearing.  The licensee concluded that on November 11,2014, while performing Work Order 356569, maintenance personnel had removed and inadvertently inverted the sight glass, which caused the scribe mark to be below the bottom of the bearing rollers (see diagram below).  With the sight glass inverted, the scribe mark was 3/8-inch lower than if it was in the correct orientation.  Post-failure measurements identified that the scribe mark was 5/8-inch below the correct position.  The licensee concluded that adequate oil was initially provided following the sight glass reinstallation.  Evidence to support this included multiple successful surveillance tests with no increase in vibrations, including a 24-hour surveillance on January 12, 2015, and having an oil sample from the bearing on June 22, 2016, with no indications of abnormal wear.     On June 22, 2016, maintenance personnel changed the oil in the inboard generator bearing after taking an oil sample in accordance with Work Order 52656389.  Maintenance personnel documented leaving the oil level within the procedural limits relative to the scribe mark.  Vibrations and system performance were normal during the post-maintenance runs and surveillance tests on June 26, 2016.  The licensee determined that the inboard generator bearing had not been leaking oil between the oil change on June 22, 2016, and the September 16, 2016, surveillance failure.  
The licensee concluded that the oil in the inboard generator bearing heated up slowly during the 24-hour surveillance.  After 14 hours, enough oil had vaporized within the bearing casing that the liquid oil level became inadequate to lubricate the bearing, resulting in bearing failure.  Therefore, the inspectors concluded that the emergency diesel generator could have failed approximately 14 hours after the start of a postulated event between June 26, 2016, and September 16, 2016.  The inspectors concluded that multiple causes led to the lack of oil lubrication.  The licensee failed to incorporate vendor manual instructions to properly set and verify the correct oil sight glass scribe mark into Work Order 356569 for maintenance performed in 2014.  The licensee failed to train maintenance personnel to adequately identify and control critical parameters during maintenance, specifically the effects of sight glass installation and maintenance on bearing lubrication.  The inspectors also noted that work instructions in Work Order 52656389 did not specify the correct amount of oil to add when replacing the oil, or else measure the amount of oil removed and ensure that a like amount of new oil was added to the bearing.   On October 11, 2016, following bearing reassembly, a system engineer checked the level of the new sight glass and identified that the scribe mark was too low again.  The licensee found that the new oil level had been marked on the bearing housing prior to assembly, but that the oil level was below the vendor-recommended level.  In response, the licensee planned and executed a work order to set the oil level in relation to the generator shaft centerline in accordance with the vendor recommendations.  The inspectors noted that the licensee had not yet implemented corrective actions to prevent recurrence of the problem in that licensee’s work plans failed to include verification of the oil level relative to the generator shaft centerline after reassembling the bearing.  The licensee subsequently corrected the sight glass position and developed training and improved work instructions to ensure that the bearing oil level would be correctly established.   After the diesel bearing failure, the licensee verified through walk downs that all sight glass marks and bearing oil levels were adequate for other safety-related rotating equipment.  The inspectors also independently reviewed bearing oil levels.


January 20, 2017
Description.  On August 12, 2016, operations personnel noted elevated chilled water outlet temperatures on essential chiller B.  Specifically, chilled water outlet temperatures reached between 45 and 46 degrees Fahrenheit, compared to the 42 degrees Fahrenheit maximum allowed temperature.  Due to the elevated temperatures, at 5:04 p.m., operations personnel declared essential chiller B inoperable.  Because essential chiller A was already inoperable due to a previous component failure, and there was no applicable technical specification action statement, the licensee entered Technical Specification 3.0.3, which required the licensee to begin shutting down the reactor within 1 hour.
The licensee placed essential chiller AB into service in place of essential chiller B, and at 6:02 p.m., declared essential chiller AB operable, which provided the plant with one operable train of chilled water.  This, in turn, allowed the licensee to exit Technical Specification 3.0.3.  The licensee remained in Technical Specification 3.7.12 and the associated 72-hour shutdown action statement until maintenance was completed on essential chiller A and it was declared operable at 11:00 p.m., providing the plant with two independent, operable trains of chilled water.
Following inspector questions, the licensee performed a calculation showing that adequate cooling capacity could be provided by the essential chillers with an outlet temperature of 46 degrees Fahrenheit.
In troubleshooting the event, the licensee found that the guide vane arm and actuator linkage for essential chiller B was assembled inappropriately.  The guide vane actuator was previously replaced during an essential chiller B outage on April 11, 2016; however, post-maintenance testing, an activity affecting quality to ensure that the safety-related chiller would perform satisfactorily in-service, did not discover the inappropriately assembled components.
Following the April 11, 2016, outage, the chiller was found repeatedly tripping on low refrigerant pressure.  However, the licensee mistakenly believed the failures were due to a faulty capacity control module.  The licensee replaced the capacity control module and returned essential chiller B to service on April 22, 2016.  Again, post-maintenance testing did not discover the inappropriate guide vane arm and actuator linkage.  The inappropriate guide vane and actuator arm linkage assembly went undetected until a large load from switchgear ventilation was placed on the chiller on August 12, 2016, resulting in its inoperability.
During extensive troubleshooting following the August 12, 2016, failure, the licensee inspected the guide vanes and found them inappropriately aligned.  The licensee corrected the guide vanes, restarted essential chiller, and declared it operable on September 3, 2016.
The guide vane and actuator arm assembly work is normally performed by a mechanical maintenance technician, but during the April 11, 2016, maintenance outage, the work was performed by an electrical technician.  The work was not verified by anyone from mechanical maintenance.

No comments: