Aaron Hyatt
07.26.2022
$800K Cooling Tower Fan Gearbox Failure
September 2021, a power generation facility experienced an unplanned failure on a cooling tower fan gearbox after approximately 49 minutes of operation. This was a repeat failure with a similar event occurring in January 2021 on the same gearbox. In both instances particulate and wear particles were suspected to have plugged an oil spray nozzle and relief valve, resulting in the gearbox failure.During the first failure in January, a low oil pressure alarm alerted Operations and allowed the fan to be secured. Inspections found that the tubing to the external pressure switch failed, resulting in the low-pressure alarm and oil being discharged to the environment. Following repairs to the tubing, an oil change, flush and inspection of the gearbox was completed, and it was deemed fit for service and tested satisfactorily.
During the September failure, the low oil pressure alarm actuated after the failure. Subsequent inspections found that the drive shaft had sheared causing the shaft driven oil pump to stop and actuate the low oil pressure alarm. Extensive damage to the gearbox, pinion bearings, shafts and gears required replacement of the gearbox with a warehouse spare.
Resulting Failure
After the low-pressure alarm was actuated in the control room, operations were able to investigate the cooling tower and found it showered in oil. The shaft was broken, and the high-speed pinion gear was glowing red with heat during inspection. Post-failure analysis showed that the high-speed pinion bearing went unlubricated for 44+ minutes, causing the shaft to overheat, shear, and separate.
Cost of Failure
The only replacement unit available had evidence of brinelling from extended periods of storage. The station was faced with an expedited timeline of repair, creation of new components, and testing to maintain proper uptime elements by governing bodies. Due to these expedited needs and costs, the station was faced with staggering costs estimated at $450,000 for labor, $300,000 for a replacement gearbox, and $50,000 for minor parts, contracted assistance, and associated fees.
Oil Analysis Data
An investigation was performed to identify events that may have led to this failure, including an inspection of the mechanisms that are put in place to provide predictive maintenance opportunities. Oil analysis is one of those mechanisms and was a component of this facilities maintenance program. The site began using Trico’s oil analysis services in 2019, however this unit was sampled on a semi-annual basis. Prior to the January 2021 failure, there had been two samples submitted that indicated critical issues with viscosity and increasing wear debris. Post-failure analysis and incorporation of historical lab data identified the trend below:
In addition to reduced viscosity, wear debris generation trends increased, particle counts skyrocketed, and physical wear was observed during ferrographic analysis:
Data-Driven Recommendations and Subsequent Action
After observing the first sample in 2019, and based on the reduced viscosity, recommendations were made by Trico to resample and confirm the difference in viscosity expected for an ISO 150 lubricant.The next sample frequency did not occur until a year later, at the end of October 2020. This sample confirmed the reduced viscosity and now identified a significant increase in wear debris presence. Per facility requirements, two subsequent “Critical” rated samples require an Action Request (AR) to be written.
During the requested maintenance period just prior to the failure in January 2021, debris was observed on the internal heater. An effort was made to remove as much debris through an inspection port utilizing lint free rags. Due to cold weather conditions, the technicians opted to pump oil back into the reservoir through a drain line which likely resulted in washing down debris hiding out in inaccessible areas in the gearbox to the false bottom where the oil pump suction resided. It is presumed that the debris was picked up by the pump, forced through internal passages to the nozzle and relief valve resulting in clogging, and causing excessive pressure resulting in a crack in the pressure switch tubing.
Root cause analysis identified that the viscosity dropped due to thermal degradation and eventual cracking of oil due to an oversized watt density heater. The excessive heat and cracking potentially caused molecule shearing, decreased viscosity, and polymerization, resulting in poor lubricant conditions and generating excessive particulate. Once this was identified, the heater was de-energized.
As 2021 moved on, and the repaired gearbox was in use, lubricant samples were submitted in April and July. Both samples indicate the continued presence of contamination, with ISO particle counts reaching >28/27/16, iron concentrations above 40 ppm, and observed wear debris in the form of red and black oxides. Both samples identified these as a third-tier or “Serious” rating by Trico. The recommendations were to identify the source of contamination and remove through an oil change, filtration, or confirmed equipment inspection. Unfortunately, maintenance was not performed and in September 2021, the unit experienced a similar pressure event, however operations was unable to detect the alarm resulting in the failure described.
Revised Process
The following gearbox failures prompted an investigation into internal processes and communications around predictive maintenance. As a result, this facility revamped their process of requiring two subsequent “Critical” samples to write an Action Request, allowing key indicators, such as viscosity changes to indicate action needed.In collaboration with Trico Corporation, increased attention to lubricant limits, equipment specifications, and recommended action have been followed. An increased focus on the lubricant chain of custody allows for the right product, in the right location, in the right amount, at the right time, and in the right condition by the right person.
New resources and tools have been instituted by the facility in the form of hand-held filter carts to allow for efficient maintenance activities that extend the life of the lubricant and equipment. An increased attention to results, combined with the tools to address contamination and efficiently extend lubricant and equipment life, can help prevent events such as the gearbox failures in this report and save available labor hours and maintenance costs.
Case for Improvement
Gaps in sampling frequency, reliance on waiting for two Critical samples, and inability to access operating equipment created the momentum leading up to the gearbox failures. Oil analysis and other predictive maintenance strategies are designed to not only identify the trends leading up to failure, but to put value back into the maintenance program by putting time back on the clock and reducing maintenance costs. Using this scenario and gearbox issues as a case study, a return-on-investment case can be made for flexible and expedited maintenance procedures and filter carts which has potential to generate a powerful savings outlook.
In evaluating oil analysis trends at the site, it is easy to see that the most common struggle for their program and equipment is that of particulate contamination (Figure 1). Contamination usually doesn’t cause immediate equipment issues, however as identified in the case study presented, it can gain momentum and expedite the failure of equipment and cause major downtime.
The facility purchased seven Trico hand-held filter carts of high and low viscosity specifications to address particulate issues. The upfront cost for these units is less than $20,000 and will seek to deploy an escalated maintenance strategy that will allow for filtering of operating equipment, removing downtime, saving on oil changes, and extending lubricant/equipment life. Trico and the client collaborated to present a case for return on investment of escalated maintenance with enhanced filtration.
Cost Assessment
- An estimated 20 oil changes occur at annually on sample points included in the oil analysis program with Trico (Figure 2)
- An oil change procedure was approximated to cost roughly $1,900 and include seven different employees to be involved throughout the process (Figure 2)
- Utilizing these factors, oil changes cost roughly $38,000 annually. At just 20 oil changes a year, it also leaves approximately 40 occurrences of elevated wear debris/particulate identified by oil analysis untreated. (Figure 2)
- Using oil analysis data, approximately 60 occurrences of wear debris and particulate would indicate a need for maintenance in 2021 (Figure 3)
- Filter cart purchases will aid in extending lubricant and equipment life, reduce the number of labor hours, and individuals necessary to perform maintenance actions. (Figure 3)
Return on Investment
Preventative maintenance programs draw their value from detecting potential unplanned events and by providing efficient maintenance actions that save time and money. The cooling tower gearbox situation described in this report stemmed from wear debris and particulate that built up overtime and went untreated. Contributing to this momentum was the reduced sampling frequency to build oil analysis trends and the inability to act on the elevated results when they were detected. The station has developed a plan for addressing these issues by adding filter carts and escalated maintenance opportunities for maintenance to occur. Figure 4 demonstrates the monetary value associated with the developed plan.
By end of year three, the applied strategy has reduced the number of oil changes necessary, saved on lubrication costs, and provided savings for the maintenance program. On top of the monetary benefits, this strategy:
- Preserves more labor hours previously consumed by oil changes
- Gives flexibility and attention to units based on oil analysis driven results
- Reduces lubricant consumption
- Provides a greater chance of identifying significant failures before they happen, allowing for planned maintenance strategies instead of reactionary measures (Figure 5, Cooling Tower Gearbox Failure)
