Items tagged with: issue
Perhaps the single most complex, insidious, and long-lasting mechanical problem in the history of commercial aviation was the mysterious rudder issue that plagued the Boeing 737 throughout the 1990s.…
Article word count: 2662
HN Discussion: https://news.ycombinator.com/item?id=19389983
Posted by IFR (karma: 153)
Post stats: Points: 278 - Comments: 88 - 2019-03-14T15:19:43Z
#HackerNews #1990s #737 #boeing #issue #plagued #rudder #that #the #throughout
Perhaps the single most complex, insidious, and long-lasting mechanical problem in the history of commercial aviation was the mysterious rudder issue that plagued the Boeing 737 throughout the 1990s. Although it had long been rumoured to exist, the defect was suddenly thrust into the spotlight when United Airlines flight 585 crashed on approach to Colorado Springs on the third of March, 1991, killing all 25 people on board. The crash resulted in the longest investigation in NTSB history, years of arduous litigation, and a battle with Boeing over the safety of its most popular plane. Flight 585 proved to be hardly alone; over the subsequent years, more planes crashed due to the same rudder defect, including USAir flight 427, which killed 132 people when it suddenly rolled over and crashed on approach to Pittsburgh, Pennsylvania in 1994. As it turned out, these were but two of the most serious of hundreds of incidents involving the rudder on the Boeing 737. This is the story of the origin of the defect, its consequences, and Boeing’s efforts to cover it up. Images sourced from The Seattle Times, the NTSB, Boeing, Tails Through Time, the Colorado Springs Gazette, The Times of India, Wikipedia, TribLIVE, The Flight 427 Air Disaster Support League, and Forbes. Video clips courtesy of Cineflix and the Weather Channel. Special thanks to the Seattle Times for its series of articles on the subject in 1996, which brought to light many of the details referenced here.
The long and complex story begins with the original design of the Boeing 737 in the 1960s. Unlike other Boeing aircraft, the 737 had only one rudder and only one Power Control Unit (PCU), a device which translates pilot inputs into motion of the rudder. This gave it less redundancy than other models, but Boeing got the design certified by showing that a failure of the PCU was so unlikely that it was a non-issue. Nevertheless, reports of mostly minor uncommanded rudder deflections on the 737 began almost immediately. Boeing blamed the yaw damper, a device that constantly adjusts the rudder a few degrees in either direction to counteract an aircraft’s natural tendency to stray off a straight course. These issues were frequently “fixed” by replacing or repairing parts of the yaw damper.
Pilots continued to report problems with the rudder on the 737, however. No one was keeping track, but there are estimated to be several hundred reported incidents, including some that were very serious. Among these were four reported in-flight upsets on Frontier Airlines 737s, including two on the same airplane in 1973. In the two latter cases, both over the Dakotas, flight attendants had been injured by the sudden rolling movement, but the pilots were able to regain control. Boeing again responded by modifying the yaw damper; no attention was paid to the PCU, despite the fact that the yaw damper was limited from moving the rudder far enough to cause the sort of major upsets reported by the Frontier crews.
It is now known that the problem was probably with the PCU all along. To understand the crashes that occurred later, it is first necessary to summarize the specific vulnerability that was causing these incidents. The most important component of the PCU is a small device called the dual servo valve, shown in the above diagram from the Seattle Times. In simple terms, the valve consists of an inner and outer slide which, when lined up in different ways, can direct the flow of hydraulic fluid to push the rudder left or right. (A more detailed explanation can be seen in the diagram.) A spring and an end cap are supposed to prevent the slides from moving too far.
However, if the spring and the end cap were slightly misaligned, the slides could extend beyond their design limit. This would cause a “rudder hardover,” where the rudder suddenly moves to its maximum deflection, resulting in extremely serious control problems. But this was only one part of the issue. When the slides in the dual servo valve extended too far, it would also misalign the openings on the outer slide and the valve body, allowing hydraulic fluid to move into the wrong channels and causing a rare phenomenon called “rudder reversal.” Essentially, the misaligned valve would allow hydraulic pressure intended to push the rudder in one direction to be channeled into a part of the actuator that would push the rudder in the opposite direction. In a rudder reversal, the effects of the rudder pedals would be reversed; trying to turn the rudder left would in fact turn it right, and vice versa. Because a rudder reversal could only happen when the slides were extended too far, it could only happen at the same time as a rudder hardover, right when pilots need to react quickly to counter the sudden roll.
On the third of March 1991, this hidden flaw turned deadly for the first time. United Airlines flight 585, a Boeing 737, took off from Denver, Colorado with 20 passengers and 5 crew members on board for a short 23-minute hop to Colorado Springs. As the plane approached Colorado Springs, all appeared normal. Suddenly, the plane rolled hard to the right and dived toward the ground. The pilots fought to save their plane, but they didn’t have enough time or altitude, and flight 585 nosedived into a park in the Colorado Springs suburb of Security-Widefield, killing all 25 people on board. The plane came in so low that witnesses saw terrified passengers inside, and the explosion shattered apartment windows in a nearby apartment complex.
Witnesses and emergency services rushed to the scene, but soon realized that no one could possibly have survived. The wreckage had been thrust into a 10-foot-deep crater and crumpled like an accordion; there was virtually nothing at the site that was recognizable as having been part of a plane. Investigators faced an enormous challenge, but they did soon cast their suspicions on the PCU. However, the yaw damper, which was an early suspect, had been compressed to one tenth of its original size and could not be examined. The servo valve was also heavily damaged, but investigators wanted to test it anyway, so they took it to the manufacturer for analysis.
The remains of the valve were taken from the United Airlines headquarters to the headquarters of Parker Bertea, the company that designed and built the valve, in Irvine, California. Investigators discovered upon their arrival that someone had made off with the spring and end cap, but at the time they did not know the significance of this act. The NTSB and Parker Bertea replaced the spring, the end cap, and several other parts that were ruined in the crash and began running tests on the valve. Nothing abnormal was found. Boeing, which had packed the valve for shipping, did not explain why it kept the spring and the end cap. It instead tried to steer the NTSB toward a conclusion that the crash was caused by a wind rotor, a phenomenon similar to a sideways tornado that could sometimes be found along the Rocky Mountains. The NTSB did not buy the theory, but it also could not find any evidence that the dual servo valve had failed. Then, in 1993, the captain of a United Airlines flight alerted maintenance to apparent problems with the plane’s rudder while on the tarmac in Chicago, and the dual servo valve was found to be faulty. But investigators were unable to prove that this same malfunction had happened on flight 585, and the investigation ended without determining a probable cause.
Three more significant incidents possibly involving this malfunction occurred in 1994. In April of that year, a Continental Airlines Boeing 737 suddenly rolled hard to the right while flying near Honduras. The pilots struggled against the roll for 18 minutes before making a successful emergency landing. When Boeing got wind of the incident, it blamed it on the yaw damper, even though it was demonstrated that the yaw damper couldn’t cause a roll as severe or long-lasting as the one the pilots of the Continental Airlines plane experienced. In a report acquired by the Seattle Times, the flight’s captain, Ray Miller, wrote: “I have been told by my company . . . that the FAA and Boeing (were) aware of the problems with the spurious rudder inputs but considered them to be more of a nuisance problem than a flight safety issue. I was informed, that so far as everyone was concerned, the rudder hardovers were a problem but that the
industryʼ felt the losses would be in the acceptable range. I was being mollified into thinking the incident did not happen, and for thegreater goodʼ it would be best not to pursue the matter. In other words I am expendable as are the passengers I am responsible for, because for liability reasons the FAA, Boeing et al cannot retroactively redesign the rudder mechanisms to improve their reliability."
In August 1994, a Sahara India Airlines Boeing 737 conducting training approaches at New Delhi airport suddenly rolled over and crashed after a touch-and-go takeoff, striking an Ilyushin aircraft that was parked on the tarmac. All four crew members were killed, as well as five people working on the Ilyushin. Indian investigators said that the trainee pilot had pressed the wrong rudder pedal while responding to a simulated engine failure, but American investigators also found that the spring and end cap on the servo valve were non-standard; they had actually had their original serial numbers scrubbed off and replaced with new ones. Tests on the unusual servo valve found that it would sometimes reverse, but the crash was not officially blamed on a PCU malfunction despite this evidence.
Finally, in September 1994, the series of events took a new, disturbing turn. USAir flight 427, a Boeing 737 carrying 127 passengers and 5 crew from Chicago, Illinois to Pittsburgh, Pennsylvania, was on final approach when it suddenly rolled violently to the left. The pilots, caught by surprise, fought to bring the wings level, but due to the rudder reversal, they were actually steering their plane into the ground. It entered a vertical dive straight toward the ground, spinning around like a top. The captain managed to make a radio call to air traffic control, exclaiming, “Four two seven, emergency!” He accidentally left the radio on, broadcasting the final seconds of the flight into the control tower, until the plane slammed into a wooded ravine, killing all 132 people on board.
This time, NTSB investigators suspected that the dual servo valve was responsible and they wanted to prove it. They ensured that the device was carefully preserved and was not tampered with, but again, they couldn’t test it without replacing some parts. The tests were inconclusive. However, metal particulate matter was discovered in the hydraulic fluid, which could have caused the valve to jam in the rudder hardover position. Boeing tried to dismiss this claim by showing that a metal particle jammed inside the valve would leave scratch marks when the valve moved; such scratch marks were not present on the valve from USAir flight 427. The NTSB was not convinced, and investigators believed that smaller particles in large concentrations, or fewer small particles in low temperatures, could cause a jam without leaving a mark. They also were able to demonstrate that cooling the valve to -40˚C and commanding it to pump hot hydraulic fluid could cause it to jam, but could not explain how such a large temperature differential could occur in flight. However, they did also discover that one in five Boeing 737s had much more particulate matter in their hydraulic fluid than recommended, and that the outer slide of the dual servo valve could jam for long periods without being noticed, making the otherwise rare simultaneous jam of the inner and outer slides much more probable.
While the USAir investigation was ongoing, more incidents continued to happen. In June 1996, Eastwind Airlines flight 517, a Boeing 737 carrying 53 passengers and crew from Trenton, New Jersey to Richmond, Virginia rolled sharply to the right on landing. The pilots applied full left ailerons to right the plane, but reported that the rudder pedals were unresponsive, indicating that a jam had taken place. The captain held the plane in a precarious sort of sideways flight for thirty seconds. After this harrowing half-minute, the plane leveled itself, and the pilots landed the plane. One flight attendant was injured, but a catastrophic accident had been avoided.
After the Eastwind incident, Boeing took the captain of that flight on a test run in which they simulated a yaw damper failure. The result did not at all resemble what the Eastwind pilot had actually encountered. Boeing’s narrative was further undermined when tests of an alternative theory for the USAir crash failed to support their hypothesis. Instead, Boeing tried to claim that flight 427 crashed because a pilot had a seizure and depressed the rudder. NTSB investigators dismissed this as ridiculous. Now with all the evidence they needed thanks to the additional information provided by having an intact aircraft and living pilot after the Eastwind Airlines incident, they were ready to blame Boeing’s PCU valve. The NTSB report recommended that the valve be redesigned, and the Federal Aviation Administration mandated that the changes be implemented by November 2002. Since then, no 737s have crashed due to rudder hardover or rudder reversal.
Boeing had no choice but to carry out the changes, but the company never stopped trying to deflect blame. While the investigation was ongoing, it adopted a philosophy of trying to avoid paying out damages to families of crews because this could be legally interpreted as an admission of responsibility. It had tampered with the PCU from the Colorado Springs crash and repeatedly tried to misdirect the investigation with “alternative” theories. It is widely suspected that Boeing knew about the problems with the PCU for decades but had done nothing, despite the hundreds of reported incidents. Because no one was collecting all the accounts of rudder deflections, it was likely that no one except Boeing realized how common they were. It was not until people started dying in crashes that enough scrutiny was placed on the 737 to uncover this history of ignoring the problem.
Another lesson to come from this sorry series of events is the paramount importance of finding the cause of every disaster. Families of victims believe that if the investigation into the crash of United flight 585 had been conducted properly, the tragedy of USAir flight 427 never would have happened. And there are other crashes that might be connected to the same failure—some still argue that rudder hardover caused the 1992 crash of Copa Airlines flight 201 in Panama, ruled to be pilot error in response to a malfunctioning instrument; and the 1997 crash of Silkair flight 185 in Indonesia, ruled by the NTSB to be pilot suicide, but determined to be rudder hardover by its Indonesian counterpart.
The crashes also highlighted the vulnerability of the NTSB to corporate meddling. In 1996, According to the Seattle Times, the safety board had only 90 employees and relied on manufacturers to provide technical expertise in cases like the United 585 and USAir 427 crashes, which made it much harder to investigate cases where the manufacturer knew that it was responsible. Boeing’s obfuscation at every turn was pure corporate expediency: fixing the problem would require a massive recall costing hundreds of millions of dollars, not to mention millions more in compensation that would have to be paid out if Boeing admitted responsibility. Even when the flaw began to result in deadly crashes, Boeing stuck by this policy. Had the failure been easier to detect and prove, they might not have been able to get away with it, but—thanks in part to Boeing’s muddying of the waters—they never faced the massive backlash that they should have received.
HackerNewsBot debug: Calculated post rank: 214 - Loop: 138 - Rank min: 100 - Author rank: 382
Content warning: I do confess: I used to be a user, and [i]just[/i] a user of social media. Hubzilla with all its options can be quite intimidating, therefore I didn't dive into all the details and opportunities like I possibly should have. During the last 2-3 months, com
Diaspora doesn't federate with anything, the other platforms federate with it/them[?]...
(Mike Macgirvin, as seen in some comment thread recently. I forgot where exactly.)
So, I've seen this problem's description from my d* stream only because and therefore after the issue had been solved, but I wasn't sure about that at this point. It got solved by helpful co-Hubzillians in a comment thread of another help-seeking post, where noone complained and noone blamed this user for not knowing. The topic itself was intriguing for me. For that reason I wrote a (hopefully not too) comprehensive comment to help out with my findings. I wrote all that from my d* pod, just as a reminder.
Sure enough, I didn't see from my limited d* perspective what other people, coming from Friendica or Hubzilla, added to the comment thread. And most probably vice versa. Then I tried to double-check whether I might trick broken federation by clicking on the "permalink" symbol, just to find yet another issue: Long-version Hubzilla permalinks are broken on diaspora* and it's a known bug. Let me explain and come back to my initial topic afterwards:
In case you (as in: d* users) didn't know: By clicking on the little chain lock link symbol next to the "date posted" in a d* stream one can open the long-format permalink to a post that has made its way to your pod and stream. For posts created on Hubzilla, this will result in a "404 - not found", sported by the/your d* pod, while the short version (from clicking on the date/time combo itself) works, but only on this very d* pod, e.g.
What I learned so far:
- one could use the long permalink version (with the so-called GUID) on every single d* pod as long as the first section corresponds with the pod they're on, i.e. just replace
https://dspr.io/by your own pod-URL
- long permalinks will work as a public link even if the reader is not logged into any hub (as long as the post was public. Maybe even if not, haven't tested.)
- short "perma"link will only work on this very pod internally.
https://dspr.io/posts/xxxx5xxxx10xxx15xxx20xxx25xxxx31diaspora*: 31 digits
https://dspr.io/posts/xxxx5xx8-xxx4-xxx4-xxx4-xxxx5xxxxx12Friendica: 5 groups, with a total of 36 digits/characters
https://dspr.io/posts/xxxx5xxxx10xxx15xxx20xxx25xxx30xxx35xxx40xxx45xxx50xxx55xxx60xxx@[subdomain].[domain].[tld]Hubzilla: 63 digits plus user ID
Now we're slowly approaching the main topic again. Not only does Hz sport communities, their developer team is incredibly responsive as well. I've seen Mike Macgirvin join discussions on literally every Zot-related platform I am aware of, always helpful, always outcome-oriented. I do not know (as in: have seen their profiles) of each single team member, but Mike's presence is remarkable, and just the opposite of a statement he made over at gnusocial:
Fediverse developers can be an incredibly toxic mob. I've found they are often more polite if you talk to them using their own software.
Should I file an issue then? Sure, if it's helpful..? I searched for "perma" over at Hz's issue tracker and found this: https://framagit.org/hubzilla/core/issues/1214
Hubzilla, Diaspora & permalink
[...]The Hubzilla permalinks generate an 404 error from a Diaspora account[...]
Exactly what I was looking for! ... But wait, that issue is still open, although it's 6 months old..? Oh well. But wait, there's more, this is not the end of my story, not by far --- thank you for your patience, by the way, please bear with me 😀
And again, I'd like to make this point clear: the following is absolutely not meant to be offensive, nor would I want to point out a single person. Every developer of whatever fediverse platform has my honest respect, not only for working on their project(s) in their spare time. I wish I could participate more, I wish I had the knowledge to do so.
Jun 8th, 2018 - The person who filed the issue to the Hz issue tracker did so as well on diaspora*'s GitHub repository, As you can read from the framagit comment, that was a duplicate. He/she got redirected to a slightly older issue from May 15th, 2018 which you can find here. A common procedure. That timeline reads as follows:
- May 15th - initial comment with description and suggestions.
- same day - 6 comments later the 3 participants (author plus 2 project maintainers) agree on a way how it is possible to fix it. They have immediately and open-heartedly stated that GUID permalinks from Hz don't work, but used to do so. Some background info is shared, including remarks about coding philosophies and guidelines.
- Jun 8th - nothing visible has happened so far. The duplicate reference is documented.
- Sep 4th - a team member adds labels: "bug" (for the first time) and "weird" (I love that one!). 3 1/2 months have passed.
- Oct 4th - Some change has been merged, declaredly having fixed the issue. Issue is closed, because the fix has been approved.
- Oct 5th thru Oct 8th - this is where it gets confusing for me, but interesting as well: A bump, some pull requests and, if you're still not tired of following all the cross referencing links, the attempt of a willing-to-help person to "spam" the maintainers with his solution for organizing pull requests. He gets answered elaborately. Appropriately? Please make up your own mind and think of what Mike said.
- Oct 5th - somehow something didn't work, the ticket is open again, still not fixed.
- Dec 31st (today) - Another 2 1/2 months passed. The issue is 7 1/2 months old, still open although seemingly quite easy to fix.
Did GUID permalinks ever work for Hubzilla? --- The statements differ. If they did, at some point some d* developer has broken it. If not, why would this platform need to produce as many links prone to 404 status as there are posts on a pod? For web crawlers, this is just a nightmare and it takes no wonder why decentralized networks technically only(!), as seen from a search engine's view have mendable reputation.
Why did it take over 3 months to label the issue a "(weird) bug"? --- Was there no structure to the issue list before? Of course, labels alone won't make for a well-organized work flow, there are many different and efficient ways to approach a growing list of todos and priorities. Well, someone finally felt the need of labelling it.
The issue had been worked on, got marked fixed and closed. Why the withdrawal? --- This is where it gets opaque, but maybe it's just me not reading the documentation thoroughly enough. I hope it's legit to apprehend that there is a massive problem concerning capacity, availability and of course time.
A link that isn't producible is a link not shareable... --- which makes d* sort of a "closed" network, doesn't it? I didn't join MeWe for that reason (amongst others).
From what I have learned about development-related people over at diaspora*, they're focused but open-minded, they know how they're doing things and why. Besides a whole lot of time they put pride in their work. Unfortunately it looks like there are not enough of them..?
This means for me as a user that I most probably can or should not count on major improvements in functionality, at least not very soon. There are several more issues I personally see in the d* concept, but it took me a while to find out. To put these in here as well would be overkill. Diaspora* really shines when it comes to simplicity but ease of use is not my main priority.
My verdict: Hubzilla has --by far-- much more potential than diaspora*. The team behind Hz appears to be very present, numerically adequate and striving for constant improvement. With all the extra features Hz has built-in in contrast to d* it's the one platform I can think of self-hosting it, d* won't offer incentive enough to do so right now.
If you (like me) want diaspora* to become an even better platform, the platform needs your help.
If you (unlike me) are a skilled Ruby, JSON, you name it wizard, please consider participating, the platform deserves your support.
And if you really have read this post completely: thank you! I hope I didn't stress your patience too much -- it was your decision to spend your time this way, I've warned you! 😉 This was very probably my last post for 2018. I would appreciate your feedback nevertheless.
Happy New Year!
#hubzilla #diaspora #github #framagit #issue #issuetracker #bug