Summary: a variable overflow can simultaneously shutoff all four generators on a 787 and cause it to crash.

The FAA just issued an emergency airworthiness directive (AD) that applies to the Boeing 787.

A Boeing 787 is an electric fly by wire airplane, meaning the computers fly the airplane. The computers and the flight control actuators require electrical power to run. No power means the plane crashes. This is bad.

The 787 gets its power from 4 generators, 2 on each engine. The plane can fly with only one generator operating, so there is considerable redundancy built into the system. This system was carefully reviewed both by Boeing and the FAA, and the control software was developed using software safety standards.

Each generator has a generator control unit (GCU) which manages the generator such that it is protected from faults and overloads. Built into the software of the GCU is a time variable that runs at 100 Hz. The variable is stored in a 32 bit signed integer. Thus it wraps from 0x7fff ffff (most positive value) to 0x8000 0000 (most negative value) in about 248 days. When that wrap occurs, the GCU faults out and turns off the generator due to the effect of time going backwards suddenly on the various monitoring algorithms.

The GCUs are kept powered during flight and generally kept powered on the ground through ground or auxiliary power. So they can remain on for very long periods of time. If, by chance, all four GCUs were booted at about the same time, and if, by chance, they remain powered for 248 days without interruption, all four of them will fault out at roughly the same time due to the counter wrapping. The possible consequence is that the plane loses all AC power and thus loss of flight control. This is very bad.

This bug is an example of how redundancy can be compromised by the fact software in each separate unit can have the same bug and they can trigger at the same time.

The FAA's required fix is classic. The 787 must be rebooted every 120 days. All power removed and restarted. 120 days was chosen so that there would be at least two chances in 248 days to do a reboot before the bug can manifest itself. This procedure will likely get removed with a software revision at some point in the future, but for now, this will "fix" it.

Note that if they had used an unsigned integer, the time to manifest the bug would have been twice as long. Negative time doesn't mean anything, so the variable should have been unsigned from the start. The misuse of signed and unsigned integers is a classic software mistake.

This bug is almost as good as the F-22 software bug that disabled their navigation and communications when they crossed the international date line.

https://s3.amazonaws.com/public-inspect ... -10066.pdf

Mike C.

So what happens when you suck birds into both motors? How long will the plane fly on its batteries?

I don't think the batteries can run the flight controls for very long, if they can at all.

The 787 has a RAT (ram air turbine) which would deploy in case of double engine flame out. RAT provides electrical and hydraulic power.

In a minute or two, they can get the APU started (and it may automatically start on its own with power loss, I seem to recall that is how it worked).

I think the FAA may have overstated the potential the 787 will crash from this bug, perhaps to justify the emergency status of the AD. I'm thinking 248 days for a GCU to be on is also very rare.

Mike C.

My buddy is a captain on 787. I'll tell him to scribble into his checklist to turn the plane off on occasion.

Funny,

Long time ago, way before Windows NT came along, we a had a bunch of Windows machines acting as a gateway between AS/400 (screen scraping terminal windows) and an Oracle database. We had a ton of issues. I solved them all one day with this:



The machines would simply get mechanically rebooted once a day at set intervals. It was a cheap fix, it worked…I would not at all be surprised if it's still up and running in a closet at one of America's largest credit card processors. Glad I'm mostly retired these days...

Technology is great, but I still do not like fly-by-wire stuff. I have seen too many cars go into "limp home" mode because the Gas pedal pots do not agree or do not match the throttle body position and the car goes to idle. I like a good ole fashioned linkage.

we consultants appreciate guys like you, that is NOT a solution a problem but rather a solution to a symptom

we consultants appreciate guys like you, that is NOT a solution a problem but rather a solution to a symptom

http://www.xkcd.com/1495/

Mike C.

If it was only that easy…Spoken like a fellow who never worked in the financial industry, with software written in 1960s, running on custom hardware and operating system, without a single programmer who designed it still around. It would have taken probably a million bucks just to find the source code for the original custom hardware/software, then another 10 to find the portal source between that and AS/400 that IBM designed in the early 1980s, then another who knows how much more to get IBM to fix their screen scraping custom library they built just for the company in the early 1990s. We had a deadline to move to all databases going back to 1960s into Oracle and accomplish it on time. If was a fixed bid project that Cap Gemini or whatever they were called at a time already billed the client millions for and failed to deliver. My bid was twice as much, but it was bonded. I pulled it off with 6 or 7 guys, 20 or so OS/2 machines running around the clock screen scraping 24 terminal windows (max we could open before AS/400 would crash) and had about 90% profit margin. It wasn't pretty, but it worked. Not a single kid who worked for me back then had a computer science degree, half were probably high school dropouts, but we pulled it off.

Timers were precisely because we could not do anything to debug the 10 layers we had no access to. It was a connectivity nightmare…Custom network cards/protos for mainframe access, once again with no source code for the drivers.

I remember being laughed at then too, by fellows with fancy degrees and certificates. I laughed all the way to the bank...

It was built in Seattle, wasn't it? Where's the Ctrl-Alt-Del?

[quote
This bug is almost as good as the F-22 software bug that disabled their navigation and communications when they crossed the international date line.

https://s3.amazonaws.com/public-inspect ... -10066.pdf

Mike C.[/quote]
I thought I remembered this taking place and checked with youngest son (F-22 driver in Alaska). Did happen but was about 10 years ago. Langley based birds. He was flying F-15's at Langley then. All fixed now! He likes to call his radios and such just computers that think they are communication and navigation devices. My setup in the Bo with a WAAS GPS, Aspen 1000, autopilot, and Foreflight on a mini iPad is much better than his avionics. However, I do go slower.

I think the ctrl-alt-del technology is a bit east in Redmond WA

The inventor of ctrl-alt-del sticking it to Gates:

https://www.youtube.com/watch?v=ZiH-iA9EO_Q

So the in house secret short cut is now a cultural icon.

Mike C.

> This bug is almost as good as the F-22 software bug that disabled their
> navigation and communications when they crossed the international date line.

Yup! They liked that feechur so much, they repeated it a second time!