Gyros and Gimbals, oh my! — The James Webb Space Telescope Reliability Lessons
Success through Reliability — Webb Lesson 3
Apollo 16 landed on the Moon 50 years ago this week & the James Webb Space Telescope took its first test image last month. The story of these two different milestones has a thread connecting them which is a reliability story with many lessons. This thread also connects us to the venerable Hubble Space Telescope. Both Apollo and Hubble had near failures during use of gimbals. This is the story of the how and why this occurred, how the issues were resolved — and how Webb will avoid them.
Before describing the lessons, we’ll need a brief explanation of what gyroscopes and gimbals are for. A gimbal is simply a support structure which lets whatever it is holding rotate in a single direction (i.e. left/right, up/down, pitch/yaw). If you connect three gimbals together then the gyroscope they hold will remain stable no matter in which direction you turn it. This kind of device can be used to hold a camera and keep it stable while you move. But gimbal mounts have other uses, such as pivots.
In space exploration the two most common uses are either to keep track of the spacecraft’s orientation (a three gimbals-gyroscope) or to help move it (i.e. by pivoting the spacecraft engine)
So what happened with Apollo 16’s engine gimbals 50 years ago?
The flight plan called for the two parts of Apollo 16 — Orion, the lander and Casper, the orbiter —to separate and for Casper to change orbit around the Moon and wait for Orion to return. Changing orbit meant using gimbals to change the direction of the main engine and firing it. Casper had two sets of control systems to move the gimbals and, just before using the primary system to activate them, astronaut Ken Mattingly tested the backup system.
And as soon as he did — the spacecraft began to shake. It rocked like a car hitting a pothole on the road.
Since NASA’s mission rules called for “two functional, redundant, control systems for the main engine at all times” this put a halt to the Moon landing while NASA engineers checked and double-checked to find the cause of the rattles and shakes in the spacecraft.
One of the dilemmas about judging the safety of the control systems was that this was an inconclusive edge case for the mission rules. They called for a “functional” control system. Now, the backup control system was functioning. It was simply shaking while it did so. To a certain extent the questions became “What is a functional control system?” and “How much can it shake and still be considered functioning?”
This is similar to the Site Reliability Engineering adage that “slow is the new down” — it’s not enough for the system to be working, it must be working in a way that satisfies the consumer. But what is too slow? Who makes the judgment?
After a few hours of investigation, the conclusion was reached that Mattingly would be the human backup for the spacecraft’s backup control system. If the main system failed AND the backup didn’t point correctly due to shakes, he would be able to manually aim the spacecraft and correct any mistakes made by the control system.
Reliability would be kept — there were still two functional and redundant control systems, just that the second one would be supported by a human instead of being fully automated.
Apollo 16 landed successfully on the Moon on April 21st 1972 and, after spending three days on the surface, returned to Earth safely without having to use the faulty backup control system. The engine gimbals did their job flawlessly.
Decades later the Hubble Space Telescope also uses gimbals to orient itself, but instead of being used to aim engines these gimbals contain gyroscopes to sense orientation¹. Using gyroscopes to orient itself means that Hubble can be extremely precise in pointing — and just as importantly, stay pointing —at specific objects in the sky which it is photographing. To be clear, there are more gimbals in Hubble — for example, large spinning wheels which use momentum to change the direction the telescope points — but I won’t be discussing them here.
All the gyroscopes are powered by the never-ending electricity which Hubble’s solar panels generate and not by fuel which would have run out during Hubble’s 30+ years in space.
The gyroscopes are so important that while only three are needed for Hubble to point accurately, it was launched with six gyroscopes, so that it would have plenty of spares.
Due to the constant use and general wear and tear on the gyroscopes over the course of Hubble’s operational lifespan many gyroscopes have failed. They were all upgraded and replaced (again and again) by astronauts maintaining Hubble with the Space Shuttle in 1993, 1999 and 2009. Over the years further failures have reduced the number of working gyroscopes back down to three in 2022.
Today NASA has devised backup plans which will enable them to point Hubble with only one or two gyroscopes, but with certain compromises. The goal is to keep Hubble working with less gyroscopes, even if it wouldn’t keep its full capabilities.
Since astronauts will not be able to make any repairs on the James Webb Space Telescope, NASA decided to completely change the design of the pointing mechanism.
- Instead of using mechanical spinning gimbals, the gyroscope is a “hemispherical resonator gyroscope”. In plain English, this means that there are no mechanical spinning wheels and instead there is a special quartz crystal which has an electrical current sent through it. Less moving parts, less chance of failure.
- While Hubble moved the entire spacecraft to point the telescope, Webb has a special mirror designed to keep the telescope pointed accurately even when the spacecraft drifts slightly off point or jitters. This means less stress on the systems to keep perfectly aligned.
During Webb’s period of initial commissioning and testing the pointing mechanism is of vital importance since the telescope is being aimed, aligned and focused simultaneously. Any jitter in the gyroscopes will not allow the mirrors to be fine-tuned correctly.
When looking at the way NASA approached resolving issues with gyroscopes we can consider how SREs learn from incidents and improve the future reliability of their systems.
Hubble’s gyroscopes, while state-of-the-art for the 20th century, had an inherent weakness due to their mechanical nature (spinning wheels). Every gyroscope upgrade made it more dependable. This is the equivalent of fixing bugs and adding automations to increase the Mean Time Between Failures.
Another improvement was creating new pointing algorithms to achieve Graceful Degradation — while 3 is best, Hubble can work with 1 or 2 gyros too.
So Webb was designed with the lessons of Hubble in mind — Webb’s gyroscopes are not more dependable than Hubble’s because the mechanical components are improved but because the potential of mechanical failure has been avoided in the first place. A new technology has the benefit of both reliability and simplicity — an SRE’s dream come true!
Not only that: Webb was designed from the ground up to function with only two gyroscopes, which means that it is less sensitive to failure.
Lastly, we can see the how each of these solutions is more automated than the previous one — Apollo used humans “in the loop” to actively act as a part of the system, Hubble used humans to perform maintenance² on the active system, Webb is designed for human-free operations.
On the 28th of April I’ll be speaking at the WTFisSRE virtual conference in the session called “9 billion dollars of Reliability lessons from the James Webb Space Telescope”.
If you’ve been enjoying this series of articles, I hope you’ll join me there. Even if you don’t, I’m sure there will be other sessions you’ll enjoy!
While you wait, you can check out some of IBM’s space tech plans with the Endurance Cubesat — https://endurancein.space/
Articles in this series:
For future lessons and articles, follow me here as Robert Barron or as @flyingbarron on Twitter and Linkedin.
- Yes, Apollo 16 also had gyroscopes, but they’re not part of this story.
- Hubble’s maintenance was not just to repair malfunctioning components but also to upgrade older equipment with more modern and capable ones. This is the reason why Hubble has continued working for decades in space.
