2009-12-31

Opportunity Sol 333 (Spirit Sol 354)

Surprise, surprise, the drive went exactly where it was supposed to go. Most of the participants in the SOWG meeting couldn't be happier about it. The TPS [Thermal Protection Subsystem] folks don't seem too happy, though; SAP's reachability maps are showing that the areas they're most interested in aren't reachable by the IDD.

But they're having trouble getting this idea across, because they keep using the word "targets" to try to express themselves. This confuses the other science and engineering teams, because they think the TPS folks are trying to tell them that the SAP targets we picked yestersol are no longer valid, and they keep responding that yes, that's how the tool works, you just need to reselect them thisol. (When you pick out imaging or IDD targets in SAP on one sol, they're not automatically propagated to the next sol. This has been a common source of problems and confusion, so it's what the science team is geared to think the TPS guys are having a problem with.)

After a few minutes of this, I bring my translation skills to bear. I walk over to the TPS team and say, off mike, "You're just trying to say that the reachability maps show that the IDD can't go where you want it to, right?"

"Right!"

"You need to say that without using the magic word 'targets.'"

They look at me skeptically.

I give them my best Harrison Ford grin. "Trust me."

With one eye on me, one of them keys the mike and says what I told her to. It works instantly, breaking the confusion loop. I go back to my seat.

The meeting also stumbles over the lack of terrain meshes. SAP's reachability maps are just an approximation, and might be confused in this case by the weird (non-rock-or-soil-like) shape of the flankshield; Frank and I suspect that the areas the TPS team wants to see, though marked unreachable in SAP, are in fact reachable. RSVP can tell us that quickly, but we need a terrain mesh. The meshes are produced by another team, the MIPL team, and there seems to be all kinds of variability in the process.

Frank's getting annoyed, both by the delay on the meshes and by what he sees as the TPS team's negativity about the job we're doing for them. After answering the third or fourth question about reachability with "Well, we could tell you that if we had a terrain mesh, but ...," he's had it. "There seems to be a correlation," he mutters grimly, "between the trickiness of reaching targets and the latency of the fucking meshes."

There's also a tension developing between the science team and the TPS team, if I'm reading the room right. I think the scientists are ready to get out of here -- going to the heat shield was fun, but they're ready to get back to science. They want us to start charging across the plains to the etched terrain to the south of us. But the TPS folks would spend the rest of the mission here if they could; they want to explore the heat shield to death. Ray looks like he's as tired of these guys as Frank is, though he disguises it better, at least until the end of the meeting, when he snaps at last: "Okay, I'm fed up with this. Just pick some targets we can reach and let's get this thing done already."[1]

Later, after the TPS folks work with Alicia Fallacaro (one of the scientists) to pick targets, they sit down with me and Frank to ensure we're up to speed on their goals. At one point, they're showing us an image of the section they want an image of. "For scale," one of them says, "the width of this section is point six five."

"Point six five ... centimeters?" Frank asks.

She looks at him blankly. "No, inches."

"Inches?!" Frank and I cry in unison. We can't believe it. It's as if she'd given us a measurement in furlongs, or cubits. Or as if she'd specified her dates using the Mayan calendar.

But she doesn't know when to stop digging the hole. In fact, from the blank look on her face, I don't think she realizes she's in one. "I can't think in centimeters," she says innocently.

Frank and I are nearly livid. "How'd we ever get to Mars with you guys?!" he asks, incredulous.

"We don't always," I remind him.

This is a reference to the destruction of Mars Climate Orbiter ... you know, where Lockheed Martin Aerospace computed thrust in imperial units instead of metric, and just gave JPL the number without the unit, in violation of LMA's contract?[2]

And five years later, the problem's not solved. THAT'S EXACTLY WHAT HAPPENED JUST NOW. And -- wait, a NASA engineer who can't think in metric? It's as if she'd said she can't do arithmetic, or tie her own shoes. The guy with her isn't much better; he can convert to metric, but it didn't occur to him to do it until Frank and I blew up.

Holy. Fucking. Shit.

So they leave, and when Frank and I calm down, we return to working on the IDD sequence. By this time, the terrain meshes have mystically appeared. The areas the TPS team was interested in are indeed reachable, at least mostly, which is the good news. The bad news is that to reach one of the regions, we're going to have to move the IDD close to the right front wheel -- closer than we've ever done, and closer than the IDD flight software will allow. By default, the IDD flight software masks out certain regions around the rover's body, effectively making "no-go zones" where we're not allowed to place the arm. This is, obviously, a safety measure, but as with many safety measures, it's computed conservatively.

We dig up the commands to turn off wheel-volume collision testing and use RSVP to simulate putting the IDD where it needs to go. It's close. There's air, but it's close. And just because it's safe in RSVP doesn't mean it's necessarily safe in the real vehicle, which has cabling we don't model. Frank and I look at each other. "Testbed."

Frank and I don't know how to use the testbed rover -- it's supported by lots of complex test equipment, and we have no idea even how to boot the stuff -- so he manages to find Ashitey and Mark Maimone, and we all head down to the testbed to try it out there. (Ashitey's such a great guy. He's on shift driving Spirit and could easily have gotten out of this if he'd wanted to, but he comes down to help us out anyway.) Before leaving, I split out the dangerous part of the sequence, so that we can do the rest even if we can't prove that the scary bits are safe.

The testbed is -- well, it's the testbed. It takes hours to do anything there; it's all false starts. (In the end, it takes us three hours to run a ten-minute sequence, and this is pretty good by testbed standards.) I spend the time getting to know Mark's intern, who's in school but hopes to be a third-generation JPLer. His mom worked on Voyager, and his grandfather also worked here. Weirdly, his grandfather hired Jim Erickson, who is now MER's project manager. I try to get him to tell me some embarrassing stories about Jim, but he won't. ("Jim got me this job," he laughs, "I can't squeal on him." Damn engineers, with their honesty and consciences!) He does mention something about "Little Jimmy," though ....

And one of the other engineers is bringing a group of friends through. He asks someone to talk to them about what the rovers are doing now, and being an attention whore, I tell them about the heat shield exploration and why we're in the testbed. The cute Asian girl has her friend take her picture with me. I like being an attention whore.

When we get the sequence running at last, it's clear it's safe. I take a movie of the rover with my kick-ass Palm Zire 72, so I can show it to Leo in case there's any question later.

We get done just barely in time for the CAM, which now goes smoothly since we've validated the parts we're most worried about.

Frank and I are done, but neither of us goes home for a while. I've found a HyperDrive bug -- a potentially serious one; the code doesn't save and restore the state of the vehicle suspension. Thisol was exactly the sort of rare sol where that bug could have bitten us: we're moving the IDD close to the wheels, telling the vehicle to ignore collisions because we're going to do the checking manually, and the tools are lying to us about where the wheels are ... potentially very bad. It happens not to bite us this time, since the bug makes the rover look like it's on flat ground and Opportunity really is on flat ground, but Frank can't let it go. ("We can't go into the new year like this!") So he fixes the bug on the fly, and installs an updated version for testing.

Well, we'd been running the officially delivered version for about two weeks now. It was time we moved on. Ahem.

It's time to leave. I shake Frank's hand on the way out. "Pleasure working with you, dude. Happy New Year."

"Same to you, my friend."

[Next post: sol 358 (Opportunity sol 337), January 4.]





Courtesy NASA/JPL-Caltech. Flankshield, dead ahead, cap'n.




Footnotes:

[1] The TPS folks were connected to the heat shield manufacturer, Lockheed Martin Aerospace -- I think they were JPLers who'd worked with LMA on the heat shield design -- so they'd been part of MER's development. However, they were not part of the MER ops team, and it showed. This is not their fault, but ops has a certain pace, a certain rhythm, that outsiders just don't appreciate -- and, to be fair, we must somehow fail to communicate it to them. But this wasn't the only time we've had this sort of problem; it generally happens whenever we have any kind of outside expert participating in a tactical shift.

[2] Not to be pedantic (who, me?) or anything, but ... this is why it's doubly irritating when halfwits sneer at NASA's supposed inability to convert metric units. The mistake was a lot more complicated than that, a communication error (or perhaps a system engineering error) rather than an arithmetic error, and it was LMA rather than NASA (or JPL) that fundamentally erred. It's fine to be snide, when it's deserved -- as long as you are also right.

Also, strictly speaking, I suppose that LMA didn't compute using imperial units but rather US customary units, although for the unit in question -- foot-pounds, god help us -- the two systems are the same.

8 comments:

Ed Davies said...

...but you're still using centimetres?

Come on, use millimetres or metres and other powers of a thousand. You can spell it "meter" if you like, though, we'll know what you mean.

zub said...

once blue, now silver... ohhh mighty palm zire 72... amazing lil machine

Scott Maxwell said...

@Ed Heck, I've been known to use decimeters. But mostly just as a joke. :-)

Scott Maxwell said...

@zub I still have mine, though my Centro has entirely replaced it. (I also have an iPhone, god help me.) I'm amazed that I carry something like 1000 times as much computing power on my hip as I used to have in my house when I started using computers.

Kyle said...

First time commenting. Absolutely fascinating blog.

I still run my Zire 72. Wonderful little machine - way more capable than most people give it credit for. Although it is silver now, as mentioned earlier.

Anonymous said...

But you guys still use non-SI units of time for daily planning of rover operations! You broke that when you divided the martian day into 24 martian hours, 1440 martian minutes and 86400 martian seconds.

David Noble said...

@Anonymous: it's probably even more complicated than you think... rover operations uses a multitude of time types, though UTC is one of them (for Earth Received Time of downlink data).

Some further reading for folks interested in the topic:

Basics of Space Flight - References Systems
CHRONOS tool for converting between times
The Care and Feeding of the
Mars Exploration Rover (MER)
Ground Data System (GDS)
, slides 22-23

Scott Maxwell said...

@Anonymous There's a distinction between durations and absolute times.

We almost invariably use Earth seconds (the SI unit) -- or Earth minutes or Earth hours -- for durations. I can count on one hand the number of times we've used Mars-time units for durations; it's definitely frowned upon, and I don't think it's ever done any more.

Mars time gets used for absolute times -- local noon on Mars is a lot easier to express in Mars time than in, say, UTC.

Incidentally, my sequencing software (RoSE, the Rover Sequence Editor, part of the RSVP suite of rover-driving tools developed for MER) was written in the wake of the MCO debacle. It is consequently scrupulous about unit issues -- ensuring that you specify units and displaying what units you specified -- and it doesn't even offer you the option of using inches or pounds. :-)