Transterrestrial Musings

Comments

Oh Rand, it's actually worse than you think!! I saw film of the start of the race on 2 of the Sunday morning news shows. One of the entrants was a motorcycle. NOT a four wheeler or even a trike. An actual 2 wheel dirt bike with an electronics package attached. It rolled about 8 feet and fell over. Surprise, surprise as Gomer would say. If that was the level of expertise involved, the governments money was safe. Come to think of it, the entrants I saw reminded me more of those old newsreels where they showed goofy inventions.

Posted by Steve at March 14, 2004 08:24 AM

All of the teams had to run a qualification loop, filled with obstacles, in order to be able to run in the actual race. Even that "actual two-wheel dirt bike" that "rolled 8 feet and fell over" had successfully run the qualification course. The best funded and most expensive effort, CMU / Red Team had a near "fatal" accident when their system misjudged a situation (on a dirt road, I believe) and rolled over, destroying their entire sensor platform (luckily they had a spare available).

Most of the problems seen were related to the requirement for no human input whatsoever -- systems "got lost" and could never re-acquire the track they intended to take. We don't put that constraint on most robotic systems; at best, they're only as 'smart' as toddlers when it comes to their environment. But that was DARPA's constraint, and I suppose it makes for a more interesting puzzle.

If you dig around the project and team pages you can see the sort of tests that they ran, qualification courses, etc.

Explanation? This is a very difficult problem to solve. Gosh, even skilled 4x4 drivers on jeep trails are constantly burying wheels and breaking springs and axles.

Conclusions? 2006 (the next run) should be interesting. People's expectations were too high; I doubt anyone at DARPA really thought anyone would finish the 147-mile course, and most of the teams certainly didn't. The best funded teams (CMU/Red Team & Team SciTech II, which was actually Rockwell) did well, but not that much better than one lesser known CalTech team (Golem Group) and one self-funded team (Digital Auto Drive).

And an observation: a little public money on the table (and I do mean a "little": the reward for the military of having a reliable autonomous vehicle for battlefield use would be worth at least six orders of magnitude more money than this reward) got a lot of people to get out there and try to build something. I've noticed that people like to bash robots here, I assume because they are seen as a threat to manned programs, instead of an adjunct. But Moorcroft, Lewis, Powell and Fremont didn't refuse to take information from native guides because it would threaten their exploration programs. Montgomorie sent two agents, Nain and Mani Singh, into the hostile environment of Tibet (hostile in that white people to tried to enter would be killed), to survey it for those who would come later. We never would have understood the role of the Van Allen belts without remote observation, nor would we have had as much luck in picking lunar landing sites without mapping and landing robots. Not to mention that just flying missions from here to where we want to go, manned or otherwise, helps us make those missions safer for those who will go later.

And we will go later. And an autonomous robot car will make it to Las Vegas from Barstow.

Posted by Kenton A. Hoover at March 14, 2004 09:37 AM

The only reason I can see to run it as a race is that that means everyone is dealing with the same weather, season and time of day conditions. (I hope the next race has to deal with an unexpected thunderstorm.)

As for the robot bashing-- sorry, but it's entirely reasonable to expect better than eight feet in any circumstance. I mean, how the hell does any system "get lost" in eight feet? Spare us the spin, please.

Posted by Raoul Ortega at March 14, 2004 11:49 AM

Other points/questions:

Has any robot made the trip without human intervention, under any circumstance or condition, no matter how long it took? If not, then why even bother to run this?

Back when I did in-house production software, we'd first test on selected data. Then we'd use live data and watch what happened, Finally, we'd run in parallel with the existing system and figure out why the results didn't match. Only then would we consider switching over to the new system. Sounds like these tests tried to make the leap from the first stage to production without going through the middle steps.

So I'm wondering if environmental considerations are such that the permitting and paperwork make it easier to just run everyone at once every so often, because you can't do it on your own or set up a trial as Rand suggests.

Posted by Raoul Ortega at March 14, 2004 11:57 AM

As a programmer who has built a few (personal) embedded devices and “toy” robots and has an interest in AI, I can tell you this is a VERY hard problem. Back in the early sixties, computer scientists thought that computer vision, navigation, etc. would be “easy” because computers had already done many things very hard for humans. Therefore, they reasoned, there must be “magic algorithms” for the really easy stuff – like vision. What they didn’t realize is that much of our brain, and that of other animals, is dedicated to making sense of the environment and moving about in it. We aren’t conscious of that, because that would serve no useful purpose. We both have to duplicate the massive processing power and figure out how to use it – tricky when the “hardware” doesn’t work the same, and we don’t know the process. The "easy" things had been easy because our brains hadn't been built to do them, and we understood the processes in detail.

Until recently, it was a very big deal to get a robot to autonomously cross an idealized room for a few feet without crashing into something. The sensors and other hardware would have run in the tens of thousands of dollars at least, with a radio link to a high end computer or array of computers. It is only because of the exponential growth in processing power and dropping prices that this race was a possibility at all.

DARPA said they didn’t think it likely anyone would win this time. But, this race got people outside of a few small groups thinking about what would be needed. Most are still working with very restricted environments. Think of it as more like the prize for human powered flight – it isn’t going to happen immediately. I’m impressed some of the teams got as far as they did.

By the way, I don’t remember anyone here “Robot Bashing.” I think semi-autonomous robot spacecraft are great for preliminary and assisted exploration, and I haven’t seen much argument with that point around here. But for detailed exploration, humans are far better. And, there just isn’t much POINT to running around with robots unless people get out there too, not just to explore, but to live.

Posted by VR at March 14, 2004 01:26 PM

I don't bash robots in general, and I certainly didn't in this post. I'm bashing the people who designed the "race" and the robots. If it's that hard, then set a challenge that's more achievable and realistic, but don't pretend to run a race of over a hundred miles when no one can even demonstrate ten.

If there are environmental issues, then find someplace where there aren't, but at least do a real test, rather than just hoping it will work on race day. That's not engineering--it's wishful thinking.

Rather than falsely accuse me of robot bashing, Kenton, why don't you address what I actually wrote? I repeat the question--why even enter a race when you haven't even demonstrated the ability to run the course at any speed? What was the point?

Posted by Rand Simberg at March 14, 2004 03:29 PM

And I should add that, even if I did "bash robots," it wouldn't be because I consider them a threat to manned space programs. It will be a long time before robots threaten the only manned space programs that really count--I've never seen a robot that wants to take a lunar vacation.

Posted by Rand Simberg at March 14, 2004 03:33 PM

I was assuming that they kept the route secret because they didn't want anyone pre-programming everything into the robots. Yes, there are other ways to do this (i.e. more baby steps), but it might be hard to prove that someone didn't just pre-program the whole thing.

Posted by Matt Simmons at March 14, 2004 04:08 PM

Fine, keep the route secret. My point remains that they ought to be able to get from, say, Palm Springs to Blythe if they want to have any hope of Barstow to Vegas.

Posted by Rand Simberg at March 14, 2004 04:40 PM

I wondered about the testing too.

I spoke with one of the Caltech team members a week ago, and he already seemed to be looking to the 2006 race. I'm not going to pass judgment until I hear more details.

Posted by Kevin Parkin at March 14, 2004 08:36 PM

From what I read, these are mainly engineering students who may not have had much exposure yet to incremental development concepts. It sounds like a fun project, and it would be easy to get carried away hacking a solution that may not work.

Posted by Jeff Arnall at March 14, 2004 09:29 PM

> What's up with that?

Simple. The goal was too ambitious, but the teams, having invested a lot of time, money, and effort had nothing to lose and possibly a million dollars to gain by giving it a shot. No mystery there.

BTW, it is possible to do quite a bit better than these teams did (and people have), but a shot at a mere $1M is not nearly enough to draw the kind of talent and investment you'd need to do so.

Posted by Erann Gat at March 14, 2004 09:42 PM

check www.plyojump.com/weblog for very good insights on this.

Posted by kert at March 14, 2004 11:26 PM

Matt is right on the preprogramming. This sort of thing is far easier if you can program in “hints.” Real time is also an issue. Note that the Mars rovers move very slowly and are given lots and lots of hints. They had to be given the same course at the same time, and a race is just more interesting to news agencies.

Rand, I agree that it was too ambitious (I’d be looking at perhaps a 25 mile course for a $250,000 prize) but I see three possibilities: (1) What I mentioned before – this wasn’t expected to be an easy goal that was likely to be solved in the first round, (2) the DARPA sponsors didn’t realize just how hard the problem is (that is very common, as evidenced by some of the comments here), or (3) they knew very well how hard the problem was and wanted to demonstrate it publicly. There is a Congressional mandate that 1/3 of ground combat vehicles be able to operate unassisted by 2015. That is an incredibly ambitious goal. When it doesn’t happen, this will be good ammunition when they’re asked “Why not?”

Posted by VR at March 14, 2004 11:31 PM

The general problem is with traditional western approach to robotics. Most of the GC competitiors come from that old "AI first, useful robots later" school of thought, and thats why you see converted humvees with couple of generic navigation sensors and computers slapped on entering the race.
Eastern robot builders have for some time now focussed on building sensor-laden dexterious bodies that are suited for their environments, and can sense it appropriately. For instance, im quite certain navigating this course would have been a lot easier on four-legged or two-legged robot.
After a couple of trials, im sure some of the competitors will get the hint.
If SandStorm should have been able to feel the "pain" in tires before they caught fire, and backed up by reflex.

Posted by at March 15, 2004 12:14 AM

Kenton,
My point in saying what I did about the motorcycle was this, riding any two wheeled vehicle takes up a lot of computing cycles. Not to mention, the slower you go the harder it gets. When you reach a point where any stopping is required, its almost impossible for the average rider. Now what I'm talking about is a human guided two wheeled vehicle, not an autonomuos machine.

They may have run a short course, but over the long haul it seems an impossibility to me. How similar, in obstacles, is the short course to the actual course?

If you have ever seen an Enduro race, you will see the real problem. Riders, in an Enduro, must complete a very demanding course of mud, rocks, water, etc. WITHOUT putting a foot down. It's a very balance oriented sport and I doubt it could be learned quickly. In fact, like many other sports or physical challenges, it relies heavily on the a persons natural abilities. Some people are naturally faster, stronger, HAVE MORE BALANCE, than others.

I doubt even a motorcycle with a gyro, could complete the course as a radio controlled device. And the challenge is autonomy for the vehicle.

Posted by steve at March 15, 2004 07:15 AM

Rand - to answer one of your questions, no one ran the course ahead of time because the course route was given to the entrants only a couple of hours ahead of time as a series of waypoints.

Darpa funded some private companies several years ago to find a solution to this problem and they have so far done nothing, so they decided to throw it open to the public with the idea of attracting talented hobbyists.

(I believe I read the motorcycle group forgot to turn on some sort of stabilization (gyroscopes?) system before the start of the race. Doh!)

Posted by PWahl at March 15, 2004 07:47 AM

As I said, the issue isn't whether or not they could run the course. The issue is whether or not they could run any course of over a hundred miles through the desert. It would appear that the answer is "no."

And I'm not criticizing DARPA's concept of a prize--I'm a huge fan of them. I'm questioning their implementation of the concept.

Posted by Rand Simberg at March 15, 2004 07:57 AM

Eastern robot builders have for some time now focussed on building sensor-laden dexterious bodies that are suited for their environments, and can sense it appropriately. For instance, im quite certain navigating this course would have been a lot easier on four-legged or two-legged robot.

Darpa wants a military vehicle that can keep up with M1s and Bradleys. The race was 150 miles in 10 hours. I also don't believe that the robots can be refueled along the 150 mile route. It would take one impressive walking robot to win the Darpa Prize.

Rand:

Here are the rules

http://www.darpa.mil/grandchallenge/rules2.htm

Posted by MattJ at March 15, 2004 09:01 AM

"Darpa wants a military vehicle that can keep up with M1s and Bradleys."
Something like that ?
http://www.plustech.fi/Walking1.html

This aint designed for speed, but it could easily be adapted to. Or perhaps that:
http://www.rhex.net/

Legs are simply a better solution for rough terrain, particularly when the driver isnt guarnateed to be very smart.

Posted by at March 15, 2004 09:19 AM

At the very least, the highly publicized competition was good PR for DARPA, an agency that could stand to improve its public image. DARPA is a frequent victim of conspiracy theory nuts who think that DARPA is involved with Area 51 alien technology.

Robots, even billion dollar robots, are just toys. Creating a near perfect robot is probably an unattainable goal. It is not scientifically valid to say it yet, but one day somebody will quantify the old saw that says, "The creation cannot be greater than the creator." I may be wrong, but we are probably a long....long....long way from the "Johnny Cab" seen in TOTAL RECALL.

The DARPA race is probably little more than an interesting taxpayer funded stunt, but doesn't mean that we should not keep trying. - Jim

Posted by Jim McDade at March 15, 2004 09:57 AM

"Robots, even billion dollar robots, are just toys"

You could not be more wrong.
http://www.usnews.com/usnews/issue/040315/biztech/15eerobots.htm
http://roboticnation.blogspot.com/

Posted by at March 15, 2004 10:58 AM

The problem with testing is that no matter how well you have planned things out, unexpected events happen. I expect that many of the participants had "overplanned" for all possible contingencies, and had for that reason added features to the original specs. As any programmer knows, add in a few more features to the mix, and they start creating more problems.

In a lab situation, and even in a testing situation, you can never encounter all possible scenarios. I expect that the government will have to run this race several more times to get a winner. And I'll make a prediction - the simplest machine will win.

Posted by Linda at March 15, 2004 01:25 PM

the contest could be viewed as 100% sucessful, if you believe a rumor about the true purpose for it.
Basically, DARPA has been throwing lots of money into CMU's robotics research, and was getting pissed at the lack of progress. The contest was mainly ment to send a message to CMU.
seems to have worked {g}

Posted by at March 15, 2004 11:06 PM

no-name: I notice that the leg-havin' device you link to is made for forests. And is only shown in nice, cleared, vaguely flat forests.

Having been in the area of California and Nevada taht the test route ran through, I can tell you there is no way in hell such a device could make it. Now, perhaps one of the bipedal types I've seen demos of could... in a decade or two. Right now, from the video I've seen, they want a flat, smooth surface and IIRC a cable to an external control computer (though that's not a problem if you scale up, admittedly).

Balancing is "tricky", and I'm not at all sure legs give you really any great advantage over, say, a 6 or 8 wheel articulated system, for example. (They might give an advantage over a solid-frame 4-wheel system, if only by being more "flexible", but it looks a lot like the costs outweigh the benefits at this point, especially given the DARPA requirements for a usable combat vehicle.)

(Plus, re. keeping up with a Bradley or M-1, I cannot countenance the idea that a walking machine with current or near-future technology can do 60+ MPH over rough-ish ground... Perhaps in a decade or so. And what advantage it would have over wheels would seem to be at least partially balanced by the relative inefficiency of legs vs. the ability to roll. But I confess I haven't studied the area. Am I wrong about this?)

Raoul: Why bother to run it, if nothing's going to complete it? Well, apart from the "proving it's very hard" part others have mentioned, it at very least isn't any worse than another test run, is it? Lessons to be learned, data to be acquired and processed, etc.

Posted by Sigivald at March 16, 2004 12:03 PM

You would think that a more sensible short-term goal would be a robot that could participate in a convoy with a lead human driver.

Clearly, no one had done enough testing.

Interestingly, the self-funded project that came in third (a Toyota pickup with high-resolution stereoscopic vision plus miscellaneous) only got stuck when it was manually paused - overriden - to let a wrecker pass. It stopped with one wheel in a depression and couldn't figure out it needed to apply higher throttle to start moving again. Pity.

Posted by mike earl at March 17, 2004 07:57 AM

Why do robot builders always try to tackle the hardest problems (stereoscopic vision) when the goal doesn't require it? Distance sensors (sonic or laser) work very well at defining the local conditions for navigation.

Then make the robot stable by design. How about a large clear ball with a wheeled robot inside? (So it's laser sensors then?) Make sure it can tip itself out of any expected hole or incline and you're there.

Posted by at March 17, 2004 01:09 PM

Sonic and laser sensors have a very hard time detecting barbed-wire fences, which apparently were the end of several robots.

Posted by mike earl at March 17, 2004 02:31 PM

I was at the QID (qualification, inspection, demonstration) trials in Fontana as well as at the starting line (and the hoped-for finish line) ... and the first answer to the question (why not run a full-up test in advance?) would be that a good number of the teams knew they weren't ready to go the distance... They were still tinkering with the systems right up to the end. All that goes double for the guy with the motorcycle.

I suppose the biggest impediment would be ... who would let a driverless vehicle carom over any stretch of 100-plus miles without supervision from the authorities? DARPA says it had 80 Marines and hundreds of other personnel on the route watching out for stray civilians, desert tortoises, etc. Some of the teams tried to do advance tests on tracks, on frozen lakes, etc., but that's just not the same.

The best-prepared team was CMU's Red Team - the favorite, and the team whose vehicle went the farthest. They basically rented out the Nevada Automotive Test Center to run simulations the week before, and that's when they suffered the rollover.

You could pretty much tell after the QID that no one was going to finish the course. I would have bet that the Red Team's Sandstorm vehicle would go farther. But I think the folks running the show saw the Grand Challenge as a success, not a failure. It was very cool to see these vehicles take off (or even wander around the starting chute) without a driver. To me, it's amazing that several of the vehicles, even vehicles that didn't have the voluminous corporate sponsorships that the Red Team did, could end up going 5 to 7 miles on their own.

By the way, DARPA says they plan to talk with the teams about the thought processes they went through, pay them for their time, and apply the innovations they saw during the race. I definitely didn't get the sense that the DARPA folks were going through all this just to show how undoable the task is. DARPA says they plan to do this again, maybe with a shorter course, in 2005 or 2006. DARPA Director Tony Tether noted that the permits they were granted for running the race are good for the next four years.

Posted by Alan Boyle at March 17, 2004 06:04 PM

Some of you need to hang out at racetracks more, because you're completely missing the point. The million-dollar prize was obviously not going to come close to covering the cost of building the vehicles, so the contest was always going to be about a mixture of pride and publicity.

Having a race with a deadline made this helped both of these causes, as can be seen by the news coverage that the Grand Challenge received. With the element of competition, there's accomplishment in just qualifying for the race, much less beating half the teams by moving a mere 500 yards. The underfunded and inventive nature of the teams is apparent to most people, which means that even going one mile nets sponsors positive press in the relevant communities.

I think it's a good start to government prizes, and that there's a lot of future potential here.

Posted by Jake McGuire at March 17, 2004 10:20 PM

Jake, I've hung out at lots of race tracks. In my younger (and dumber) days, I even raced SCCA events in an MGA.

I never attended a race in which none of the contestants could make it a quarter of the way around the course, forget about winning.

A good government prize is one that is realistic, and doesn't have to wait for an arbitrary date to win. The Longitude Prize took decades to win, but there was no insistence that it be performed on a specific date against its competitors.

Posted by Rand Simberg at March 17, 2004 10:33 PM

Rand, I can't tell you what happened to the other teams, but I can
tell you what happened to Sandstorm. (I am not a roboticist, but I am
a computer scientist, and a friend of mine is one of the lead
engineers on Sandstorm.)

As Alan Boyle points out, most of the teams there were not expecting
to complete the course; there were only a few that were realistically
trying to win, including the Red Team. Naturally they tested the
vehicle. In fact, at a test a few days before the official race,
Sandstorm went (if memory serves) 120 miles in comparable terrain.
But, there are a lot of unknown variables (other than just terrain),
and it's hard to predict which ones are going to be significant. The
one that ended up killing Sandstorm was the width of the course DARPA
gave them.

The race was not merely to get from point A to point B; the vehicles
were also required to stay within a particular course. They were
allowed to use GPS to track their compliance with the course. In all
their tests, Red Team ran with a fairly wide course. Unfortunately,
the vehicle was never tested with a course narrow enough that the
margin of error in GPS was a factor.

In the race, DARPA gave them a course not much wider than the vehicle
itself, so the error of the GPS was a significant factor. (This was
apparently in an attempt to be helpful: the course was narrow enough
to miss all major obstacles.) This revealed a bug that had never come
out in testing: they did not compensate properly for the error in
GPS. As a result, Sandstorm found itself adhering tightly to a course
a few meters off from where it was supposed to be. The vehicle
correctly identified the obstacle that killed it, but thought that it
was not allowed to steer around it.

Those are the specifics about what happened to Sandstorm, but I think
the real point is more general. When you don't know what the
circumstances will be, you cannot necessarily run a test under similar
circumstances. There are too many unknown variables, and you can't
test against every combination. DARPA could have eliminated this
particular problem by announcing the width of the course in advance,
but no one knew in advance that this variable was the key one.

Posted by Karl Crary at March 19, 2004 08:12 AM

It is interesting how many people were willing to accuse the competitors of "zip, nil, nada testing" when there was a lot of evidence *readily available* to the contrary.