This Robo-Truck’s Cameras See Twice as Far as Any Lidar

TuSimple's computer vision system can see as far as 1,000 meters—nearly triple the range put forward by most of its competitors, and a compelling answer to one of self-drivings most dastardly problems.
TuSimple

We’re heading southeast on Interstate 10, headed into Tucson, Arizona, when we pass the group of men in orange jumpsuits and hard hats working on the side of the highway. “Inmates Working,” the sign on the back of the truck parked on the shoulder says. It’s the sort of sight that can generate a swirl of curiosity, pity, and distaste in a person, but the robot doesn’t register anything about who these men are. It’s thinking about that parked truck and the rule buried in its code that says when something’s on the shoulder, it’s supposed to clear out of the right lane. Checking it has plenty of room, it makes a quick juke to the left, pulling its 18 wheels and four human passengers over the dashed white line.

This lane change is just one of a variety of impressive maneuvers TuSimple’s self-driving truck executes during a 40-minute demo ride I took with CTO Xiaodi Hou on a sunny Friday morning last month. Between a smooth merge onto the highway and an easy exit, it cruised steadily, adjusting its speed and position as necessary to accommodate its human neighbors.

Powering the impressive showing are the sensors studded around the truck, including two lidar laser scanners and a forward-facing radar. The key to the system, though, is the handful of cameras looking forward, to the side, and to the back. In an increasingly crowded robo-trucking field, TuSimple’s bid to stand out hinges on those cameras, which Hou says let the vehicle see as far as 1,000 meters ahead—nearly triple the range claimed by by most of its competitors. With them, Hou is trying to solve one of the most dastardly problems facing engineers who are trying to make vehicles that drive themselves.

LEARN MORE

The WIRED Guide to Self-Driving Cars

Hou has been working on computer vision since he was a graduate student at CalTech. After earning a PhD in the subject in 2014, the Beijing native fled academia (“more and more politics”) and started a company that applied computer vision to online advertising. That soon flopped, and he refocused his skills on autonomous trucking, founding TuSimple in September 2015. The company has headquarters in San Diego and Beijing, and a testing facility outside Tucson, benefitting from Arizona’s sunny weather and light regulatory touch.

While operating robots in cities remains an uncracked problem, putting them on the highway is a lighter lift. “Trucking is a relatively narrow scenario,” Hou says. Since big-rigs spend nearly all their time on the highway, they don’t have to worry much about how to navigate through complex urban scenarios, with their intersections and pedestrians and cyclists. “And it has a lot of potential to go to extremely high reliability,” he adds.

If, that is, the vehicle can make proper sense of its surroundings. Perception is likely the hardest part of self-driving, and in recent years, lidar has been seen as the top tool for the job. That laser-based system, though, has a limited range—the most powerful systems can detect objects out to 250, maybe 300 meters.

TuSimple expects to have its software functionally ready by the end of 2019, with a production launch not too longer after that.

TuSimple

Cameras have the promise to see much farther and don’t have the same finicky hardware and high costs that beleaguer lidar developers. They are more reliable, see farther, and offer excellent resolution, but come with their own challenge. It’s difficult for a computer to convert the 2D data they generate into a 3D understanding of the world. Writing the software that can distinguish shadows and reflections from real objects, or see where one object ends and another begins, is a brutally tricky task. Whether anyone can accurately use cameras to do things like distinguish between a traffic turtle and a human being about to cross the street remains unclear. But Hou, at least based on the demo drive I took with him, has managed to make cameras work for the far less complex problem of rolling down a highway and picking out what’s what.

As we roll into central Tucson, the I-10 gets complicated. It expands to four lanes, which branch off onto different highways. The rate at which cars are moving onto and off the freeway ticks up. The TuSimple truck, though, keeps its composure. I’m sitting in the back of the cab, with Hou to my left. To my right is a mini-fridge sized black box with cables running out of it—it holds the servers turning those camera feeds into computer-friendly data. As a safety driver sits behind the wheel and a TuSimple engineer rides shotgun, Hou shows me what his truck can see. On the monitor mounted between us, the system identifies other vehicles in a Skittles-y sprinkle of colors (assigned at random). It shows how far away each of them is, and how fast it’s going relative to the truck, which is rumbling along at 65 mph. Like an experienced human driver, the truck uses that data to inform its decisions. It decelerates to let slowish cars merge onto the highway ahead of it. It slides over for emergency vehicles. It safely passes slowpokes.

Cameras that see things 1,000 meters away is a bit of overkill, partly because it’s almost never possible to see that far without other vehicles or a turn in the road blocking your view. (At one point, though, the system detects a vehicle about 900 meters away, on an uphill straightaway.) The value of seeing past the 300-meter mark, however, is manifest. At 65 mph, you cover 300 meters in 10 seconds. That’s not much time for a lumbering truck to register a crash, lane closure, or stopped firetruck, make the decision to brake or change lanes, and execute it. Triple how far you can see, and you get a luxurious 30 seconds to make the move.

VCs already recognize the value of Hou’s vision—they’ve invested $83.1 million in TuSimple to date, according to Crunchbase. Now the company is building relationships with the shippers, truck fleets, manufacturers, and industry suppliers who will make putting its tech on the road possible. It hopes to have its software functionally ready by the end of 2019.

The demo drive also, however, makes clear there’s work to be done. After 40 minutes of, uneventful robo-driving, a computer voice interrupts our conversation: “Autonomous driving off.” Without my noticing it, the safety driver has retaken control. “I’ve got a freeze on my visual,” he tells Hou. “I shut it down for safety.” A network card in that server box next to me disconnected, likely due to the truck's vibration, freezing the system's human-machine interface and blocking the bit that does the motion planning from the bit that executes its commands.

And so this flesh-and-bones human holds the wheel and works the pedals for the next 20 minutes, taking us off the highway and back to TuSimple’s garage. The computer crash mars what started off as a very impressive ride, but hey, there are worse kinds of crashes.

  • What causes hangovers, and how can I avoid them?
  • A civilian's guide to Fortnite , just in time for season 7
  • The promise—and heartbreak—of cancer genomics
  • Waymo's so-called robo-taxi launch reveals a brutal truth
  • PHOTOS: Dress rehearsal for a mission to Mars
  • 👀 Looking for the latest gadgets? Check out our picks, gift guides, and best deals all year round
  • 📩 Want more? Sign up for our daily newsletter and never miss our latest and greatest stories