October 4, 2019
CARLA democratizes autonomous vehicle R&D with free open-source simulator
Enter CARLA, a free, open-source simulator powered by Unreal Engine that has been designed from day one to support the development, training, and validation of autonomous driving systems.
The birth of an open-source simulator
The initial inspiration for CARLA came from earlier work of Research Scientist Germán Ros, who is now CARLA Team Lead, and Professor Antonio M. López of the Computer Vision Center in Barcelona. Ros has been working on projects related to autonomous vehicles since 2013, while Lopez has several decades of experience in the field of autonomous systems.In 2015, while finishing his PhD at the University of Barcelona, Ros, under the supervision of López, started working on SYNTHIA, an environment for autonomous driving. Since its initial release in 2016, SYNTHIA has become well known in the research community, especially in projects that have to do with perception. However, because of the licensing restrictions associated with the real-time engine on which SYNTHIA was based, the original team was unable to freely distribute the project or provide access to the source code.
While presenting SYNTHIA in Las Vegas at CVPR (Conference on Computer Vision and Pattern Recognition), Ros and López were approached by Vladlen Koltun, Senior Principal Researcher and Director of the Intelligent Systems Lab at Intel. Koltun revealed that he admired the team’s work, but would like to see something similar that could be made openly available to anyone. Ros and López agreed wholeheartedly, and the CARLA project was born with Intel as the key sponsor.
The desire to freely distribute CARLA was a key factor in the new team’s choice of Unreal Engine, which is also free and comes with full source-code access. “We really wanted to share it with people,” says Ros. “We wanted people to be able to modify everything. So the idea was to have something that was completely open, and have access to all the source code of the engine and the source code of our platform and everything. We really believed that having something that is totally open is helpful for the community, since it enables them to adapt it for different use cases and scenarios.”
Starting from scratch, they built a new simulator, taking inspiration and lessons learned from SYNTHIA. At this point, Ros’ day job was at the Toyota Research Institute, where he was working on simulation projects using different tools. The lessons he learned from this research also influenced CARLA.
Ros was able to work on CARLA during the day thanks to Toyota’s support of open-source initiatives. “They were happy with that because everyone was learning out of this experience,” says Ros. “If the tool didn't work, we would learn why, and if the tool actually worked, we would use it. So it was a win-win situation.”
CARLA was first officially released in 2017 at the Conference for Robot Learning (CoRL). It was very well received by the community, which was rapidly evolving in response to the growing interest in autonomous driving. Today, CARLA has a dedicated development team of about 25 members around the world, who continue to put out new releases every two or three months, and a user base of more than 1,600 active users, split between academia and industry. Ros claims that virtually every university that is currently researching autonomous driving is using CARLA in some way, and so are virtually all corporations large enough to have an R&D unit. Some corporations are even using CARLA as their main simulation platform, although not all of them are open about it, preferring to keep it as a “trade secret.”
Democratizing autonomous vehicle research and development
From the beginning of CARLA’s development, the team understood the importance of the open-source model in helping it democratize autonomous vehicle travel.“Having the progress of autonomous driving be dependent on just the huge corporations with big pockets is not good enough,” says Ros. “We also need academics and small companies to participate in this if we really want to expedite and accelerate autonomous driving. We want autonomous driving—we think it’s important, it’s going to save lives, it’s going to make our lives better—so why don't we try to make it happen as soon as possible? For that, we need the collaboration of the community.
“One of the main goals of CARLA is to bridge that gap so we can democratize part of this race, such that even a small team in the middle of a small university in a small country can participate in cutting-edge research and development for autonomous driving, even though they don't have any access to a large fleet of vehicles or a large collection of data. Having this simulator for all to share helps to put together new algorithms and solutions.”
A second of CARLA’s goals was to give people who couldn’t afford the time or budget, or who couldn’t find available talent to build a fully staffed simulation team, the ability to build on an existing platform instead of starting from scratch.
“Maybe you shouldn’t focus on creating your own simulation team,” says Ros. “If you can take a tool that is out there that is good enough, and extend it if you need to, instead of having to come up with your own simulation team with twenty engineers, maybe you can take CARLA and work on top of it, and just hire one or two people.”
Finally, the team wants CARLA to serve as a common language. “OK, you can have your own internal simulator, you can have very sophisticated tools, and that's fine,” says Ros. “At some point you're going to need to share with your competitors, you’re going to need to share results with legislators. Why don’t you use an open-source tool for that—one that speaks all the standards that need to be spoken and allows you to send your data to the community in a transparent way so that they can understand its current status?
“That's why we decided to deliver CARLA in an open-source format where you can take everything from the assets, to the code, to everything, and do anything you want with it, including commercial use. Whatever you want, you’re free to take CARLA and do the best you can with it.”
Addressing the need for validation and verification
While the autonomous driving community is healthy and active, its members are a long way from having all the answers, which is where CARLA comes in.“The domain at the moment is messy,” says Ros. “In 2016, everybody thought that autonomous driving would be solved by 2019, that the only thing they needed to do was to simulate trillions of miles. Then people started considering things like validation and verification. And I think at some point they realized that they were not so close to production, so simulation became more useful for research and development, for expediting their R&D cycle testing, validating ideas, and developing algorithms very fast.
“I think people realized that for full-stack verification and validation, it’s a bit more complicated. Maybe they weren't really ready for running trillions of miles and maybe that didn't make much sense at the time. A couple of companies like Waymo and Uber have the strength to do this sort of full-stack verification. But the rest of the OEMs and tier ones are still trying to determine the right combination of tooling for doing that.”
Verification is managed by legislative bodies such as NHTSA (National Highway Traffic Safety Administration) and the US Department of Transportation. But even these bodies need help to determine the right tools and metrics to accomplish this effectively and safely. The CARLA team have been in conversations with them.
“Through CARLA, we're trying to help them to understand and to discover how to come up with this definition,” says Ros. “It’s an interesting time, but it’s also a bit chaotic: the question is still out there, how people are going to really use this tool to validate their stack. We're happy that so far it has been helpful in the process of research and development, because it gives people an opportunity to try things fast that will take months or years to do in a real vehicle—compare that to quickly using a simulator, and you can understand the problems that you're going to have if you implement that in the real car.”
The CARLA Autonomous Driving Challenge
To further contribute to the fair validation and verification of automatic vehicle software stacks, the CARLA team have introduced the CARLA Autonomous Driving Challenge.The challenge requires participants’ autonomous driving agents to reach a target destination following a predefined route without incurring traffic infractions. Along the route, agents face challenging traffic situations modeled after pre-crash traffic scenarios collected by NHTSA, such as avoiding unexpected obstacles, recovering control after losing it due to bad road conditions, and crossing traffic running a red light at an intersection.
By testing participants over the same routes, traffic situations, and weather conditions, the challenge is able to offer a fair comparison between different stacks. This is in stark contrast to the voluntary reports that OEMs and startups are currently able to make to NHTSA, where they can declare various statistics of what their vehicles have been doing in the past year.
“When you see those numbers and you try to compare them, try to get the big picture of the scene, it's just not possible because those numbers are not calibrated,” says Ros. “So for example, if a small company declares that they had five interventions over 1,000 miles, OK that's totally possible, but where are those 1,000 miles? Was their vehicle just going around a block, without traffic, without traffic lights, in very idealistic scenario? How do I compare those results with someone at a different company that had 45 interventions across 200 miles? Maybe those 200 miles were more challenging. Maybe those miles were in training conditions.”
The CARLA Autonomous Driving Challenge has proved to be a great success; this year’s awards ceremony took place at CVPR, one of the top conferences for machine learning and AI.
The latest release: more realistic pedestrians
A key part of CARLA’s value is its free-to-use assets: urban layouts, buildings, vehicles, and pedestrians. When the team redesigned the entire platform between versions 0.8 and 0.9, pedestrians were temporarily lost. The latest 0.9.6 release, unveiled in July, not only brought back pedestrians but added a whole new AI layer, introduced new children characters, increased variations for existing models, and sported a new interface that offers control over every bone in the skeleton hierarchy. It’s this last aspect that is particularly exciting to Ros.
“You can create very complex choreographies,” he says. “We think that that's going to help a lot in the area of human machine interaction, for example to help improve the field of forecasting. Forecasting is one of the most important modules of an AV stack. You need to be able to tell what a pedestrian or a vehicle is going to be doing in the next five or 10 seconds. Out of the current and the past history of two or three or five frames, you need to be able to predict what that pedestrian is going to do next. Is he going to cross, or is he going to stay on the sidewalk? For that you really need to have very advanced clues—you need your pedestrian to be acting in a very realistic way with their body language. The full skeletal control makes that possible.”
There’s no doubt that there is more to come from the CARLA team, which continues to strive to offer the best possible simulator to anyone who wishes to use it, modify it, or build on top of it. For free.
Interested in finding out how you could use Unreal Engine 4? Get in touch and we’d love to start that conversation.