The September 2022 issue of IEEE Spectrum is here!
IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy.
Intelligent infrastructure makes autonomous driving safer and less expensive
This test unit, in a suburb of Shanghai, detects and tracks traffic merging from a side road onto a major road, using a camera, a lidar, a radar, a communication unit, and a computer.
Enormous efforts have been made in the past two decades to create a car that can use sensors and artificial intelligence to model its environment and plot a safe driving path. Yet even today the technology works well only in areas like campuses, which have limited roads to map and minimal traffic to master. It still can’t manage busy, unfamiliar, or unpredictable roads. For now, at least, there is only so much sensory power and intelligence that can go into a car.
To solve this problem, we must turn it around: We must put more of the smarts into the infrastructure—we must make the road smart.
The concept of smart roads is not new. It includes efforts like traffic lights that automatically adjust their timing based on sensor data and streetlights that automatically adjust their brightness to reduce energy consumption. PerceptIn, of which coauthor Liu is founder and CEO, has demonstrated at its own test track, in Beijing, that streetlight control can make traffic 40 percent more efficient. (Liu and coauthor Gaudiot, Liu’s former doctoral advisor at the University of California, Irvine, often collaborate on autonomous driving projects.)
But these are piecemeal changes. We propose a much more ambitious approach that combines intelligent roads and intelligent vehicles into an integrated, fully intelligent transportation system. The sheer amount and accuracy of the combined information will allow such a system to reach unparalleled levels of safety and efficiency.
Human drivers have a crash rate of 4.2 accidents per million miles; autonomous cars must do much better to gain acceptance. However, there are corner cases, such as blind spots, that afflict both human drivers and autonomous cars, and there is currently no way to handle them without the help of an intelligent infrastructure.
Putting a lot of the intelligence into the infrastructure will also lower the cost of autonomous vehicles. A fully self-driving vehicle is still quite expensive to build. But gradually, as the infrastructure becomes more powerful, it will be possible to transfer more of the computational workload from the vehicles to the roads. Eventually, autonomous vehicles will need to be equipped with only basic perception and control capabilities. We estimate that this transfer will reduce the cost of autonomous vehicles by more than half.
Here’s how it could work: It’s Beijing on a Sunday morning, and sandstorms have turned the sun blue and the sky yellow. You’re driving through the city, but neither you nor any other driver on the road has a clear perspective. But each car, as it moves along, discerns a piece of the puzzle. That information, combined with data from sensors embedded in or near the road and from relays from weather services, feeds into a distributed computing system that uses artificial intelligence to construct a single model of the environment that can recognize static objects along the road as well as objects that are moving along each car’s projected path.
The self-driving vehicle, coordinating with the roadside system, sees right through a sandstorm swirling in Beijing to discern a static bus and a moving sedan [top]. The system even indicates its predicted trajectory for the detected sedan via a yellow line [bottom], effectively forming a semantic high-definition map.Shaoshan Liu
Properly expanded, this approach can prevent most accidents and traffic jams, problems that have plagued road transport since the introduction of the automobile. It can provide the goals of a self-sufficient autonomous car without demanding more than any one car can provide. Even in a Beijing sandstorm, every person in every car will arrive at their destination safely and on time.
By putting together idle compute power and the archive of sensory data, we have been able to improve performance without imposing any additional burdens on the cloud.
To date, we have deployed a model of this system in several cities in China as well as on our test track in Beijing. For instance, in Suzhou, a city of 11 million west of Shanghai, the deployment is on a public road with three lanes on each side, with phase one of the project covering 15 kilometers of highway. A roadside system is deployed every 150 meters on the road, and each roadside system consists of a compute unit equipped with an Intel CPU and an Nvidia 1080Ti GPU, a series of sensors (lidars, cameras, radars), and a communication component (a roadside unit, or RSU). This is because lidar provides more accurate perception compared to cameras, especially at night. The RSUs then communicate directly with the deployed vehicles to facilitate the fusion of the roadside data and the vehicle-side data on the vehicle.
Sensors and relays along the roadside comprise one half of the cooperative autonomous driving system, with the hardware on the vehicles themselves making up the other half. In a typical deployment, our model employs 20 vehicles. Each vehicle bears a computing system, a suite of sensors, an engine control unit (ECU), and to connect these components, a controller area network (CAN) bus. The road infrastructure, as described above, consists of similar but more advanced equipment. The roadside system’s high-end Nvidia GPU communicates wirelessly via its RSU, whose counterpart on the car is called the onboard unit (OBU). This back-and-forth communication facilitates the fusion of roadside data and car data.
This deployment, at a campus in Beijing, consists of a lidar, two radars, two cameras, a roadside communication unit, and a roadside computer. It covers blind spots at corners and tracks moving obstacles, like pedestrians and vehicles, for the benefit of the autonomous shuttle that serves the campus.Shaoshan Liu
The infrastructure collects data on the local environment and shares it immediately with cars, thereby eliminating blind spots and otherwise extending perception in obvious ways. The infrastructure also processes data from its own sensors and from sensors on the cars to extract the meaning, producing what’s called semantic data. Semantic data might, for instance, identify an object as a pedestrian and locate that pedestrian on a map. The results are then sent to the cloud, where more elaborate processing fuses that semantic data with data from other sources to generate global perception and planning information. The cloud then dispatches global traffic information, navigation plans, and control commands to the cars.
Each car at our test track begins in self-driving mode—that is, a level of autonomy that today’s best systems can manage. Each car is equipped with six millimeter-wave radars for detecting and tracking objects, eight cameras for two-dimensional perception, one lidar for three-dimensional perception, and GPS and inertial guidance to locate the vehicle on a digital map. The 2D- and 3D-perception results, as well as the radar outputs, are fused to generate a comprehensive view of the road and its immediate surroundings.
Next, these perception results are fed into a module that keeps track of each detected object—say, a car, a bicycle, or a rolling tire—drawing a trajectory that can be fed to the next module, which predicts where the target object will go. Finally, such predictions are handed off to the planning and control modules, which steer the autonomous vehicle. The car creates a model of its environment up to 70 meters out. All of this computation occurs within the car itself.
In the meantime, the intelligent infrastructure is doing the same job of detection and tracking with radars, as well as 2D modeling with cameras and 3D modeling with lidar, finally fusing that data into a model of its own, to complement what each car is doing. Because the infrastructure is spread out, it can model the world as far out as 250 meters. The tracking and prediction modules on the cars will then merge the wider and the narrower models into a comprehensive view.
The car’s onboard unit communicates with its roadside counterpart to facilitate the fusion of data in the vehicle. The wireless standard, called Cellular-V2X (for “vehicle-to-X”), is not unlike that used in phones; communication can reach as far as 300 meters, and the latency—the time it takes for a message to get through—is about 25 milliseconds. This is the point at which many of the car’s blind spots are now covered by the system on the infrastructure.
Two modes of communication are supported: LTE-V2X, a variant of the cellular standard reserved for vehicle-to-infrastructure exchanges, and the commercial mobile networks using the LTE standard and the 5G standard. LTE-V2X is dedicated to direct communications between the road and the cars over a range of 300 meters. Although the communication latency is just 25 ms, it is paired with a low bandwidth, currently about 100 kilobytes per second.
In contrast, the commercial 4G and 5G network have unlimited range and a significantly higher bandwidth (100 megabytes per second for downlink and 50 MB/s uplink for commercial LTE). However, they have much greater latency, and that poses a significant challenge for the moment-to-moment decision-making in autonomous driving.
A roadside deployment at a public road in Suzhou is arranged along a green pole bearing a lidar, two cameras, a communication unit, and a computer. It greatly extends the range and coverage for the autonomous vehicles on the road.Shaoshan Liu
Note that when a vehicle travels at a speed of 50 kilometers (31 miles) per hour, the vehicle’s stopping distance will be 35 meters when the road is dry and 41 meters when it is slick. Therefore, the 250-meter perception range that the infrastructure allows provides the vehicle with a large margin of safety. On our test track, the disengagement rate—the frequency with which the safety driver must override the automated driving system—is at least 90 percent lower when the infrastructure’s intelligence is turned on, so that it can augment the autonomous car’s onboard system.
Experiments on our test track have taught us two things. First, because traffic conditions change throughout the day, the infrastructure’s computing units are fully in harness during rush hours but largely idle in off-peak hours. This is more a feature than a bug because it frees up much of the enormous roadside computing power for other tasks, such as optimizing the system. Second, we find that we can indeed optimize the system because our growing trove of local perception data can be used to fine-tune our deep-learning models to sharpen perception. By putting together idle compute power and the archive of sensory data, we have been able to improve performance without imposing any additional burdens on the cloud.
It’s hard to get people to agree to construct a vast system whose promised benefits will come only after it has been completed. To solve this chicken-and-egg problem, we must proceed through three consecutive stages:
Stage 1: infrastructure-augmented autonomous driving, in which the vehicles fuse vehicle-side perception data with roadside perception data to improve the safety of autonomous driving. Vehicles will still be heavily loaded with self-driving equipment.
Stage 2: infrastructure-guided autonomous driving, in which the vehicles can offload all the perception tasks to the infrastructure to reduce per-vehicle deployment costs. For safety reasons, basic perception capabilities will remain on the autonomous vehicles in case communication with the infrastructure goes down or the infrastructure itself fails. Vehicles will need notably less sensing and processing hardware than in stage 1.
Stage 3: infrastructure-planned autonomous driving, in which the infrastructure is charged with both perception and planning, thus achieving maximum safety, traffic efficiency, and cost savings. In this stage, the vehicles are equipped with only very basic sensing and computing capabilities.
Technical challenges do exist. The first is network stability. At high vehicle speed, the process of fusing vehicle-side and infrastructure-side data is extremely sensitive to network jitters. Using commercial 4G and 5G networks, we have observed network jitters ranging from 3 to 100 ms, enough to effectively prevent the infrastructure from helping the car. Even more critical is security: We need to ensure that a hacker cannot attack the communication network or even the infrastructure itself to pass incorrect information to the cars, with potentially lethal consequences.
Another problem is how to gain widespread support for autonomous driving of any kind, let alone one based on smart roads. In China, 74 percent of people surveyed favor the rapid introduction of automated driving, whereas in other countries, public support is more hesitant. Only 33 percent of Germans and 31 percent of people in the United States support the rapid expansion of autonomous vehicles. Perhaps the well-established car culture in these two countries has made people more attached to driving their own cars.
Then there is the problem of jurisdictional conflicts. In the United States, for instance, authority over roads is distributed among the Federal Highway Administration, which operates interstate highways, and state and local governments, which have authority over other roads. It is not always clear which level of government is responsible for authorizing, managing, and paying for upgrading the current infrastructure to smart roads. In recent times, much of the transportation innovation that has taken place in the United States has occurred at the local level.
By contrast, China has mapped out a new set of measures to bolster the research and development of key technologies for intelligent road infrastructure. A policy document published by the Chinese Ministry of Transport aims for cooperative systems between vehicle and road infrastructure by 2025. The Chinese government intends to incorporate into new infrastructure such smart elements as sensing networks, communications systems, and cloud control systems. Cooperation among carmakers, high-tech companies, and telecommunications service providers has spawned autonomous driving startups in Beijing, Shanghai, and Changsha, a city of 8 million in Hunan province.
An infrastructure-vehicle cooperative driving approach promises to be safer, more efficient, and more economical than a strictly vehicle-only autonomous-driving approach. The technology is here, and it is being implemented in China. To do the same in the United States and elsewhere, policymakers and the public must embrace the approach and give up today’s model of vehicle-only autonomous driving. In any case, we will soon see these two vastly different approaches to automated driving competing in the world transportation market.
Shaoshan Liu is CEO of PerceptIn, an autonomous vehicle startup in Fremont, Calif.
Jean-Luc Gaudiot is a Distinguished Professor of electrical engineering and computer science at the University of California, Irvine.
smart roads seem to be more challenging and expensive, I think smart cars can handle roads after some technological advances better
it is not a permanent solution but a temporary one which time and technology will take care of it soon
Lowering the accident rate is certainly a worthy goal, but the real leverage of a smart infrastructure coupled an autonomous vehicle would be a smart highway. A smart highway could use the existing road infrastructure to carry 3-6 times more traffic per lane. The central system would actually take control of the car in these smart lanes allowing cars to drive a couple of feet apart at high speed. If a lead car sensed a hazard all the cars behind it would slow down in unison. Back of the envelope calculations suggest that you could increase the capacity of a lane 3-6X. It is conceivable to actually SOLVE DAILY COMMUTE TRAFFIC JAMS! This would benefit society so much more than just replacing drivers with autonomous cars.
Ray Liu on the importance of diversity and inclusivity
My great-grandfather was the kindest man I ever met. A self-taught accountant without formal education, he lived in the impoverished countryside of southern Taiwan. Legend has it that because members of the community viewed him as trustworthy, humble, and hardworking, they hired him to do their bookkeeping. He escaped a future of being a struggling farmer and was able to build a successful business.
He was also a vegetarian. While there were many special occasions and festivals in his sleepy village where meat was served, he always chose to eat his simple vegetarian meals away from others. I always wondered why. I came to learn that it was because my great-grandfather made a promise to Buddha.
Many years ago, my grandfather was a medical student studying in Japan. After a vacation home to Taiwan with my grandmother, they returned to Japan by boat, during which time he became very ill. World War II was raging, and medical supplies were limited. Unfortunately, by the time the necessary supplies reached my grandfather, it was too late. Later, my great-uncle went to Japan to study medicine and he also became quite sick. That was when my great-grandfather made a promise to Buddha: If you let my second son live, I will forever be a vegetarian to honor you. My great-uncle survived. My great-grandfather’s wish came true and until he passed away in his nineties, he never broke his promise, a simple vow between him and Buddha. A promise is a promise.
I strongly believe that integrity is the essence of everything successful, and character is what you do when no one is watching. Ethics serves as my ultimate guide for how I conduct myself both personally and professionally.
Because we are in a profession that often deals with innovation and safety, all members of our community—engineers, technologists, scientists, practitioners, and entrepreneurs—should have the highest ethical standards.
IEEE has recently revised its Code of Ethics, which encourages our members to strive to comply with ethical design and sustainable development practices. This is crucial given the global scale of the environmental, social, and political challenges that threaten to rapidly and critically impact the living conditions of current and future generations.
At the time of my grandfather’s death, my grandmother was pregnant with my mother. Because of this, she dropped out of medical school. During this era, a widow was expected to remain as such, even though my grandmother was only in her twenties. She chose to get remarried, an act that was unforgivable then. Thus, my mother was raised by my great-grandfather.
My grandmother was ahead of her time and dared to defy tradition. She was an outstanding athlete and an excellent student, with both ambition and potential. However, she lived during an era when women seldom received an education, not to mention the opportunity to attend medical school. In the end, tradition won. After remarrying, she had more children and became a housewife, not a doctor as she had planned. How many women in the world still face such obstacles?
Making progress in engineering, technology, and science is a global endeavor with worldwide implications. This progress is guided best by having an open, diverse, and inclusive mindset with the goal of developing and sharing innovative solutions for the benefit of all.
LinkedIn: https://www.linkedin.com/showcase/ieeepresident
Throughout my lifelong relationship with IEEE—my professional home—what I have treasured most is having the opportunity to befriend colleagues from around the world, from different cultures, with different beliefs, ways of life, and languages. My work, with IEEE and professionally, has taken me to several continents to collaborate and work with others. Through this global network I have been able to learn from the best and brightest minds in the world, and have come to greatly appreciate the diversity of cultures of IEEE members. Everyone has their own unique wisdom to offer.
IEEE continues its efforts to strengthen diversity and inclusivity across the organization and in the broader technological community. The establishment of the IEEE Diversity and Inclusion Committee as a standing committee of the IEEE Board of Directors follows the organization’s long-standing commitment to maintaining an environment in which all are welcome to collaborate and contribute to the community, to support the growth of the profession and colleagues, and to advance technology for the benefit of humanity.
Additionally, IEEE has devoted considerable effort in the past few years revising policies, procedures, and bylaws to ensure that members have a safe, inclusive place where all feel welcome. These activities help ensure all our members have full access to the benefits of membership, including opportunities for professional development and recognition.
As president, I made a commitment to help shape the IEEE of the future by examining ways in which the organization can evolve to best meet the needs of all technical professionals. And I made a promise to every member from all of our diverse groups and regions, especially women and others from underrepresented communities, that they will have a fair opportunity for participation and leadership in our professional home. And a promise is a promise.
Please share your thoughts with me at president@ieee.org.
This article appears in the September 2022 print issue as “A Promise Is a Promise.”
The Institute focuses on fascinating people and the technology they create
Harry Goldstein is Acting Editor in Chief of IEEE Spectrum.
IEEE members recently featured in The Institute [clockwise from top left]: Arti Agrawal, Ramneek Kalra, Jack Dongarra, Sandra Johnson, and Michael Kagan.
The Institute has appeared, off and on, in the pages of IEEESpectrum since debuting as a four-page insert in the December 1976 issue. At various times during the ensuing decades, The Institute—or, as we call it in Spectrum’s offices, “TI”—has been published as a stand-alone monthly and then quarterly broadsheet. It even had its own website for a few years, before being integrated into Spectrum’s site in 2019 and appearing in the print magazine as a section on a quarterly basis.
While Spectrum dives deep into emerging technologies and delivers expert voices from the bleeding edge, The Institute has focused on IEEE members, featuring their stories, celebrating their accomplishments, and telling them about IEEE products, services, elections, and volunteer opportunities. The Institute also brings members all the news relevant to the functioning of the association, such as the reconfiguration of its geographic regions to ensure there is equitable representation across its global membership.
The Institute staff—Editor in Chief Kathy Pretz and Assistant Editor Joanna Goodrich—say they are most excited about showcasing the diversity of IEEE’s membership and highlighting those who have made IEEE their professional home. Pretz says there’s a misconception that most of IEEE’s members work in academia. That’s why The Institute has put more emphasis on profiling the careers of those working in industry, such as Taiwan Semiconductor Manufacturing Co.’s senior vice president of R&D, Yuh-Jier Mii.
“I’m always amazed how accessible top leaders at high-tech companies are when I tell them I would like to interview them for an article in IEEE’s member publication,” Pretz remarked recently. “And all of them are so humble about their accomplishments. TSMC’s Yuh-Jier Mii was no different.”
Pretz and Goodrich also write about the careers of young professionals who are the future of the organization. In this month’s issue, they spotlight Eddie Custovic, the recipient of the IEEE Theodore W. Hissey Outstanding Young Professionals Award.
“It’s very rewarding to be able to interview members who have developed technology that I use every day.”
In addition to putting a spotlight on the accomplishments of individual engineers, The Institute also covers the history of technology. “It’s very rewarding to be able to interview members who have developed technology that I use every day, as well as members who are developing the next breakthrough tech that will impact society,” Goodrich says. “For example, Steven Sasson invented the first digital camera, which we use every day in some shape or form. Eddie Custovic, on the other hand, is working to develop an AI platform that will help solve food shortages that have been predicted for 2050.”
Indeed, IEEE members have for decades been turning their ideas into successful businesses. That’s why TI features entrepreneurial members who have launched their own ventures, like IEEE Fellow Alex Bronstein of Embryonics. And, of course, TI celebrates major anniversaries and milestones, like the 25th anniversary of IEEE Women in Engineering.
In future issues, readers can look forward to a profile of computer pioneer IEEE Fellow Erol Gelenbe, whose invention of the packet-voice telephone switch made Zoom possible. In December, look for a piece on IEEE executive director Stephen Welby, who is leaving the organization at the end of the year. Kathy, Joanna, and I invite you to celebrate these engineers and their extraordinary accomplishments with us in the coming months.
The prevalence and complexity of electronics and software in automotive applications are increasing with every new generation of cars. The critical functions within the system on a chip (SoC) involve hardware and software that perform automotive-related signal communication at high data rates to and from the components off-chip. Every SoC includes general purpose IOs (GPIOs) on its periphery.
For automotive SoCs, GPIO IP is typically developed as Safety Element out of Context and delivered with a set of Assumptions of Use. It is important that the GPIO blocks are treated as a safety related logic. In this role, GPIOs need safety analysis to mitigate any faults occurring in them before the result of fault occurrence causes a system-wide failure.
This white paper describes some of the commonly used safety mechanisms in an automotive-ready GPIO library suite. It will then describe how safety related deliverables are helpful to SoC integrators in their design of safe SoCs.