The Real Trolley Problem for Self-driving Cars

On March 18, 2018, Uber murdered a women with one of their (apparently not) criminally dangerous “self-driving” cars. The reason was that she was crossing the road outside a pedestrian’s crossing, so the car did not recognise her as a pedestrian (it didn’t expect people outside of walkways). Instead, it fluctuated between recognising her as a car, an unknown object and a bike, before finally recognising her as a person and slamming into her. The autopilot in the car is able to track objects persistently, but resets each time changed from one type of object to another, therefore failing to track her.

The internet has been quick to condemn Uber’s programmers for overseeing this obvious and trivial bug: of course pedestrians cross illegally, of course you should keep tracking objects even if they change type, and of course you should stop regardless of the type of object. Except, it is not that simple. The internet seems to be suffering from a good ol’ case of hindsight bias: seeing all the obvious signs after the fact. This ignores that there were thousands of false signals that would need to be processed at the same time. For example, people like to condemn the US government for overlooking the 9/11 attacks even though they received reports suggesting an attack might be imminent; this ignores that they receive many such reports each day and have to pick out the one credible threat from a sea of false threats, preferably without putting the nation in an emergency every day.

Similarly, the Uber error is not necessarily as easy to spot as all the brain-geniuses of the internet like to claim. The fault required three failures: object recognition fluctuating because the right answer was ruled out, object tracking failing because the object changed type, and the system not indiscriminately stopping when something was in its way. Each failure is easy to understand and probably non-fatal alone, but together they killed a woman. Independetly, all of these failures are also easily explainable. The system cannot just stop anytime it thinks there is an object in its way, as that would cause issues with paper or leaves blowing in the wind as they are objects in the way – perhaps even rain would trigger an emergency stop in the middle of a highway. The system needs to decide whether or not to stop for an object and here it fatally decided not to. The system decided to reset its persistent object tracking every time it changed type. This seems silly in isolation; why stop tracking an object just because it changed instead of remember previously collected information? Well, objects in the real world do not change from a bike to a car. Except due to optical illusions. For example, if a car is blocking view of a bike, overtakes it, and drives past the field-of-view of the autonomous system, revealing the bike. Now, a car has “changed” into a bike. Except it hasn’t, of course, one object disappeared and gave place for another. Uber’s system would correctly track this situation. Forcing it to track a single object changing type, would cause it to assign the bike a too high speed, potentially leading to introducing an entire new class of accidents. Finally, the system didn’t anticipate pedestrians outside pedestrian crossings. This is harder to defend depending on the details, but the flaw could also have occurred without this failure simply due to flaky object detection in general.

I am not defending Uber’s fault, I am just truing to impress on you the notion that the fault is not nearly as obvious as it would seem at first due to hindsight bias. Feature interaction is extremely complex and most if not all bugs I see in my day-job come from unanticipated interactions of systems that are otherwise well-written by qualified people. Even when performing a lot of testing, it is not possible to test all interactions, especially not manually and realistically (though emerging technologies like my own Model-based Testing Workbench does help). The Uber murder was caused by three concurrent failures of four systems with conflicting or at least unfortunately intersecting goals. While I wouldn’t necessarily anticipate the error being caught by the individual developers implementing the individual systems nor the owner in charge of the part of the system including the four, the Uber murder failure is simple enough that it should have been caught in testing. At least the part about jaywalking pedestrians not being recognized as pedestrians, though that would only reduce the risk of the failure, not eliminate it.

Even while the Uber murder failure should have been caught in testing, I also posit that it is impossible to catch all such murder failures in testing. Every non-trivial system will have failures. No matter if we don’t allow real-world testing to autonomic driving for the next 5, 10, 20 or 50 years, when the system meets reality, murders will happen. We can likely reduce the number of murders by increasing the time we wait, but never to zero, and there will be a diminishing return on time spent testing. It is not a matter of whether there will be another murder (Tesla already did a handful or so, I believe) but when.

This finally brings us to this post’s title: the trolley problem. It is a thought experiment at the intersection of ethics and psychology. In the experiment, a run-away train is certainly going to kill a number of people and by performing some action, the subject can change the outcome to instead killing another number of people. For example, the train might inevitably kill 3 people if the subject does nothing, but the subject can push a button to redirect the train to another track where it will instead kill 1 person; the question: is taking action to murder one person worth it to save three people? It is not a trick question and there is no clever “third option” to get out of killing people; what you see is all there is: if you don’t do anything people on one track will unquestionably be killed, if you do the suggested action, people on the other track will be unquestionably be killed saving the people on the other, if you do anything other than the suggested action, it will fail and the people on the first track inevitably and definitely die. The parameters can be changed to figure out where people’s limit for taking action to kill people are, which actions they are willing to perform (push a button to direct a train, pushing a person in front of the train, and similar death porn for psychologists with too much grant money), how many people to save/kill, the type of person, etc. Here you can try it for yourself (albeit only murdering cartoon people on the screen), and this video shows having people perform the thought experiment in reality:

The trolley problem has been mentioned with respect to autonomic cars because they will have to make that sort of decision. A normal driver has to make such decisions: crash your car into a tree to save a child on tricycle while risking your own life, get drunk into the car because you’ve “only had a few and can totally handle it” risking your own life along with anybody else on the road, speed thru a busy city center because you have an important meeting. A self-driving car faced with an impending crash has to make similar decisions: avoid people to save them while endangering others (either other people in traffic or people in the car). A fair solution would be to always murder the people in the car: they bought the speeding murder machine with insufficient intelligence to save them, they are first on the chopping block. This, of course, would not sell any autonomic cars, and, worse, would not cover all the cases where an autonomic car would murder people and cannot avoid it by killing its owners instead.

The problem is, the real trolley problem facing self-driving cars is not who should they kill in case of a crash (though that also matters and should be dealt with), it is the much more immediate problem of how many people should we kill to develop the technology? People argue that the technology will save lives in the long run because autonomic driving will be safer than human drivers. Maybe. Here, the people on the track where the train is currently heading is all the people that will be killed by human drivers in the future and people on the side track we can redirect the train to are all the people the will get murdered testing what is obviously currently a flawed technology. We can redirect the train at any time; the crowd on the side track gets thinner, so delaying redirection will kill fewer people on the side track. The people on the track the train is currently on seem to be constantly spread. When do we redirect the train? Immediately? In 5 year? 10? 20? 50? Never?

The trolley problem has the interesting characteristic that people are not “rational” when it comes to deciding who to murder. People would not murder two people to save three in general. People also don’t like to be close to the murder, and are willing to save fewer people to murder by pushing a button to redirect a train than they need to save to murder by pushing somebody in front of the train. The proximity to the people saved/murdered also matters, with family and friends ranking higher than foreigners and people similar to the subject (racially and social status) are also more likely to be spared. This all adds up to a very uncomfortable question that we would rather avoid. Avoid to the point that no psychology ethics board will likely allow such an experiment; the one in the video above was made by a Youtuber unaffiliated with a university (albeit attempting to make it as ethical as possible while still seriously fucking with people’s heads).

The problem is, we cannot avoid the trolley problem regarding whether to allow development of self-driving cars. One person has already died and more will die, regardless of whether we continue: either from the current track (regular traffic murders) or from the side track we are already veering onto (from testing wildly unready technology in dangerous situations). Since Uber, Tesla, Waymo and probably others are already testing on public roads, we have already started killing people for the technology. We also cannot just say that we accept no more casualties for the technology and that the companies should just test better on closed tracks; as has been illustrated, the complex feature interactions guarantees that people will be killed during real-life testing or during production run, no matter the amount of testing. The only way to stop testing from killing people is to permanently reject the technology. That is a legit answer to the problem, but at one extreme. Some companies have a financial motivation to move closer to the other end of the spectrum: Uber is burning thru VC money and has bet its future on self-driving cars instead of illegal taxis and needs this to happen before VC money runs dry. Tesla is perpetually 3 months from bankruptcy and depends heavily on a likely mentally unstable CEO who keeps pumping up the value by promising things that are not even close to not being within reach, they are not within eyesight yet. Companies like this need to show the technology is ready Real Soon Now and to make it so as fast as possible. Move fast and break stuff, in this case the spines and skulls of the innocent.

I don’t have too much of a horse in the race. I don’t really see a need for self-driving cars but also don’t see myself getting killed by one as testing is happening abroad (except…). In fact, I would not mind if cars were outlawed altogether. They murder thousands of people directly and more indirectly due to pollution. Electric cars and autonomous cars will always just be a band-aid on that broken leg (still killing people, just moving parts of the pollution elsewhere and keeping other parts). On the other hand, I also realise that others disagree and have a harder time getting by with a bike and public transport. Perhaps a compromise would be to revoke licenses for autonomous testing from anybody who does so recklessly or causes failures that might not be obvious, but should have been caught under safer testing conditions.

Uber should be forced to stop all public road tests as they have demonstrated that they are not ready for it. They should not be allowed to hide behind the driver, but be forced to take responsibility and fix their shit before they get back on the streets. And if they get back on the streets, they should be forced to do so in a safe manner. They have to find a way and not just put a random person off the street at minimum wage behind the steering wheel to take the fall. Failures should have real and significant consequences with fines at the level of GDPR measuring in percentages of world-wide revenue so failing just once is expensive and requires adjustments to earning forecasts but (maybe) survivable, whereas failing 10 times is not survivable for the company.

Tesla should be forcibly dissolved. They are hyping up their glorified cruise control to the point where people use it as autonomous driving. A 6 point footnote on page 83 of the manual (only available online) saying that you should perhaps stay awake while your completely safe and infallible self-driving car takes you anywhere without any risk will not do. Their failure of a CEO keeps promising it will be ready in 3 months, and billing people for the functionality. They are actively putting people in danger just to keep the hype going and avoid bankruptcy for another quarter. If forcible dissolution is not a option, they should be forced to put in proper precautions that cannot be circumvented, so people only use cruise control as that. And I’m not talking about wink-wink-nudge-nudge precautions that be circumvented, I am talking about holding Tesla 100% responsible for any accident on the level described for Uber if people circumvent it. And, of course, refund anybody buying the autonomous driving package under false pretenses.

Waymo can probably be allowed to continue testing. They seem to have been actually careful and avoid hyping the feature. Same with the established car manufacturers, which all have technology that is ahead of that of Uber and Tesla but carefully avoid overselling it and block abuse. Any indication this is not correct will mean they lose the license to test in public, of course.

So, that is the real trolley problem for autonomous cars: how many people are we willing to murder to develop the technology? I don’t have the answer and am happy I don’t have to make the decision. We do need the discussion, tough, because right now, we’ve already redirected the train and started murdering people, and we should perhaps stop doing that until we at least know what we are doing and are confident that the bloodshed is going to be worth it.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.