Diesel-gate, and the testers who found VW's $33bn feature.

This is the story behind the VW emissions scandal, that so far has cost the company over $33bn. We look into the technology issues VW faced and the investigations that uncovered the problem. Scroll down for full transcript! Resources used to research and compile this podcast include: Wikipedia: Volkswagen emissions scandalhttps://en.wikipedia.org/wiki/Volkswagen_emissions_scandal VW Logohttps://commons.wikimedia.org/wiki/File:Volkswagen_Logo_till_1995.svg IEEE: How They Did It: An Analysis of Emission Defeat Devices in Modern Automobileshttps://ieeexplore.ieee.org/document/7958580 Meet the Man Who Brought Down Volkswagenhttps://time.com/4119981/the-man-who-brought-down-volkswagen/ ICCT: EPA's notice of violation of the Clean Air Act to Volkswagenhttps://theicct.org/news/epas-notice-violation-clean-air-act-volkswagen-press-statement WVU: In-Use Emissions Testing of Light-Duty Diesel Vehicles in the United Stateshttps://theicct.org/sites/default/files/publications/WVU_LDDV_in-use_ICCT_Report_Final_may2014.pdf Daniel Lange (DLange), Felix "tmbinc" Domke: The exhaust emissions scandal („Dieselgate“)https://www.youtube.com/watch?v=d9HJw3AUvGk THE VW NOx EMISSIONS GROUP LITIGATION IN THE HIGH COURT https://www.judiciary.uk/wp-content/uploads/2020/04/VWJudgment-002.pdf James Liang: Rule 11 Plea Agreement https://www.justice.gov/opa/file/890756/download U.S. v. Volkswagen, 16-CR-20394https://www.justice.gov/usao-edmi/us-v-volkswagen-16-cr-20394 Exhausted by Scandal: ‘Dieselgate’ Continues to Haunt Volkswagenhttps://knowledge.wharton.upenn.edu/article/volkswagen-diesel-scandal/ Faster, Higher, Farther: The Inside Story of the Volkswagen Scandal Kindle Editionhttps://smile.amazon.co.uk/gp/product/B01N5VDD2I/ Volkswagen Diesel Old Wives' Tale 6 Diesel is Dirtyhttps://www.youtube.com/watch?v=RMFaBXiBaZA Audi Green Police A3 TDI Ad (Super Bowl XLIV 2010)https://www.youtube.com/watch?v=GemJWrp0nAM  Show Transcript: Pete Houghton (00:00):Hello, and welcome to investigating software. In early 2014, the independent council on clean transportation asked a group of testers based at the West Virginia university to do a study on diesel engine emissions. Their budget was tiny, just 50,000 us dollars, In exchange they would locate and comprehensively road test emissions from three European cars that are used in the United States. So unlike previous tests where our scientists testers used the standard lab equipment, while the cars were on rollers, they would take these cars out on the road. They tested the free cars on the actual roads around Southern California, and they even took one car on a road trip up to Washington state. That's a journey of almost 4,000 kilometers in total. Our tester-scientists collected this off cycle road, test data and published it on may the 15th, 2014. The results of those relatively cheap road tests would result as of June, 2020 in over 33 billion us dollars in fines, penalties, financial settlements, and costs for the Volkswagen group. Pete Houghton (01:06):This is a story of VW diesel gate and the $33 billion feature. Before I get into the technology software and testing and alike, I'll quickly outline what happened over the decade before the EPA called out VW in September, 2015, in 2006 Volkswagon was working on its new diesel engine EA 189 to support a new line of diesel cars. It hoped would boost it's flagging us sales. Historically Volkswagen had some success with it's beetle or bug as it was affectionately called as well as the VW camper van. They had been hippie icons back in the seventies, but post millennium, the VW brand hadn't done so well in North America while other European brands like BMW had made headway at the sort of premium end of the market. So in 2006, the recently promoted chief executive of VW Martin Winterkorn devised a plan to grow global sales from around 6 million cars a year up to 10 million.Pete Houghton (02:00):It was one of those stretch goals designed to take VW up to the top tier and would involve beating its principal rivals in a top spot like GM and Toyota. The largest car market in the world at this time was the United States. It would be a few years before China overtook the U S as the biggest market. So at that time it made perfect sense that any attempt to boost overall sales would involve trying to boost sales in the U S one of VWs biggest problems was a different focus in the U S markets while their diesel engines sold well in Europe, especially since the perceived fuel economy of diesel helped on a continent with high fuel prices. The U S market had much stricter air quality regulations. And as we will see later, stricter enforcement in the U S market Toyota led the field in fuel economy with the Prius an affordable hybrid car from a manufacturer with a reputation for quality. Pete Houghton (02:49):Now, this is before the negative publicity for Toyota, from the sudden unintended acceleration publicity starting in around 2009. So that left VW sort of boxed in while its diesel engines were relatively efficient. Why would people switch from buying a Toyota Prius from a trusted mainstream brand? Also, unlike in Europe, diesel wasn't widely used for domestic vehicles in the U S so they'd need to overcome a natural bias from people who'd only ever bought petrol cars, the solution deliver an economical and environmentally friendly vehicle under the motivating banner of clean diesel. Here's an example of a Superbowl advert that epitomizes the marketing strategy. They used diesel was no longer a reason not to get the car. In fact, save the planet. Advert (03:33):Tragedy strikes tonight where a man has just been arrested for possession of an incandescent light bulb. What do you guys think about plastic bottles now? The water setting is at 105. "Green Police" song... You got a TDI here? Clean Diesel, You're good to go sir! "Green Police" song... Pete Houghton (04:02):It worked VW saw a dramatic increase in sales from 2010 through 2012, not just in its diesel cars, also in its petrol cars and the perception of their cars as clean and economical help to win over many environmentally concerned customers. In fact, they were still pushing the clean nature of their diesel cars in the second half of 2015. Here's an extract of an advert from their old wives tale series of adverts. Advert (04:29):How do you like my new car? Isn't diesel dirt say it's beautiful for Christ's sake. I think it's beautiful, but aren't, diesel's dirty. Yeah, that's true. Oh, that used to be dirty. This is 2015. No, no, no. Listen, to me Terry diesel in Latin means dirty. I'll prove it to you. You're going to ruin your scarf. Oh, I'll look what she's doing. See how clean it is. Pete Houghton (04:53):Interestingly, these adverts were being aired even after the EPA and carb had privately presented the off cycle emissions data that they were worried about to VW and asked them to explain what was going on. Just a quick note. EPA stands for the environmental protection agency in the United States and CARB stands for the California air resources board, sort of local version of the EPA focused on California. So what are these emissions tests that the EPA and CARB do and how come the problem with emissions didn't come to light as soon as the cars were first released or even before then? Well, the car emission testing process is actually quite complex and takes into account the desires of multiple stakeholders. For example, in the US the fuel efficiency and emissions rules have taken into account at various times in their history, the wishes of the United auto workers, that's a labor union in the US to limit the import of smaller, more efficient foreign cars. Pete Houghton (05:50):The concerns of the national highway traffic safety administration, or NHTSA about the increased risk of from accidents involving some smaller cars, their desire of manufacturers to produce a range of vehicles of different sizes, powers, and class, Oh, and obviously the harmful effects of emissions themselves on people and the wider environment to get a better handle on these, not so simple emissions rules in more detail, let's inspect that second to last rule, the desire of manufacturers to produce a range of different types of vehicles. Let's say you want to get into the car business. You've designed your new car already. It's called the Goliath. It's a big, heavy gas guzzler with all the inefficient extras, but you've got a problem that Goliath is never going to pass those new emissions standards. So what do you do? Luckily, the emissions rules have your back. You only need to meet the standards on average, across all the cars you sell. Pete Houghton (06:44):Technically it's, what's called a harmonic mean, but it has a similar effect. So the answer is to release so much smaller and more efficient car as well. We'll call it the super mini David. So at the cheap and clean end of the market, we have the super mini David and at the gas guzzling expensive end. We have the Goliath. Luckily for us EPA, won't look at the Goliath's emissions in isolation. They'll take into account that you sold a bunch of smaller, cheaper, and more importantly, cleaner and more efficient. Supermini Davids. That's done on a year by year and per company basis. So each year your company can produce plenty of gas, guzzling, Goliath cars. As long as you get enough, poor environmentalists to buy your supermini David. But unlike the biblical story that our cars are named after our supermini David's are not slaying Goliath, they are literally justifying its existence without the clean efficient car, the inefficient dirty car would just not be legal. Pete Houghton (07:35):That's the basic idea behind the CAFE or corporate average fuel economy rules in the U S I've simplified them a bit, but the basic principle is true, and it's not just the US that you, you have that sort of similar system in place. Now it's not all bad. The rules do tend to tighten up things like fuel economy and emissions slowly over time. And some would argue the increase in car efficiency or MPG has been at least in part due to these sorts of regulations. So as you can see, regulating and therefore testing emissions can be complicated. Each of those new models of super mini cars has to be certified appropriately to ensure it all balances out overall. And of course the engines are just subject to the laws of thermodynamics, and there's only so much wiggle room a manufacturer has when they develop a new car. Pete Houghton (08:19):So for finding out what the emissions of your new cars actually are, you need testing, and this is where the US and that you, you differ. They both have similar standards as we've seen, and they have similar concepts of the sort of tests that need to be applied to the cars, but in the U S the testing is done by the relevant state and federal agencies like the EPA or CARB in California in Europe, your car company can go out and hire a favorable company to certify new vehicles. Europe has historically not had the power to ensure companies actually comply with the regulations while the EPA in the US has a history of enforcing auto makers to comply and fining those that don't. But despite the EPA has greater powers and its history of enforcement initially did fail to pick up the issues that would later dog VW. It was this nonprofit called the international council on clean transportation, or ICCT that hired the team of scientist-testers from West Virginia university. A small grant from the ICCT was just enough to enable the team to Jerry rig the mobile test and equipment they needed. And to hire the three vehicles they needed, all three cars were diesel and two were from VW. And the third came from BMW. Here's a clip from John German, the guy at the ICCT who hired the testers. John German (09:32):I worked for international council on clean transportation. We're a nonprofit research organization. We deal primarily with government regulators worldwide, mostly in developing countries do have an office in Europe. Um, and we... We've been working on this for at least six years. And we've been trying to fill in the holes. There's a lot of reasons why these emissions might be high. Pete Houghton (09:54):What was different about these tests was that although they would do the usual mix of suburban, rural, uphill, downhill & highway driving as used in the standardized tests, these tests were on the road in normal traffic and in normal weather conditions. When you're out on the road, there's a lot more going on for a start the road isn't necessarily straight, and you have to make the odd turn, the other cars, aren't all moving at the same speed. And your journey might not take the same amount of time. Every time you go out. And also the weather is changeable. For example, the weather up in Seattle was slightly cooler and wetter than the weather they experienced down in Los Angeles. When they were testing their, the standardized tests used by the EPA and CARB were the exact opposite. The rules were published in advance, and they had fixed values of speed, time and distance and standard temperatures and humidity. Pete Houghton (10:41):So you kind of get the picture about how one was very homogenous and in real life, they were much more variable due to just the everyday conditions. Now it makes sense these tests were published and standardized, at least in part, it would be a bit unfair for the manufacturers. If they had no idea what the tests were going to be. I mean, hypothetically, they might engineer a car that was super clean and efficient for suburban use in say residential middle America, but unbeknownst to them, the test turned out to be all Hill climbs in icy conditions. That wouldn't actually be fair. So hence the standards from a software point of view, these standardized checks would make great unit tests. So you could quickly test the car for serious EPA violations. Every time you made a change, but of course they wouldn't constitute all of your tests. Pete Houghton (11:25):And of course you can't run the car on the rollers and do an emissions test. Every time you make a code change, the team from West Virginia university are all scientists, and you can tell from seeing their interviews, they aren't idealists. They understand they're dealing with machines, subject to the laws of thermodynamics built by imperfect humans and configured to meet the hurdles they expect to see out in the real world, they do to flee, perform their on-road and exploratory tests using their jury rigged equipment. They'd carefully calibrated the test equipment against the standard dynamometer systems. Back in the workshop, the dynamometer tests are the standard tests done by the EPA, etc, and involve the car running on rollers under sort of control conditions. What these exploratory road tests showed was that two of the three vehicles exceeded the Knox and mission standards between 15 and 35 times in one vehicle and five and 20 times in the other. Pete Houghton (12:14):Now the testers didn't know why this was the case. In fact, they doubted their own results. They repeatedly calibrated their mobile jury rigged equipment and the cars against the standard dynamometer equipment and subjected their cars to the standardized tests to check. If they still passed, the cars kept passing the standardized tests, but the off cycle or on road tests showed new concerning information. The report from the West Virginia university is a masterpiece. The results are detailed, comprehensive, and full of written information that will help you recreate the same experiments they performed. For example, there are maps of the routes they took, tables showing the speeds the car was going and even details about the time of days that the journeys took place. They also fully detailed the cars under test giving details of the engines, power, size, emissions control technology, and the particular class of emissions standard. The car should belong to. They also recorded the cards, weight drive type, and previous mileage. It's impressive note taking. I mean, they even detailed the weights of the test equipment, they'd loaded into the cars. The report also included the details of the machines used to detect the emissions while the cars were moving. The report also try to match some of their on road tests to those include on the EPA standardized tests. For example, the report states this about one of their tests Report (13:29):Essentially represents the Los Angeles route four, which was ultimately used in developing the original FTP vehicle certification cycle with some minor modifications at locations where the traffic pattern or roads have changed since the FTPs development. Pete Houghton (13:45):This is clever. They were not only testing the cars, but they were testing the tests themselves and allowing us to see how the standard dynamometer tests back in the workshop compared with their direct real world counterparts. The engineers reported their findings that basically two of the 3 cars had broken the NOx emission standards by a wide margin when used actually on the roads the third car had generally done. Okay. And had only gone over the limits on a Hill climbing section of the tests. That was the BMW. An interesting point is that the report doesn't specify the manufacturers of the cars tested. It uses the anonymous terms, vehicle A, vehicle B and vehicle C who, if she looked fairly through the document, you could deduce that at least one of the cars was a Volkswagen as there's a reference in the report that refers to some test procedures for the Volkswagen diesel engine, but it isn't mentioned explicitly. Pete Houghton (14:36):So the team didn't exactly advertise what they worked on. It was pretty low key, and they were pretty clinical about the whole thing. It's this report that provided the initial evidence, that something wasn't quite right with the VW engines. Again, at this point, I'm sure that some people suspected that the cause was nefarious, but the cause could in theory, just have been an accident or a fault that perhaps plagued this particular model of car or these particular versions of those cars. With so few people testing the cars and the car test critically. It's not surprising that the problem had laid hidden for years. Each new car was just subjected to the same standards required for the EPA and its Californian counterpart CARB until the relatively tiny sum of $50,000. That's less than the sticker price of vehicle C in the test was spent by the ICC T to try and test a little better only then did we learn something new and important. So what was actually going wrong in their engines, or when you drive a modern car, we still use the same controls we've used for decades, steering wheels, pedals, and the dials on the dashboard have little changed, but behind the scenes, all those taps to the pedals and now just messages fed into a computer, the computer, hear's your call for more acceleration, let's say, and allows more fuel and air into the engine so far. Pete Houghton (15:51):So good. That's what we want. But the modern diesel engine can't be just left to its own devices. Unfortunately, diesels on sales are dirty and produce either Soot hazardous to health or NOx, which are oxides of nitrogen gas, also toxic and hazardous to the environment as well. In fact, they will typically produce both and other pollutants like carbon monoxide as well. So things initially look good for diesel in the age of tightening vehicle emissions, but luckily there were solutions to all of the above issues. So firstly, the soot that's just made up of tiny particles, similar to the smoke you get from a garden fire or from cigarettes even it's easy enough to create a filter that will clean out. Most of those particulates. The problem though is like with all filters, eventually it will get clogged up and full. And in this type of filter, that means it's going to become a lot less able to catch the harmful particles and they'll just pass straight through out the exhaust, but clever engineers have a solution. Pete Houghton (16:41):They can switch the engine into what's called active regeneration mode. This actually happens automatically when you drive the car faster on a motorway or a freeway and in effect, it burns off the particles emptying the filter. Unfortunately though this active regeneration mode actually increases other pollutants like NOx. So what about these NOx gases? How do we solve that problem? Or there are a few options here and I'll run through them. Firstly, there's EGR or exhaust gas recirculation, a fancy name. That just means a percentage of the gas coming out the exhaust is sent back through the engine to be burnt again. That's great for reducing NOx that causes more soot to be produced. Then secondly, there's LNT or Lean NOx Trap. That's like the particle filter I mentioned, but this one is for the toxic gas NOx. It stores the NOx and periodically regenerates itself to get rid of the toxic gas it's already stored. Pete Houghton (17:34):But this regeneration mode, which is different to the other filter based one, produces more soot, see the pattern. Also this regeneration isn't very efficient. And as the regenerations are every few seconds or at most, every few minutes, it has a noticeable hit on fuel economy. Now this third method is the most effective it's called the selective catalyst reduction or SCR. This involves a special catalyst or converter that converts the toxic NOx gas into water and nitrogen two things that occur just naturally in the air. Anyway. So they're pretty harmless. A downside is that it requires a tiny amount of an additive to be sprayed into the conversion chamber. As the car is being driven. That means we need to squeeze a big tank of this additive into your car somewhere. Also, the additive needs to be topped up every few thousand miles. Another chore for the owner that they obviously wanted to avoid. Pete Houghton (18:22):They could make the tank bigger for the additive and fill it up as part of a routine servicing. But that means losing some boot space to the bigger tank and that makes the car look less attractive to prospective buyers. Luckily you as a driver, don't have to decide about how and when to use these techniques. The car's engine will automatically just adjust the settings depending on the cleaning technology installed in your car, the engine's computer or ECU constantly reviews the status of the engine, the filters, the speed, the commands you're giving it balancing the needs of fuel economy emissions and things like engine wear what the study by the team at West Virginia university uncovered was that sometimes the balance was just right when they were doing the standardized tests, for example. But sometimes the balance seemed to be way off like when they did the road tests, but remember one vehicle, the BMW didn't do too badly on the road tests. Pete Houghton (19:12):So it was technically possible to marry the objectives of fuel economy and emissions. It's just that two of the cars didn't seem to be doing that. And a telling sign was that the cleaner BMW use the same emissions technology as used in one of the Volkswagens. They both used the diesel particle filter and the additive consuming selective catalysts reduction techniques. This suggested the cause of the difference. It was not the emissions controlling hardware in theory, but instead the software in the engine that controlled that hardware. So what happened next? Now there was evidence that something at least was a mess EPA and carb notified Volkswagen of the anomaly and asked them to look into it. Now it could just have been a bug. The code and configuration in the engine's control system can have bugs and mistakes. Just like any other app, like an app on your phone or on your desktop computer. Pete Houghton (19:58):But by then, CARB had done its own test of the Volkswagens and seen similar issues. But again, they didn't know the cause. Was it a fault or was it something more sinister later confessions and investigations, would show that VW engineers had asked Bosch who supplied the engine's computer to use what they labeled and acoustic function. When the car was driven in certain ways, from the point you turn the key, the computer for example, would keep track of the car speed and compare it to its library of known standardized engine tests. If we potted these tests on a graph, they would look like narrow corridors in a maze within which the driver has to keep the car from going either too fast or too slow. The maze have sudden bands and plateaus where the test might simulate the car, stopping at a junction, and then starting up a few seconds later when the lights are changed, when the car hacker an engineer, Felix Domke examined the engine code from the VW computer and overlayed the speeds. Pete Houghton (20:51):He saw India, acoustic function, and those of these standards emissions test, they matched almost perfectly the acoustic function was what VW had codenamed a defeat device. There's a great, IEEE paper where Felix shows the graphs of the data in the computer, super imposed on all the standard emissions tests you see that they almost perfectly match. So what was happening was the defeat device compared the driver's speed and behavior against a whole selection of standard driving tests. And if it noticed the speed wasn't within the narrowly defined rules in this database, it disabled much of the emissions technology and switched to a dirtier more polluting mode. Why? Because that dirty mode usually involved greater fuel economy reduced wear on the exhaust or in fact reduced use of those exhaust cleaning additives. Remember they don't want people to use quite so much of that. So they don't have to go back into the garage, a chore. They think that buyers won't want to do in late 2015, the EPA issued Volkswagen with its now famous notice of violation. And as the EPA website describes it, EPA (21:54):The notice alleges that Volkswagen installed software in its model year, 2009 to 2015, 2.0 liter diesel cars that circumvents EPA emissions standards, these vehicles emit up to 40 times more pollution than emission standards allow Pete Houghton (22:12):Over the following weeks and the inquiries don't deeper. They found issues not only in Volkswagen and Audi, but also Porsche vehicles. Now you may ask why was Porsche affected? Porsche is actually a sister company of Volkswagen along with several others and the companies share the same parts and technology. For example, the Volkswagen Touareg and the Porsche Cayenne are very similar vehicles and share many potty parts and engine components. Yes, including the same diesel engine with VW signature emissions defeat device. As we can see here while reuse can be great for creating a simpler more homogenous production process. That's true in both cars and software, it also means a bug or feature in this case can affect much more than just one machine. As the investigations continued. Other irregularities also came to light with other companies in the VW group, including Skoda and SEAT. While this wasn't the story of a bug, but rather a feature. Pete Houghton (23:03):It's also the story of some great testing. Our engineers / Tester / scientists at the West Virginia university developed cheap, effective tools and applied them in a way that uncovered massive problems that had placed our health and the environment at risk by spending just a few thousand dollars. ICCT enabled Dan Carder and his team of students and scientists at WVU to test the cars and compare that to the standardized tests, the industry and in particular regulators had been relying on for years. They built the tools needed to investigate and work the problem of how to get realistic emissions readings that enforced the spirit intended by the regulators. The testers also did clever things like try a variety of vehicles that were either different makes or use different emissions technologies. This let them examine the results of differences and gave us important clues about where to focus. Pete Houghton (23:55):But most of all, they took super notes and with their report detailing every aspect of the investigation and including recreation's steps, they presented the results in a clear and readable format that others could reuse. So how else could this have gone down? Let's imagine an alternate history where the same ethical blunders were made but a wise and profit focused executive had decided to hide their own testers to do much of the same on-road emissions tests. I'll assume a love of people's health, the environment, or a desire, to be honest, did not motivate the executive, but purely profit. This person would likely have found the same results and being rational. She knew something was wrong. If they had done this early enough, it could have steered the company onto a cleaner track cost of development may have risen, let's say an extra billion dollars in costs and maybe lost sales. Pete Houghton (24:43):Now Forbes has the cost of developing a whole new model car around $6 billion, but we just want to tweak the engine and add a bigger additive tank. So I think a billion would more than cover it. That would still save the company well over $32 billion in fines, penalties, and costs. That's not an environmental choice or a public health choice, or even a moral one. That's a clear cut financial one for product of any complexity, the cost of not finding out what's wrong, of not investigating and testing your products will probably exceed the money you think you're saving by not doing that testing. They were playing Russian roulette with the regulator and eventually everyone loses that game. The reason I mentioned this possible alternate history is back in 2012, a group of engineers noticed that in some customers' vehicles exhausts were failing too soon. The emissions technology was essentially wearing out, the engineers investigated and ultimately uncovered the cause a defeat device. Pete Houghton (25:38):As it turned out the defeat device didn't always work. Sometimes a driver might end up staying in the defeat mode longer than VW had planned. The car was inadvertently clean. That is the bug, Was that the car didn't cheat enough. The engineers actually appeared surprised at the defeat device they'd found and they convened a meeting with their supervisors and handed over a document outlining what they'd found I'll quote the exhibit two statement of facts from the VW rule, 11 plea agreement, paragraph 48 as to what happened. Next note, this is a statement agreed by VW, not just the U S government, although Plea Agreement (26:15):They understood the purpose and significance of the software supervisors, A and E each encouraged the further concealment of the software, specifically supervisors A and E each instructed the engineers who presented the issue to them to destroy the document they had used to illustrate the operation of the defeat device software. Pete Houghton (26:36):So that illustrates my final point. It's unfortunate Volkswagen made a huge ethical blunder. It's almost worse when some of their own engineers debugged a serious issue and uncovered the problem. Their management has shut up. If you want to catch bugs or even inappropriate features, you need a culture that promotes the discovery of problems. It has to be seen as a good thing. And part of the honest practice of engineering and software development, as our heroes at West Virginia university show, it's not particularly expensive and it can save you a fortune. Thank you. I'm Peter Houghton. And you've been listening to Investigating Software.