Archive for October, 2015

concentrated technology

Sunday, October 18th, 2015

If you were to launch a new mail server right now, many networks would simply refuse to speak to you. The problem: reputation.

Email today is dominated by a handful of major services. […] It’s become increasingly unusual for individuals or businesses to host their own mail, to the point that new servers are viewed with suspicion.

And so it goes for most technology discussions: decision-making power ends up concentrated in the hands of a small handful of organisations, often to a point where those who would like to broaden their horizons are unable to.

The experience with email merely demonstrates that it’s not just an issue of standards compliance.

software security

Sunday, October 18th, 2015

On Volkswagen (again):

The explanation was at least mildly plausible, initially, though, because a modern high-end car is staggeringly complex. It requires something like a hundred million lines of code, about two hundred and fifty […] times the number of lines in the Space Shuttle. No one could know every line of that software, making it theoretically possible that engineers could have sneaked in the emissions-defeating protocol without Volkswagen’s upper management knowing. Microsoft engineers did something like that decades ago, when they slipped a flight-simulator game into the shipping version of Excel 1997.

But on Wednesday, Spiegel issued a report, based on one of the many investigations taking place at Volkswagen and around the world, saying that at least thirty managers were involved in the cheating. This squares with Barton’s skepticism, not to mention common sense. Volkswagen engineers didn’t smuggle in software that allows you to play Tetris on in-car G.P.S. screens. They wrote code that fundamentally changed how the company’s diesel cars worked. The altered software affected engine emissions, mileage, cost, and power—all things that auto executives care about. In other words, while it’s technically possible to install such software, it’s hard to imagine that it could have gone unnoticed. Modern automobile engines are made by teams that design, build, test, and tune everything to produce the desired effect. Companies have been building these engines for more than a hundred years, refining a process the leaves no room for mysteries or magic outcomes. When a car produces more power, there is a reason; when a car produces fewer emissions, there is a reason. And when, at Volkswagen, its diesel engine produced forty times more nitrogen oxide when it wasn’t being tested than when it was, many people inside would have known why.

Here’s another thought, if Volkswagen’s executive team were completely oblivious to what was going on: perhaps senior management at Volkswagen have a reputation for really not liking bad news. In that case, the level of sophistication and coordination required to effect such software starts to make a little sense, although what doesn’t make sense to me in that context is the idea that executive management can be so disconnected from the organisation they run.

(And if they are that disconnected – then that raises other, very significant, concerns in itself.)

In a powerful book about the disintegration, immediately after launch, of the Challenger space shuttle, which killed seven astronauts in January of 1986, the sociologist Diane Vaughan described a phenomenon inside engineering organisations that she called the “normalisation of deviance.” In such cultures, she argued, there can be a tendency to slowly and progressively create rationales that justify ever-riskier behaviours. Starting in 1983, the Challenger shuttle had been through nine successful launches, in progressively lower ambient temperatures, across the years. Each time the launch team got away with a lower-temperature launch, Vaughan argued, engineers noted the deviance, then decided it wasn’t sufficiently different from what they had done before to constitute a problem. They effectively declared the mildly abnormal normal, making deviant behaviour acceptable, right up until the moment when, after the shuttle launched on a particularly cold Florida morning in 1986, its O-rings failed catastrophically and the ship broke apart.

If the same pattern proves to have played out at Volkswagen, then the scandal may well have begun with a few lines of engine-tuning software. Perhaps it started with tweaks that optimised some aspect of diesel performance and then evolved over time: detect this, change that, optimise something else. At every step, the software changes might have seemed to be a slight “improvement” on what came before, but at no one step would it necessarily have felt like a vast, emissions-fixing conspiracy by Volkswagen engineers, or been identified by Volkswagen executives. Instead, it would have slowly and insidiously led to the development of the defeat device and its inclusion in cars that were sold to consumers.

Except, that software development in and around complicated systems doesn’t work that way.

Sure, incremental development is definitely a known — sensible, even — approach, but to write code that successfully and reliably identifies particular road conditions isn’t just “a few lines of […] software” — it requires consideration, planning, design, and testing. This would have come around once it was clear that the engine profile for drivability and the engine profile for the environmental tests were so far apart, and isn’t something that even a small handful of rogue engineers could reliably knock together.

Certainly, multiple engine maps are plausible: many vehicles have them. Selecting a timing and fuel delivery map based on particular conditions — transmission gear selected; throttle setting; engine RPM for example — is quite common.

However to even consider heading down that path when it comes to the more complicated scene of detecting a rolling road — particularly when the only reason to do so is to cheat surely would have caused at least one team member to have a sleepless night or two.

In summary, it’s still implausible to me that the executive management team at Volkswagen were completely oblivious to what was going on.

a further thought on volkswagen

Saturday, October 10th, 2015

Indeed.

corporate crap, aka a couple of software engineers

Friday, October 9th, 2015

Volkswagen — and their misleading and deceptive behaviour — are all the news at the moment. Diesel engines run in a continuum, with variables like particulates, fuel consumption, power output, and nitrous oxide (NOx) production in play, with a summary that to minimise particulates, fuel consumption, and reasonable power output, the production of nitrous oxides result.

In large diesels, such as those found in trucks, buses, and so on, this is dealt with by injecting a urea solution into the exhaust, where a catalyst neutralises the NOx component of the exhaust. This solution adds weight and complexity, as well as the requirement to “refuel” the urea solution regularly, so it isn’t often found on small diesels.

People though don’t tolerate smoky diesels, particularly in small passenger vehicles. The addition of a diesel particulate filter theoretically addresses some of this, but better to minimise the particulates anyway. The added bonus of minimising particulates is lower fuel consumption, and more usable torque and power. NOx aside, low fuel consumption, low soot, and higher power output is what customers do want.

Which brings the question of how small diesel manufacturers have been passing the extremely strict emissions standards, which care more about NOx than particulates, given that NOx is a main ingredient in photochemical smog. As is now turns out, at least one manufacturer — Volkswagen — has been passing the tests through duplicity: running a “fuel-rich,” probably particulate-heavy, NOx-light engine profile when under test; running a leaner, particulate-light, NOx-heavy engine profile for normal driving.

Give a one-dimensional metric to an engineer, and they’ll find a way to ‘optimise’ that metric. Measure ticket closure rates in a support environment, and tickets will be closed as quickly as possible — probably quicker than would result in happy customers.

However, I don’t for a moment believe that “a couple of software engineers” would have gone to these lengths unbidden; for starters, the test regime for such engine profiles would require significant coordination among many different people within the organisation, all the way up to — at the very least — a program manager. Dyno runs; tests to ensure the software can figure out the difference between a rolling road and a real one; the engine mappings themselves: all of this takes time, effort, expense, and coordination. If it were as simple as “a couple of software engineers,” the development costs associated with new cars wouldn’t be as large as they are in the first place.

Further, why would “a couple of software engineers” be too concerned about passing some environmental tests, unless they had been directed to be concerned about it?

My guess, having seen similar behaviours — even including tossing “a couple of software engineers” under the bus when caught — can be summarised like so:

  1. Diesel engine was developed.
  2. Diesel engine fails internal benchmarks against the relevant environmental standards.
  3. “A couple of software engineers” are tasked with fine-tuning the engine run profile to meet the environmental standards.
  4. Engine now passes benchmarks, but fails to provide the driving performance that would be preferred.
  5. “A couple of software engineers” are now tasked with finding a solution to this problem, with the inference that they still need to be able to pass the environmental standards.

Given this trade-off, the use of two separate engine maps, tuned for each use case — and additional code to determine which one to use — is almost the only logical outcome.

So yes, it probably was “a couple of software engineers” who wrote the code. It almost certainly couldn’t have been done without fairly sophisticated coordination across the whole product team.