How many of the business processes that software manages were better before the software took over? MOST companies I've seen inside were a hodge-podge of bad/missing/sloppy paperwork and poorly documented or even understood guidelines and policies. Software amplifies everything about a business, both the good and the bad. The downside is that because so much is networked today, something "going wrong" can have much larger consequences (and faster) than a wrong decision carried out on paper 30 years ago.
Software amplifies everything about a business, both the good and the bad. The downside is that because so much is networked today, something "going wrong" can have much larger consequences (and faster) than a wrong decision carried out on paper 30 years ago.
I think that's the distinction James Kwak is trying to make. He isn't saying that the business processes (in this case, buying and selling stocks) was especially clean before software. He is saying that software makes the system faster (amplifying the effects of a poor business process, as you say) while removing a human buffer. When a human order-taker sees an unexpected zero on an order sheet he or she has enough common sense to ask the person handing it to him or her whether the figure is correct. Most financial software doesn't have that "common sense", so it's easy for inadvertent slip-ups or programming errors to cascade into firm-ending chain reactions.
Obviously, the solution to this isn't to go back to slow and inefficient human processes. I think the solution is to continue doing what we're starting to see today - software that asks whether we're sure we know what we're doing when it sees a variable that's out of line. In addition, I think that many software systems are way too monolithic. That is, every component of the system implicitly trusts every other component. I would like to see software broken up into smaller communicating components, where each component can make assertions about the input and output of other components, and components can fail without bringing down the whole system. The system I've seen come closest to this is Erlang's OTP, but there's no particular reason for that model to be limited to Erlang.
Come off it, you take an old school accounts department, put in any accounting software and boom, massive ball ache solved. The majority of the accounts department suddenly turns into credit control.
There's a very good reason why the enterprise software industry sells billions. Because they save 10s, 100s or 1000s of billions.
For custom software, that's what a good consultant/PM is actually for, to translate 'what we thought we were doing' into 'what we're actually doing' into the final product of 'what it should actually do'.
I once worked for a company that sold project and document management software. The strangest call I ever got was from a small 20 person company phoning us up and saying 'What is Sue supposed to do now, she doesn't have anything to do'. That's what you bought the software for, to save money. Added bonus, management gets massive strategic oversight. But in the end it's up to them to use it.
But... how much waste have they introduced as well? I think a lot of old 'water cooler' talk may have been amplified by IM, solitaire, email, etc - much harder to visibly detect the amount of waste introduced by these technologies, although I'm sure they've saved a lot as well.
Sure, but now I don't have to go to the water cooler. Talking about a movie with the coworker down the hall is now a parallel process that takes maybe two minutes of total typing time, and while I'm not typing at him, I'm working on other things.
Compare to the water cooler, where it might take me a whole minute just to walk down there. Then we sit around and shoot the shit for maybe five minutes. Total time spent, let's call it seven minutes during which nothing got done except some simian grooming.
While this is definitely true, the amount of damage those paper-based systems was limited. As the Knight Capital case showed, it is possible to lose millions of dollars a minute with poor software. That would not have been possible with human trades.
That would not have been possible with human trades
Tell this to Mizuho Securities, where a human sold 620,000 shares at 1 yen instead of 1 share at 620,000 yen. ~$300 million loss in a single botched trade. Humans at both Mizuho and the Tokyo Stock Exchange saw the trade and all declined to intervene because they assumed they had insufficient authority to countermand the trader.
Holy crap, The Atlantic has gone down hill in the past few years. It feels like they try to become the next Huffington Post. A couple of years ago it felt like a serious competitor to magazines like The New Yorker or The Economist. Then they rebranded themselves from The Atlantic Monthly to The Atlantic and started their "social media revolution".
This is an incredibly shoddy article, it even closes with an absolutely brain-dead "what if"... the way average high school students close their essays.
I'm a huge believer in the free market, but the economics of journalism really do seem to be pushing a lot of serious periodicals into a crappy direction. Salon was never excellent, but it was good at times. Now it's utter crap. Slate was pretty good, now it's sliding fast. The Atlantic? Yep, just as you say.
Most of these places are ditching the "serious researched journalism" and even the "serious political opinion pieces" and going with fluffy bloggy chit-chatty crap.
"Round tables" are getting more an more popular, especially round tables that are just emotional reactions to lifestyle news. Four women discussing birth control. Five guys talking about Breaking Bad (Slate: I'm looking at you for BOTH of these).
And now the Atlantic is headed down the huffpo route.
All markets "clear" at some price and some volume. It may just be an unfortunate fact that in the Internet age the market for real journalism clears at $0 for 0 volume.
Again, my philosophical / political inclination is to deny this, to say that there's never a market failure, that everyone can always buy the magical pony they want at a price that sounds fair...but journalism seems to be giving me really good reasons to not say that.
> I'm a huge believer in the free market, but the economics of journalism really do seem to be pushing a lot of serious periodicals into a crappy direction.
To throw out a glib generalization, it seems that the free market is best at producing things people want, not necessarily things people need.
> And perhaps the most mission-critical of all mission-critical applications are the ones that underpin the securities markets where a large share of the world's wealth is locked up.
Really? I don't even consider the stock market software as mission critical. I mean, Jane Street's software is written in OCaml...
Mission critical is software that controls lives. Airplane, spaceship, heart pacer, ...
Unless of course ones mission is to fly an aircraft, control a power station, connect a phone call, keep someone's heart beating... Just words... The above commenter has a point. Mission critical is also about the effective control of a physical process just as much as it can be about smooth flow of money (which is just another process).
In some ways the article title is misleading as it touches on only a fraction of the software that actually runs the world - there is a shit load of software actually running the world (incl. making physical stuff go) and from what I can see most of it does a decent job, in general (of course there is always room for improvement). Disclosure: I'm a process control programmer so am biased in my view as to what actually 'runs the world' :). Taking nothing away from the money processing side of the business.
So, here's the unpleasant, unspoken, often subliminal truth about lots of enterprise software. It's supposed to be about automation to increase efficiency. In reality, it's often largely about control. Software enforces a certain workflow and can be used to allow or disallow certain actions. It's a way to enforce the procedures in the 3 ring binder.
One recurring pattern I've seen in enterprise software projects is separation between users and the developers. Often, there is this game of "telephone" where the user's managers talk to a manager above them, then there might be another layer above before information can start heading back down to the devs. Often, one is strictly forbidden to talk to users, except in exceptional situations. This is because the project is largely about control, so restrictions on communication with the users is necessary, since many of the unstated goals have to do with the frustration of the user's wishes.
What's even worse is when this power of control is used in political infighting.
The trick is to find way to knife through a couple of those layers of management to the managers who actually know what should be done. You really can't ask most users or managers because they're just not helpful.
I found that's not so straightforward for consultants like me, because the manager that brings you in is embedded in the "layers," directly in the information path. People-skills were not my forté.
We got lucky. I got hired into a project that started out in research which had/has a very loose management. It's the only part of the org structure where shit flows uphill. That gave us quite a bit of leeway in terms of establishing a cross cutting team. We've since been transferred but so fair the culture seems to be holding.
The article tries to make a point and then at the end evades the conclusion:
> The only real solution is to acknowledge that computer programs are going to fail and try to minimize the damage they can cause in advance.
[the best way to do this is to not use the software in question at all]
No, the only real solution is to provide a meaningful warranty with software (rather than typical "no warranty" clauses). Regulation can enforce this and the price will have to be paid by the customer. We have put up with these crappy licenses for decades and the result is the buggy mess we have now and no way for customers to demand a working product. Mistakes will still be made, but they shouldn't harm the customer more than the vendor.
How often have you really encountered such buggy software that's cause you significant harm as a consumer? I've had annoyances and had to get refunds, though rarely and it's hardly ever caused any significant loss.
No way for customers to demand a working product? Which products don't work? How does consumer choice and refunds for completely broken software not take care of this?
The software referenced in this article was mostly built in house, perhaps some with enterprise licensing which you can be sure included negotiations covering reliability.
Regulation is not the answer. Software is just fine.
Vulnerabilities in widely-used software (e.g. Windows, MSIE) were often fixed much too late, so the customer was at a risk or had to use 3rd party software at an additional cost just to compensate for them. Botnets that exist only because of these vulnerabilities put every business on the web at risk.
Software is just as "fine" as any other product where no accountability exists.
How do you provide a warranty against someone actively trying to destroy something?
Doors don't have guarantees against someone kicking them in. Cars aren't covered against someone putting sugar in the gas tank.
What counts as a vulnerability?
How can a regulatory body possibly decide what was a "vulnerability" they should have known about and what was something unforeseeable? I guarantee you there isn't always clear line between the two.
What about when the vulnerability is exploited by a government who has greater resources than the software vendor?
Does Microsoft have to pay out on a claim when damage is caused because a certificate is forged by the US government?
> How do you provide a warranty against someone actively trying to destroy something?
You provide a warranty that states exactly what is covered and what not (and suitability for a particular purpose, which software licenses often deny completely).
> Doors don't have guarantees against someone kicking them in
They do, at least here in the EU you get differently rated safety doors at different prices. You know exactly what kind of minimum resistance they offer.
> How can a regulatory body possibly decide what was a "vulnerability" they should have known about and what was something unforeseeable? I guarantee you there isn't always clear line between the two.
Stupid Word macro and buffer exploits are very obviously grave mistakes made by the manufacturer, who should be held accountable.
> What about when the vulnerability is exploited by a government who has greater resources than the software vendor?
Why should that make a difference? If you cannot handle the responsibility you have as a vendor, then don't be one.
> Does Microsoft have to pay out on a claim when damage is caused because a certificate is forged by the US government?
No, because Microsoft's stuff works as intended in that case, the CA should be held liable.
>You provide a warranty that states exactly what is covered and what not (and suitability for a particular purpose, which software licenses often deny completely).
So software vendors state in their warranties that running this software on an internet connected device voids the warranty, and they stop offering patches and support. All support moves to unofficial third parties.
>They do, at least here in the EU you get differently rated safety doors at different prices. You know exactly what kind of minimum resistance they offer.
And that's an objective quantifiable measurement. Levels of software vulnerability are not.
>Why should that make a difference? If you cannot handle the responsibility you have as a vendor, then don't be one.
That is stupid, no software vendor can harden consumer software meant to run on a desktop internet connected computer to the point that a large government agency can't find a vulnerability.
> That is stupid, no software vendor can harden consumer software meant to run on a desktop internet connected computer to the point that a large government agency can't find a vulnerability.
I don't buy this. If the vulnerability is in the OS or some other program, those parts are to blame. But it's perfectly possible to avoid the mistakes that are typically employed for such intrusions (usually they are in the OS anyway - you know, the OS with no warranty, no accountability, but a monopolist price). When they aren't, it was usually some mediocre C programmer's sloppy coding (there are safer languages for mediocre programmers). The whole point of such regulations would be to put some pressure on developers so they must choose proper tools, languages, methodologies to keep customers from harm, whereas now they simply do what they like with no consequences.
No accountability? Microsoft is losing marketshare and mindshare to Apple and Linux because their products have been buggy, unreliable, and generally suck. They have revamped their company to attempt to address these issues. That's accountability.
Opening software vendors to unlimited liability for flaws would certainly push me out of the software business. Particularly in the fast moving entrepreneurial word, pushing improvements that sometimes are flawed is better for consumers than perfect software.
>No, the only real solution is to provide a meaningful warranty with software (rather than typical "no warranty" clauses).
Here's just a brief list of the problems I can think of off the top of my head.
What about paid beta software?
What small part-time software developer is ever going to release anything if he is legally obligated to provide support for x number of years.
Does some kid who bangs out a lolcats viewer for iPhone need to provide a warranty?
Who's going to make up the regulatory body that governs this? Ex industry people like in every other regulatory body? Guess who this favors --big companies like Microsoft who can easily afford regulatory compliance.
Software is inherently more complex than physical products, the vastly different hardware configurations of consumer PCs make it impossible to test everything. Manufacturers will only guarantee that the software works on x model Dell Desktop, and stop offering support for anything else to avoid regulatory problems.
Software is much faster moving than hardware, lack of regulations is one of the reasons.
The biggest problems consumers have with software is malware? Do you warranty against active destruction by a third party (like a car manufacturer providing a warranty against someone keying your car)?
There's nothing wrong with allowing exceptions for software clearly described (and sold) as "beta".
> What small part-time software developer is ever going to release anything if he is legally obligated to provide support for x number of years.
One who is careful enough to use a safe language, skilled programmers and possibly insurance. I fail to see how this is any different from a small vendor of electronic devices, who needs to comply with safety regulations and bear the risk of a recall due to a design problem.
> Does some kid who bangs out a lolcats viewer for iPhone need to provide a warranty?
Yes, just like he is accountable if he fits the lolcats viewer with a trojan on purpose.
> Who's going to make up the regulatory body that governs this? Ex industry people like in every other regulatory body? Guess who this favors --big companies like Microsoft who can easily afford regulatory compliance.
How is this different in other industries where regulations that demand sufficient quality and warranties exist?
> Software is inherently more complex than physical products
So you're saying that complex products should not come with a suitable warranty? How is an image viewer for the PC (no warranty, possibly vulnerable to specially crafted images due to buffer overflows etc.) more complex than a car (comes with warranty)?.
> Software is much faster moving than hardware, lack of regulations is one of the reasons.
I disagree. The software industry keeps reinventing and breaking things and hardware has progressed to a much greater degree in the past 20 years.
> Do you warranty against active destruction by a third party (like a car manufacturer providing a warranty against someone keying your car)?
Active destruction requires physical access. If you provide physical access through a stupid vulnerability (that can be 100% avoided using safe languages e.g.), then yes, you should be held liable. If the user provides the physical access through his own neglect, it's his own fault.
>There's nothing wrong with allowing exceptions for software clearly described (and sold) as "beta".
Then all software will be sold as beta.
>Yes, just like he is accountable if he fits the lolcats viewer with a trojan on purpose.
One of those things is a criminal violation, the other is not--"just like" doesn't work here.
>One who is careful enough to use a safe language, skilled programmers and possibly insurance. I fail to see how this is any different from a small vendor of electronic devices, who needs to comply with safety regulations and bear the risk of a recall due to a design problem.
I've developed and sold small electronic devices before--regulatory compliance is a huge burden if you're only selling a few units. Paying the cost of compliance testing when you're niche products with a small number of sales is very difficult. In addition, it greatly increases the amount of capital needed to get going.
Whenever someone proposes regulatory solutions like this, I assume he has only worked for (or owns) established companies and has recently tried to start one himself.
How many small electronic vendors are there vs. small software vendors? Software that can cause death (like electronic devices can) already has to comply with safety regulations.
>How is this different in other industries where regulations that demand sufficient quality and warranties exist?
It's not, that's the point. The regulations in these industries are heavily skewed towards large companies, that can afford to pay entire regulatory compliance divisions and the divide the cost amongst their comparatively much larger sales--complying with regulations is much easier if you are selling a millions copies of something than if you are selling 100.
>So you're saying that complex products should not come with a suitable warranty? How is an image viewer for the PC (no warranty, possibly vulnerable to specially crafted images due to buffer overflows etc.) more complex than a car (comes with warranty)?.
How many car companies are there? Do you really want to live in a world where the software industry works like the car industry?
Embedded critical systems running real time software are much more reliable than desktop software because failures can kill people. But there is a huge trade off--do you want to run a desktop graphics program that is designed by engineers at Boeing?
I like the freedom of choice that I'm currently offered--I'm willing to accept a few bugs in exchange for more a better user experience.
That is what happens right now, only that they get away without labeling it that way. When beta software (known as such) is used in production environments, at least you can fire whoever made the decision to use it when it causes massive damage. But when a software vendor gets away with selling you a shrink-wrapped product that actually "does nothing" (as the legalese in the EULA claims), there's something very wrong.
> I've developed and sold small electronic devices before--regulatory compliance is a huge burden if you're only selling a few units.
Not a show-stopper then, thanks for proving my point. ;-)
> Whenever someone proposes regulatory solutions like this, I assume he has only worked for (or owns) established companies and has recently tried to start one himself.
Interesting assumption, out of the blue. I started my own (vc-funded) company in 2000 and am still working there as CEO. Now what was the ad-hominem-ish point you were making?
> It's not, that's the point
So why does software deserve special treatment, resulting in the buggy mess we have now?
> How many car companies are there?
Lots.
> Do you really want to live in a world where the software industry works like the car industry?
It would be great if software was as carefully crafted as typical cars. There's also plenty of humorous spins at that comparison out there (you don't have to drive cars with a completely different UI every 2 years) ...
> But there is a huge trade off--do you want to run a desktop graphics program that is designed by engineers at Boeing?
What exactly is so bad about engineers at Boeing that you expect me to answer with "no"? Is Microsoft a somehow much smaller, cooler and more flexible company than Boeing so I should feel like I'm getting a better deal now?
> I like the freedom of choice that I'm currently offered
Noone wants to take the freedom of choice away from you, just the freedom of vendors to sell junk ("no fitness for a particular purpose") disguised as software.
If you want this kind of ultra reliable software, have it built in-house, or hire vendors and contractors who will agree to your terms.
>Not a show-stopper then, thanks for proving my point. ;-)
You have no idea how many products I've shelved at the prototype stage because I didn't think I could recoup the regulatory costs.
>Interesting assumption, out of the blue. I started my own (vc-funded) company in 2000 and am still working there as CEO. Now what was the ad-hominem-ish point you were making?
Turns out it was correct, you haven't recently tried to start a company--you did so 12 years ago. Now you're an entrenched player with something to gain from additional regulation.
>Lots.
That's just absurd. Compare the number of software makers to auto makers. There's at least 2 orders of magnitude more software companies, probably many more.
>What exactly is so bad about engineers at Boeing that you expect me to answer with "no"? Is Microsoft a somehow much smaller, cooler and more flexible company than Boeing so I should feel like I'm getting a better deal now?
You seem to want desktop software to be as reliable as embedded real time software, that's what I mean by manufactured by Boeing. There is a huge trade off for that reliability, in the form of performance.
>Noone wants to take the freedom of choice away from you, just the freedom of vendors to sell junk ("no fitness for a particular purpose") disguised as software.
That's exactly what you're trying to do. Right now I have a huge choice of available software. You want to restrict what manufactures can produce, thus limiting my choice.
You don't like the current state of software--fine make better software, use BSD--don't tell me what I can and cannot buy and sell.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Recently I have been using SPARK and it's a pretty different development mindset than the embedded C that I had been used to. It makes me wonder if other development areas could benefit from adopting some of this mindset and discipline.
For example, just to take a random example of Java or Python, what would it look like to create a strict subset of language features which allowed for formally provable code and to create a static analysis tool which could achieve it. I realize that this sort of thing isn't for many developers or many sorts of projects. However, I would like to see more folks taking the best parts of High Integrity Software development and bringing it out to the wider world.
Not necessarily. Part of Kwak's problem with software is that it is taking humans out of the system. For example, the stock trading software that runs a modern exchange replaces the human traders who used to exchange physical slips of paper. A human trader can look at a particular trade, say, "Hmmm, that doesn't look right," and ask for confirmation. A computer can't (in the general case). So, if humans are the resilience in any system, what happens when you start taking humans out?
They do this all the time. There are quite a few pre and post-trade checks enforced by the exchanges. The net result is that if things go wildly wrong, trading will most likely stop.
This is part of the reason why Knight Capital's implosion had such a minimal effect on anything besides Knight's shareholders.
That's true, and the point that I was trying to make is that we need more of these built-in checks and assertions, even if they mean that the system isn't running at its maximum potential efficiency. I think that software development practices (especially in the financial industry) have gravitated too far towards speed (in computing and executing actions) and not enough towards safety (e.g. doing some meta-analysis and determining if the computed actions make sense given higher level trends).
It's "bad" because of three factors: (1) complexity, (2) inadequate motivation, (3) lack of understanding.
First, complexity. Bridges aren't doing several million operations per second. They either bear the weight or they don't. They handle a known temperature range. If it can function at -50 F and 120 F, then it can handle the typical 65 F day. With a bridge, people put a lot of work into solving a simple problem extremely well and reliably. Software can be developed this way (Unix philosophy) but most software isn't. Small programs are written to solve problems well, but often invisibly; large-program methodologies exist to get promotions for higher-ups.
There are few things in the world like software, which is pure logic. We don't have the tools to understand the complexity, and what tools we do have are often used to write more complex software, not understand the software we have (see: IDEs and their "four wheel drive problem" of getting people stuck in more inaccessible places.)
Second, a lot of "software engineers" aren't very good and don't have the incentives and leeway to get better. That requires lifelong learning and professionalism in the true sense of the word. We're not a profession. Many things define a profession, but some salient traits of professions are: (1) an ethical ruleset that supersedes managerial authority-- not only are you allowed to refuse your manager on ethical grounds, but you have to do so; "I was just following orders" is not an excuse-- and (2) an expectation and allowance that the professional will dedicate half (~20 hours per week) her working time to continuing education, networking, and other varieties of "off-meter" work that will never occur in a typical manager/subordinate context because they benefit the professional's long-term development rather than the manager's parochial goals, (3) a very high degree of autonomy in work regarding social-bullshit protocols, but with very strong restrictions on the things that matter (i.e. no one sets working hours or vacation limits over a research professor, but if he steals another's work or publishes results he knows to be false, it's career-ending). None of these apply to software engineering. (1) If you disagree with your boss about how things should be done on ethical grounds related to software quality or intellectual honesty, you don't have an appeal process. You just get fired. (2) Most software engineers, if they want to keep learning, have to do it on their own time; hence, the stagnation that sets in for all but the most energetic and ambitious. (3) Nope. Here, the software industry looks more like middle-middle class white-collar culture (show up at certain times, take orders) than upper-middle-class professionalism. The salaries are often professional level, but work conditions are merely white-collar.
In a truly professional environment, manager-as-SPOF is avoided like the plague and people have genuine incentives to do good work, not just enough work to appease the manager. The manager still has some power and influence, but the role is more like a graduate advisor than an overseer. Also, professionals are encouraged to build lifelong reputations and seek visibility. (In a typical software company, trying to do this gets you the smear of a careerist and a "socialite".) Professional environments motivate people to do their best work. Merely white-collar environments don't.
The third issue is that we simply don't have a good understanding of software. This may be derived from the two above: complexity and lack of professionalism. Or it may result from something else entirely.
We know that, theoretically speaking, "it's impossible to reason about code" (cf. Halting Problem). Well, my response invariably is that that's correct: it's impossible to reason about arbitrary code. But we shouldn't be writing arbitrary code. Ever. We should write simple code that doesn't rely on extremely high-level mathematical or conceptual insight (which we model as non-computational "genius", though I have no interest in getting sidetracked into the debate of whether we are or are not computers) to understand.
I think individual people are good at writing software. People don't usually set out to write "arbitrary code". There are many individual software engineers who (1) use only as much complexity as they can handle, and (2) act professionally in spite of the lack of incentives or requirements ensuring it. Which means that small software can be good. What I like about the Unix philosophy and small-program philosophy is that it enables islands of quality to exist, and eventually bridges between those islands, which can lead to generally decent systems even though it's GRAI (generally recognized as impossible) to have high quality in an entire codebase. Systems design is all about accepting the possibility of failure amid complexity. It's about using small components that do one thing really well and interchangeable parts, not single-program mudballs. Engineers get this. Managers, who conflate bigness (cf. interview questions like "what's the largest team you've ever managed?" and metrics like kLoC) with success, generally don't.
The problem is that, once a program has a certain number of hands pass over it, it turns to shit. Even if the original code was good, entropy sets in at some point. One bad apple spoils the barrel, and a "bad apple" doesn't have to be a bad programmer. It could be a skilled programmer who hasn't had a promotion in 4 years and no longer gives a shit. The reason why big-program methodologies generally produce legacy mudballs is that it's just impossible to prevent the "one bad apple" problem from corrupting the whole program.
