So sad. I find the mechanics here really challenging to overcome. The hotel management no doubt wants "really cool tech" for their hotel to show they are up to date etc. And they send out an RFQ which someone bids on, really cheaply. Knowing that by only doing the things the hotel asks for, they can throw something together quickly and cheaply for a big payday.
This is exactly the mechanism that gets people in trouble going to China for manufacturing. They say "I want you to build widgets" and they get a good price quote, and say "Wow, this is awesome!" because they have in their mind that "making things in China is cheap" but in reality its that if you cut a lot of corners you can make things really cheap, and since the contract doesn't say you can't cut corners, it is all "perfectly" legal. But the manufacturer knows what the buyer doesn't, and exploits that information asymmetry to make money at the buyer's expense without the buyer having any true recourse.
The hotel in question could have said in the RFQ, "System will be impervious to network traffic snooping and at no time will systems or a guest supplied computer be able to access the controls in another room."
Had they said that, the price quotes would have gone up and had the system the author speaks of been delivered, the Hotel could recover the costs of installing it from the vendor. But they hotel didn't even know they needed to ask for that since they no doubt would assume, "nobody would make something that shoddy would they?"
I learned about this when I saw one of the rules in a NetApp hardware contract that said "Manufacturer will install all components shown on the schematic on the final units in their designated locations." That seemed really odd. I learned that before that clause had been part of the standard contract, there had been a manufacturer who decided unilaterally that half of the noise suppression capacitors in the schematic were "unneeded." Units from that manufacturer started failing in odd ways in the lab.
> This is exactly the mechanism that gets people in trouble going to China for manufacturing.
I keep hearing about the "cheap Chinese tech", even though nowadays a lot of high-quality gadgets are really Chinese. Even Apple's products are mostly from China. And it's not even for the cheap prices, it's because the entire production chain is there[1]
The mistake is not going to China, it's going to China just in order to save money - or anywhere, for that matter. I get that your point is not actually about China, but saving money on the wrong things.
I'd just appreciate if we could stop using "Chinese" as a synonym for "cheap".
This is a great point, and there's a close parallel with the way that people make similar comments about Indian developers and completely ignore the many highly-qualified developers who just won't work at lowest-payer wages.
I was thinking the point would be that one reason for the low price is that the manufacturer wants your design so they can use it to build a competing product at their other factory down the road--a joint venture with the local Party boss. You'll never know, but even if you figure it out, what do you imagine you can do about it? Maybe get a court to block the product in the US (causing them a week's delay while they change labels and distributors), but if you're looking for cheap Chinese manufacturing, you won't usually have the resources for any worldwide legal battle against these guys.
And if the design you are paying them to take from you is for some IoT product, the local Party boss can even make bonus points with his superiors by offering them a chance to backdoor it.
I'll bet the cheap tablets in that hotel for tech conference attendees, the tablets with the ethernet and WiFi listening circuitry, could collect a lot of great technology for their makers, and I'll bet they were made in China.
I read the OP as making exactly your point. They were saying that the people they describe make the error you discuss, and get bad results on account of it.
The practice of unethical corner cutting seems somewhat rampant in China, though. And isn't there a reason why cheap tools that easily break are said to be made of "chineesium"?
The full history of commerce. :p I'd suggest looking at the history of food and drug regulation and testing, or weights and measures regulation if you want more. The early years of both UK and US food and pharmaceutical regulation is terrifying!
TL;DR Immature locations cut more corners. More regulation, experience and reputation helps. Consumers are naive. Manufacturers want you to buy the same things many times over your life.
In all markets, There's always going to be someone willing to cut corners to secure a place in a market, or make a fast buck in a mature market. Or maybe it's a new market that can't yet have matured. That's only part of the problem.
As markets mature regulation increases and companies tend to trade more on reputation (not always deserved of course. eg Beats headphones). Now there's an opportunity to trade on the good reputation of wherever (Proudly Made in America! / Britain! / Japan!), whilst selling you cheap crap. That's the other half of the problem. The meaningless label to tell the consumer it's made wherever they currrently believe is good.
In the early 80s most things Japanese were crap. Hifi sounded awful, but had lots of LEDs. Bolts, tools and vehicles were made of soft cheese. Their stainless steel rusted (I kid you not)! Honda made cam chains of special stretchy metal and probably accidentally invented cheese strings. Now Denon make very nice hifi, Teng make very nice tools and their cars are pretty reliable.
Why pay £50 for a Snap On or Britool[1] spanner when there's an almost identically packaged one, made in the same place, for £3 or £40? Only one will last longer than you in daily use. One risks breaking on first use.
So, it would be more accurate to say "isn't there a reason cheap tools break", "cheap materials break", "consumers naively expect $100 quality for $3.99" or "dishonest people are dishonest" than blame a specific locality.
The ONLY thing that has changed is all manufacturers adding built in obsolescence whenever possible. Now even the premium item is made to last "just long enough" (to get away with), but that doesn't make any one location especially good or bad at making stuff.
The only thing geography introduces is the further away it's made, the harder it is to audit your supply chain. Racism and nationalism has no relevance however.
[1] They're no longer British, or often made in Sheffield, they're just another meaningless brand of Stanley trading on 100 years of reputation. You're actually better served buying Teng these days.
I think you misread cheap. The "mistake" is assuming the low cost bids will deliver the same high quality you see in other Chinese exports,so not doing proper research / QA.
Exactly — and so you can make the point better simply by saying “cutting corners” directly rather than confusing the issue by using a nationality to imply it.
Fair enough, however with this audience I would expect they recognize that the manufacturing contracts of China, which by their number rather than their nationality, are expressly tailored this way. It is by virtue of the Chinese success as capturing the manufacturing contracts from all other nations in the world that has helped them develop expertise and skill.
This has been an interesting conversation. I found it particularly interesting that my communication came across as disrespectful to the Chinese.
Few people that I've met have any real world experience with contract manufacturing. Of the ones I do, they have mostly dealt with Chinese manufacturers, although I do know one person who worked with a Japanese contract manufacturer and one with a Vietnamese factory. Everyone who has ever asked me about this I point to Bunnie's "Made in China" blog entries [1]. Which convey the challenges and rewards of taking manufacturing to China much more clearly than I ever could.
That said, people who have had experiences with contract manufacturing in China have all had a very similar experience, that experience was that the contract manufacturers have an exquisite expertise in squeezing costs out of manufacturing through creative techniques, not specifically disallowed by the contract. Bunnie writes about this at length in his blog.
The thing here is the law of large numbers. There are so many contract manufacturers, and their business is so competitive, the ones who develop this expertise survive and the ones who don't, they don't survive because nobody accepts their bids. It is important to understand that they are this way because they are good at what they do, not for any negative reason.
It is this exact asymmetry of information which I expect befell the hotel in its attempt to have "cool programmed light switches and TVs." This mechanism, which many people who have used contract manufacturers have experienced, is that an inadequate specifications on the final product can give the manufacturer room to economize on their costs, which increases their profit, and also increases the chance that the bidder will be around for the next bid.
And it is the large number of Chinese contract manufacturers, the ease with which they can be located and contacted via Alibaba or other web sites, that means so many people have had a chance to experience this effect first hand with them. Using Chinese manufacturers as an example of the challenge in my post was my way to communicate what I was talking about in a way that folks who might look this up could find additional resources discussing this challenge (and they would probably find Bunnie's blog too).
The leap here, was to take what I wrote and assume that I said, or believed that because something was made in China, is was cheap.
That was not what I said, and certainly not what I meant. But a mix of people have both read it both ways. So it certainly could have been written more clearly.
I really do recommend Bunnie's blog. Everyone should understand the challenges of working with contract manufacturers, regardless of their nationality. Not tightly specifying a contract (and worse not knowing how to tightly specify a contract) will create situations like the one with the Android controlled light switches.
> I learned that before that clause had been part of the standard contract, there had been a manufacturer who decided unilaterally that half of the noise suppression capacitors in the schematic were "unneeded."
I don't understand how putting that in the contract is supposed to help if the manufacturer being used is pulling stupid "you didn't say the product actually had to work" semantics games that would get them smacked in any reasonable court anyway.
Welcome to the world of law. Often I've seen it said that thinking like a software developer, looking for edge cases and such, will get you smacked down by a judge who doesn't let you just use loopholes. There is an XKCD about insurance law on this topic. But the reality is that a lot of loopholes do work, and have a better chance of working if you have a really good lawyer. Part of the equation is how charismatic a lawyer is and if they can pull up records of the loop hole happening in the past (assuming you are in a court that allows past rulings to have impact).
From what I've seen it is extremely arbitrary and is extremely frustrating. I was on a jury once, where one witness was told to tell only what they had heard/seen/etc. They would try to say "I heard so and so say such and such", and the other side would object. The judge would then say to tell only what you saw, heard, etc., not what other people did or said. And I'm sitting there thinking "But what that is exactly what they were trying to do!" (of course I couldn't say a single word for risk to my own safety).
games that would get them smacked in any reasonable court
You're talking about a Chinese court, right? The guy you're planning to sue lives in China and is a long-time business "associate" of the judge who will decide your case.
This is exactly right. When you disagree with a supplier you can only disagree based on the contract, and if someone could reasonably (and there is a wide latitude here for "reasonably") argue their interpretation was within the constraints of the contract than your remediation options are limited.
If you read contracts a lot (and over the years I've probably read a couple of thousand and negotiated maybe 100 or so) you will begin to see clauses that are in the contract which specifically prevent what was clearly a problem before that had not been decided as being in breach, so the added clause insures that in future contracts it would be decided as being in breach.
My lawyer once told me that every contract tells a story if you know how to read it. The more I've read, the more I have come to appreciate that sentiment.
Why would you take an untested schematic to a manufacturer and ask them to build a product based off of it? If you are asking the manufacturer to both devise the schematic and build the resulting product, then it behooves you to ask for a prototype built from their schematic in order to evaluate the production's performance.
Re "nobody would do that": here's a quote from Destiny's Shield, one of the books in the Belisarius series:
"I was just thinking of the provisions of a typical Alexandrian rental agreement. For a house or an apartment. You know, the one about—"
Zeno smiled, nodding. "Yes, I know." His voice took on a sing-song cadence: " 'At the end of the term, the tenant shall return the house to the lessor free of dung.' "
He laughed himself, now. "It was so embarrassing for me, the first time I rented an apartment in Constantinople. I was puzzled by the absence of that provision in the contract. When I inquired, the landlord looked at me as if I were crazy. Or a barbarian."
This is a nice hacking story. But when you have physical access and expertise you can hack anything. So I don't understand what's so sad about it. I do advocate security in depth, and they should probably have added a few more "layers" of security, like hide the cables and encrypt the network traffic. But then he could just use a screwdriver or pull the encryption key from the device, etc. But they probably judged that stopping kids from playing with it would be enough. The guy is a freaking firmware developer and security expert!
Say you got inside a datacenter, or nuclear power plant, and pulled a cable from a control unit, you would probably be able to control stuff too, and probably more sensitive stuff then the room lights.
As soon as you get access to stuff you are not meant to access, it gets exponentially hard to protect from privilege escalation.
As a security exercise, assume a malicious hacker have physical access to your LAN. (shares, KVM, IPMI, MITM)
There are a few realms of business that seemingly necessitate such games. It's about as aggravating as can be imagined. I wish I knew a way around it. Different people? Closer aligned incentives? The cross-cultural aspects here make it especially difficult. You don't know what you don't know.
This is what makes people with so many years of experience so valuable. NetApp had a woman who would source parts from around the world and she had done it for long enough that she knew many (if not all) of the tricks in the book. Often her conversations would start with a new supplier and her requests with explicit constraints would tell them that she knew what she was talking about and that they had better play it straight. So they started out assuming none of the tricks they might use with an inexperienced buyer would work. That experience had tremendous value to the company.
Closer aligned incentives, yes. If you give someone money as gradually as possible they will need to make sure they can't pull a fast one and run off, they will actually need to perform.
Genuine question: Don't the other companies talk about security in their replies to the RFQ? Wouldn't that cause the original company to stop and ask the other repliers about the security they would implement (if they didn't mention security)?
Well that's just it, security is talked about but the buyer (the hotel in this case) is often not in a position to actually evaluate the vendor's claims.
You can put down "device should not be hackable" but without their own competent IT arm the hotel can't possibly verify the product delivers on the security promise.
> Well that's just it, security is talked about but the buyer (the hotel in this case) is often not in a position to actually evaluate the vendor's claims.
But at least the buyer becomes aware that security might be an issues, and thus take it into account when making the final decision. (Even if its just "take the lowest bidder that talks about security convincingly"). OTOH, this doesn't work for buyers that don't actually care.
> You can put down "device should not be hackable" but without their own competent IT arm the hotel can't possibly verify the product delivers on the security promise.
Sure, but if the vendor puts this in the contract and the hotel does get hacked, isn't the vendor then suddenly liable?
I don't understand why you use China as an example. The example you raised happens everywhere when your contract is exploitable. Pointing you finger to China does not help your case, only shows your prejudice
We still use locks on doors, even though they provide security theatre only. Not everything needs to be that secure. Criminal nuisance laws are probably enough to deter anyone actually turning on all lights at 3am.
> We still use locks on doors, even though they provide security theatre only
This isn't really security theater — the term refers to something which gives the illusion of security and doesn't deliver, not the failure to achieve absolute security. In general, door locks are about as secure as they're portrayed: they don't prevent someone from breaking in but they considerably increase the time, skill/tool requirements, and risk of detection. The other key part is that the threat model is obvious: people understand that if e.g. you put a Grade 1 lock on the door but leave the window open, it's not a failure of the lock.
> Criminal nuisance laws are probably enough to deter anyone actually turning on all lights at 3am.
This is exactly the mechanism that gets people in trouble going to China for manufacturing. They say "I want you to build widgets" and they get a good price quote, and say "Wow, this is awesome!" because they have in their mind that "making things in China is cheap" but in reality its that if you cut a lot of corners you can make things really cheap, and since the contract doesn't say you can't cut corners, it is all "perfectly" legal. But the manufacturer knows what the buyer doesn't, and exploits that information asymmetry to make money at the buyer's expense without the buyer having any true recourse.
The hotel in question could have said in the RFQ, "System will be impervious to network traffic snooping and at no time will systems or a guest supplied computer be able to access the controls in another room."
Had they said that, the price quotes would have gone up and had the system the author speaks of been delivered, the Hotel could recover the costs of installing it from the vendor. But they hotel didn't even know they needed to ask for that since they no doubt would assume, "nobody would make something that shoddy would they?"
I learned about this when I saw one of the rules in a NetApp hardware contract that said "Manufacturer will install all components shown on the schematic on the final units in their designated locations." That seemed really odd. I learned that before that clause had been part of the standard contract, there had been a manufacturer who decided unilaterally that half of the noise suppression capacitors in the schematic were "unneeded." Units from that manufacturer started failing in odd ways in the lab.