I built a service around helping podcasters automatically convert their audio podcast into a YouTube channel. I went through tons of review with Google in order to get access to the YouTube API and make sure everything I was doing was in compliance with their terms - literally months of back and forth. I had been testing in my development and staging environments against their API for 6+ months. I launched in production, got a few videos uploaded to YouTube, and they disabled my API key. I spent months emailing them and never got anything more than the same boilerplate copy/pasted answer. I could have pivoted or something, but I just shut it down and moved on. Lesson learned.
I worked for a very early billpay company where you could pay your bills online to vendors, even if the vendor didn't support it. We used API's where we could, but where we couldn't...
We had a whole team dedicated to keeping up the changes vendors would make to their websites that we scraped for info. The team was called, of course, "Scrape and P(r)ay".
If you build your scraper to find data on the page based on the shape of the data itself instead of the structure of the page then it will be resilient to most changes that don't materially change what data is displayed on the page.
So, prefer regex over css selectors, and css selectors over xpath, where possible. And don't select based on nesting or position if possible.
Depends on your development and per-action cost. And on the possible latency. It also changes your whole stack from "send a request" to "emulate each step in a browser while taking screenshots at (hopefully) the right event/delay" - that's a huge difference.
That's because Booking was also committing some type of misrepresentation and taking revenue away from Ryanair through their browser automation. Even then, the infraction was sooo bad that they got a $5k fine.
> In November 2022 the U.S. District Court for the Northern District of California ruled that hiQ had breached LinkedIn's User Agreement and a settlement agreement was reached between the two parties.
Reminder that the earlier ruling was overturned, it is no longer clear whether scraping is legal or not.
My wife hates it when 1099 her for services rendered, especially since she refuses to bill against the PO I set up for her. The kids at least accepted NET15 payment terms. Although the oldest said if I short pay him for lawn care again, he’s going to take away my early pay discount.
How does that work, though? Setting up a company has an initial cost and then recurrent costs (accountant, etc). Are the benefits that high for the average Joe?
It's hard to answer without specifics, even if you're not doing anything neferious there are a lot of benefits to putting an entity between yourself and your customers. It depends on where you live and what your business is of course.
My first Saas was built around Facebook, Twitter and Reddits API.
Facebook broke like monthly and required random updates, sometimes documented, sometimes not. Zero support.
Twitter worker flawlessly until I hit some limits and there was no way to increase them because I didn't hit some other limits. No way they would talk to me.
Reddit just worked until I gave up.
What I learned is to never again do any business based on someone else.
A good lesson, but a more nuanced one is to do sure diligence. There are companies I would never build my business around (Google, Oracle, etc.) for a variety of reasons. There are ones I know I can trust (at least until they change). There are many where I need a contract.
Change is also a constant, and there are plenty of good companies give bad, and a few in reverse. So due diligence is an ongoing process.
That dramatically raises the price of SaaS, so I use it much less than mainstream industry practice, but much more than zero.
> What I learned is to never again do any business based on someone else.
That's a little dramatic, don't you think?
Supply chains are a core of many (most?) businesses out there. They depend on others to get their business done. No supply -> no business.
What you probably meant is not building a business based SOLELY on irreplaceable business relationships. Unfortunately in a lot of cases involving the tech giants and little startups, there are no alternatives.
More or less, however I really try to not build on anyone else ever again.
I can't find a single dependency in my serious money generating chain right now that isn't replaceable with a similar product within a few minutes or hours if necessary.
My most recent project didn't even last a full year as there was no replacement that worked for me for ChatGPT in that moment. It was built to die, a gamble essentially. I wouldn't waste my time for things like this if I didn't had the end calculated in already.
Same experience, slightly different. It made me realise why some parts of an MBA (e.g. strategic place/partnership/spring boarding from a brand) actually have value I never saw before for tech companies. There are essentially king makers in certain fields - either by their apathy at first-movers, or their choice on who they allow to continue.
Sure, its $10m today, but think of all the other potential startups that they're doing this to, and how many millions they're throwing away, and billions over the long haul. Reddit, Twitter, and company all got big because they had open APIs and people were able to use them extensively for really creative things.
I agree, Google has swallowed up the video streaming market unfortunately.
I keep thinking back to how Vine was basically TikTok, and they threw it away.
Take it from someone who watched with interest, Reddit and Twitter did not get big due to open APIs. They got their first big steps before they even had APIs... In the first year of twitters launch it was in the news in the UK at 300k users in press releases about how it was a "new form of communication" etc... 2008 was actually when the original set of API-laden websites began to fade because with the recession, we could not have nice things be free so much. Many API services fell into disrepair in 2009, certainly the peak of APIs for the UK was around 2008. Check out Tom Scotts video on this for a picture of the open API internet many people thought would occur (and which did exist in some places until FAANG began to productise and dominate and use accounts across products which do not incentivize APIs). Yes it is true open network graphs/facebook etc had some API access, but the APIs before were more single purpose and numerous; facebook lets you explore their network and to some extent data, but that's not the same as the useful APIs that became restricted, paid or were cut in the aftermath of the recession.
Those startups are their competitors. You don’t have to pay millions of dollars to acquire a competitor that was never started because of your inconsistent API policy.
Google makes enough money that losing several currently non-existent revenue streams that are theoretically $10 million apiece isn’t hurting them. It’s hurting their users.
Google is all about ads - why would they give a shit about the users?
>Sure, its $10m today, but think of all the other potential startups that they're doing this to, and how many millions they're throwing away, and billions over the long haul.
Google made $300B last year. "billions over the long haul" is a lot of money, an unimaginable amount even, to you and I. But to Google?
First off - how much of revenue usage last year was on fines etc. That is to say profit they could potentially have had.
Second there is lots of revenue you use for "business" because you don't want to pay taxes on the profit and anyway it is giving you something you like to have.
How much of Google's revenue is used to do things for top executives and people with power in the company that is really something they (the executives) should be taxed on but is instead a business expense of google (cars, transport, 'working' vacations, security, super cool chefs preparing meals at the company...) Hard to say really because if you knew the answer it would actually be something they were taxed on. But it's not 0 - sure probably not a billion, but a couple hundred million splashed around wouldn't surprise me.
How much of Google's revenue usage is for wages and other forms of payout to executives etc. (stock) that does not get counted as profit but of course it is amounts those people want to have.
Google made $73.795B profit last year, and expensed slightly over 262B - some portion of which the people who run Google no doubt personally thought of the way we would consider profit in our day to day existence, and another portion of which were fines for things they did in getting the rest of the money.
Obviously the fines, which are the first thing mentioned, is relevant for how much the shareholders got.
Literally the other stuff is relevant for how remunerative the people who actually run the companies, day-to-day, feel those companies are (put in day-to-day because I got the feeling you might give me a lecture about the shareholders actually running the company)
Personally I think you have a decent point since in practice employees of GOOG are shareholders too. Although I don’t know if their interests matter to the corporate entity because I don’t know what percentage is held by insiders.
I don't think that's nearly as universal an assumption as you're claiming. Like--I made $X,000 dollars last year, but I also have a mortgage and like to eat food.
The assumtion things were great wasn’t mine. I live in a HCL area and am not able to save much so in this sense the ‘profit’ of my work is close to zero. That is not great at all…
I mean, they "make" both. "Make" does not imply profit. If you say a company "made" a revenue figure, then you mean it in the revenue sense. It's contextual.
100% contextual. In every business where I've been an executive, or had access to the budget, we talk "made" as gross revenue and EBIDA as a stand in gross profit and specifically call out net profit in internal meetings. For example, "We made 10 million with an EBIDA of 1 million and net of 200,000.00." Using EBIDA to talk to potential investors and as a guiding metric if we didn't have a well established gross profit formula that followed GAAP.
It's a testimate to the health of our free market that the company throwing away millions, and billions over the long haul, is the overwhelmingly dominant market leader.
Attempted to build a similar thing a few years ago when living in Tokyo.
Provide a selection of restaurants within a 1km radius and automagically provide 3 recommendations based on my preferences.
We had multiple API providers available, Tabelog, Gurunavi and Hotpepper, all required a paid developer license. We still needed to use the Google maps API to get the user's current location though.
This was also just when Google Maps suddenly raised their API pricing. After spending a couple of weekends building a working prototype, we stopped as we couldn't justify the cost of paying multiple API providers for basically 3 guys looking to save 10 mins deciding where to get lunch.
Also, this kind of app is a common theme if you frequent meetups in Tokyo. There's always at least 1 person that has built such a thing.
> We still needed to use the Google maps API to get the user's current location though.
I’m sure as somebody who actually built a prototype, you understand the situation better than me, but I’m curious: why would just getting the user’s lat/long require anything you’d pay for? Wouldn’t the OS or browser provide that to you?
Oops. I meant that we were using the maps/places API to get lat/long of the restaurants using the street address from the other API providers, then using that to calculate a 'N mins walk away'.
We were just testing the abilities of the various APIs. Google was the only that provided travelling time estimates.
I was thinking about how to solve this given that one of the primary problems is that of fast, global content distribution. I like the idea of paying people in crypto as part of a ledger transaction to host and serve content, like bittorrent with a crypto payment. Unfortunately I can't also think of a way to prevent such a system from being abused to distribute harmful media such as CP. I guess it's not like this isn't a problem with BitTorrent today though.
Regulation might here - something like minimum mandatory 3 months notice for shutting down the API keys. Considering the average age of our politicians, I doubt they’ll understand what “API” is, much less be willing to take on giant tech corporations even if they did understand the problems
Yeah, what’s popular is actually to crap all over the first amendment by mandating private companies to allow whatever each politician’s special definition of “free speech” is (but of course probably also not allowing whatever the politician’s definition of “dangerous misinformation” is).
As for the actual anticompetitive power of big tech, absolute crickets.
Competitive markets are by far more the exception than the rule. It's just not how capitalism works (because pretty much any competitive market is ripe for “consolidation”, which increases aggregate shareholders value by reducing competitive pressure).
If you want competitive market in a capitalist economy, then you need very active state enforcement.
The markets are absolutely competitive. Google has to pour money into youtube to keep it actively developed and popular. Yet they still lose live streamers to Twitch, video uploads to all kinds of niche specific platforms, and their paid content isn’t coming close to things like Netflix, max, prime, etc.
They are now hemorrhaging search to OpenAI that popped into public existence just recently.
To claim there isn’t competition in these markets is completely ignorant. “A big player eats 80%” isn’t anything like a monopoly/duopoly scenario where there literally isn’t any competition or product advancement for decades.
If you wanna see lack of competition, look at government granted monopolies on utilities. Guaranteed but capped rates means you reduce investment right to $0 and cut costs as much as possible since there is no other way to make money. That “state enforcement” you are calling for is how you end up with PG&E and scenarios like all insurances companies pulling out of the state.
> Google has to pour money into youtube to keep it actively developed and popular. Yet they still lose live streamers to Twitch, video uploads to all kinds of niche specific platforms, and their paid content isn’t coming close to things like Netflix, max, prime, etc.
Phew, here I was stressing out about lack of competition, but you helped me relax.
Google has to invest into Youtube lest it loses out to Twitch (Amazon), Netflix ($35bn per year), HBO Max (Warner Bros, $50bn per year), (Amazon) Prime, Plus (Disney, $90bn per year).
My faith in humanity is restored now that there are alternatives.
That's not competition, that's an oligopoly and they are mostly focus on their own niche for which there isn't more than a minority competitor (there's no competitor to Youtube on for non-live video, Twitch is in a monopolistic competition situation for game streaming, so is netflix on TV series, and Disney and HBO each have a monopoly on their IP).
Claiming that this is a competitive market is a joke.
> To claim there isn’t competition in these markets is completely ignorant. “A big player eats 80%” isn’t anything like a monopoly/duopoly scenario where there literally isn’t any competition or product advancement for decades.
Please tell me how many “product advancement” in Google search or YouTube over the past decade… From a consumer's perspective all that happened was enshitification and despite being owned by a search engine company, YouTube never managed to ship a functional search on their platform.
> If you wanna see lack of competition, look at government granted monopolies on utilities.
Utilities are “natural monopolies” though, and as such they should be state owned. Making a natural monopoly owned for profit is a recipe for rent seeking, and that's why it was promoted …
> That “state enforcement” you are calling for
No, the state enforcement I'm calling for is proper enforcement of antitrust laws, forbidding consolidation through M&A and disbanding companies megacorps. That is to say, what existed in US's golden age.
>Competitive markets are by far more the exception than the rule it's just not how capitalism works
Empirically this absolutely isn't the case; the majority of listed companies have fairly low margins, especially non-tech companies, which can be trivially seen from their financial statements. A low profit margin means a competitive market (because if it wasn't a competitive market the firm could raise its prices to obtain higher margins).
> A low profit margin means a competitive market (because if it wasn't a competitive market the firm could raise its prices to obtain higher margins).
Not necessarily: if prices are elastic then even a monopoly can aim for low profit margin (in percentage) in order to increase profit. What matters is how much total profit are being made, margin only matters when measuring risk.
Also, corporations are social structures, and low competition also encourage complacency in the corporate structure itself, which drives costs up and reduces profit margin.
Do you know if there is any open source tools in this field that a programmer can work with/automate, I have mainly textual and graphic content, a little bit of video, but thinking to branch out to podcasts and music soon, unfortunately the other products supporting text seem pricey and geared to big teams.
on edit: obviously I can write my own, but I am hoping for a project that has already done a bunch of things that I can extend for my needs, as writing my own would be at the point where it would be more cost benefit to purchase.
I believe I am entirely in compliance with YouTube's terms and policies, but I was flagged for some bullshit as well and got the boilerplate responses. So what I decided to do was hide it for logged out users so it looked like that specific feature was entirely removed to the dudes in India that review this stuff... One year later and my fingers are crossed we don't get another audit.
We went through a three-week Facebook API shutdown due to a clear glitch - Meta support couldn't go outside the script, or escalate in any way beyond "a supervisor will email you". Only resolved when our CEO found some VP at Facebook on LinkedIn and got them to escalate it internally.
(The dev community thread is full of people still impacted, so I think they literally just edited our app's flags directly.)
Even tried to invoke GDPR's rights to be exempt from automatic decision making, but their privacy questions email address responds with "nope fuck off" to those.
Basically any hope of solving a problem with a Meta property is to know employees who can escalate your issue. Sometimes your friend at Meta also needs to argue with the bug owner to not close it frivolously.
Within the customer service industry, they'd claim a great "deflection rate". That's a metric that all of these large companies hold around any kind of help/support channel that may involve humans because people are a cost. It's often covered by some kind of satisfaction metric based off post-issue surveys, but fundamentally, if you just go away that's success.
There are often complaints here about what amounts to bean counting affecting other aspects of business. Customer service at larger scale is costly so attracts a section of very analytical leaders. They don't get, or ever prioritize, the human elements. It's only when satisfaction numbers are bad, or another exec outside of customer service takes action, that things improve.
The scale part kills the customer. You can have great support at one size. Once you grow then leadership, structure, and the culture change. These analytical leaders don't want to carry over the culture and structure because it comes at a cost. It really needs force and support from outside of customer support leadership to maintain it.
Had this issue with Ubisoft recently. Installed Trackmania through Steam for free, then had to create a Ubisoft account, then had to buy the Trackmania annual pass for $20. I guess I followed the wrong flow or something, but I ended up with the purchase on my Ubisoft account, but I guess their system also auto-creates a Ubisoft account for Steam accounts? So the purchase on the Ubisoft account ended up on the account not tied to Steam.
Emailed support about it on 13/07/2024.
Got back a reply on 23/07/2024:
> Our Support Team is currently experiencing high case volumes, so we are reaching out with a message to check-in and make sure you still need help!
> If you've found a solution to your problem already, there's nothing you need to do. Just ignore this message and your case will automatically close in 2 days.
Didn't see the reply until the 3rd day.
> As we have not heard from you in the last 2 days your case has been closed automatically.
I think people, even on HN, grossly underestimate how much customer service costs at any major company that works with the public.
Most calls to customer service are easily self-serviceable. My brother used to work for X-Box Live support, and he said over half his calls were for password resets, something people could easily do on their X-Box.
But then there's bad actors. Another significant chunk of his calls were from people complaining about being banned, and he'd see the reports of their spamming racial slurs.
Not to mention how many people try to abuse customer support in attempts to take over someone else's account.
Yeah...customer service is expensive. Even if you automate as much as you can, but offer an appeal process, that appeal process with be absolutely abused by bad actors, so that will end up automated as well.
I am curious if you would be interested to retry the idea? I might have an in with the YouTube team. I feel like it's a shame to let this go. Would you be open to chat? Please reach out r@pehul.com
No. The default API quota is not large enough to upload videos. You would have to contact Google, explain your use case, and jump through several hoops to get the quota increased... which is a huge process.
Yeah. Uploading legit videos is non-trivial. And if you ever upload the same video twice (which you think you might do during testing, right?), it's a violation of their terms and they disable your access.
