Why do search engines suck now? Wait, before we get ahead of ourselves,
do search engines suck now? Are they actually getting worse,
or have they just changed in a way that I personally hate? Just by glancing at the first page of a Google or Bing search,
it's easy to find a long list of potential complaints.
Paid sponsors crowd out the first few search results. There's annoying little widgets everywhere,
and it keeps giving you barely related suggestions for what to search next,
like a lame choose your own adventure book. But why does this search page have so many ads?
Isn't an interesting question. It's because money.
The more interesting question is why it feels like it's gotten harder to find the information you want
despite all those supposedly helpful widgets.
The answer is also because money, but it's worth unpacking why.
It's difficult to get a lot of rigorous data on this subject, but at least a few researchers have tried to answer
whether or not search engines are, in fact, getting worse. One recent German study did a survey
of over 7,000 product review searches on Bing, Google, and DuckDuckGo over the course of a year,
and concluded that you could still find useful information,
but it was being drowned out by a torrent of low quality content, especially SEO spam.
Top ranked pages typically were heavily optimized
and littered with affiliate marketing links, and they also showed clear markers of lower content quality.
There's not a ton of academic papers on this issue, but there's plenty of data on the web
showing how users have changed their behavior to sidestep low quality, highly optimized results.
One possible indicator that search engines suck now is the growing use of Reddit as a de facto search engine.
Sadly, Reddit's own internal search function has long been considered what the experts call absolute trash,
but that hasn't stopped savvy users from just sticking the word Reddit
on otherwise unrelated Google search queries. It's a well-known tact for cutting out SEO-ified garbage
and vapid listicles because it effectively bypasses the weaknesses of both search engines and Reddit itself.
Reddit isn't perfect, just ask any Redditor, but on the modern internet, it does a rare and special thing.
It allows users to direct their question to a bunch of big old nerds who care more about being right
than they care about making money off the interaction. If you search site colon reddit.com search engine bad,
you'll find plenty of posts complaining about the decaying state of modern search engines
going back over a decade. There has, however, been a pretty major uptick
in such posts over the last two years.
Separately, Google Trends data shows that Reddit has been steadily gaining popularity
as a search term since 2010, when news aggregator Dig shot themselves in the foot
with a controversial redesign and started bleeding users.
That growth remained steady until December, 2021,
when users started appending Reddit to their searches at an increased rate,
over 40% of Reddit's growth as a Google search term
since 2010 is from the end of 2021 onward,
a bit over two years. Now, there are a lot of potential confounding factors here,
but this could be, at least in part, a consequence of widespread dissatisfaction
with search engines. An interesting contrast to Reddit's upward trend
as a search term is Wikipedia, which long predated Reddit as the kind of word
you add to the end of a search query in order to cut through the noise.
Wikipedia is a far more popular website,
currently ranked seventh for global traffic to Reddit's 16th,
but it's been on the decline as a search term since 2010,
in part because Google heavily prioritizes Wikipedia already,
both in search results and as part of its knowledge panel widget.
But this might also indicate that the decline in quality for search engine results
isn't hitting every search subject equally.
There's not a ton of money riding on a search query like, when was the war of 1812?
So the top results are mostly authoritative for reliable history sources.
But if the most important goal of search engines is to find useful results,
why haven't they fixed the problem? It's not that search engine companies don't care
that spam is cluttering up the first two pages of results.
They've been in an arms race with spam since the very beginning.
It's just that the spam is now apparently winning.
According to the authors of the German study that we mentioned earlier, search engine companies banning spam sites
and readjusting their parameters had a positive, but ultimately temporary impact.
There was still a general downward trend in terms of text quality and relevance
for all three search engines studied, which could imply that this isn't a problem
with search engines, but a problem with the internet itself.
Appearing on the first page of Google can be life or death for a company.
So there's massive financial incentive to game that system.
The same as how there's a massive financial incentive to game ratings on sites like Amazon
where fake reviews are notoriously rampant. Companies both big and small have realized
that word of mouth personal recommendations from a financially disinterested human being
are far more effective than traditional advertising, which means that the shadier among them
put a lot of effort and resources into infiltrating so-called organic,
user-generated systems of validation, drowning out authentic user reviews.
Not to get too metaphorical, but the only way to be heard in a room
where everybody's already yelling is to scream even louder.
Everyone's incentive is to make more and better garbage.
Large, vertically integrated companies like Google don't really help this hyper-competitive,
low-effort environment when they leverage their platform
to prioritize their own products over competitors.
Google has had long-standing beef with Yelp
since at least 2011 when the FTC investigated allegations
that Google's search algorithm consistently favored Google Places over Yelp.
That allegation was serious enough that Google actually agreed to allow online services
to opt out of data scraping. Yelp further contributed data to a 2015 academic paper
claiming that Google manipulates search results to favor itself.
Small companies perceive often accurately that the platform they essentially have to use
is a potential competitor that can and will replace them
with a store-brand version at any time. Those fancy widgets and rich snippets exist
so that the engine that can take you anywhere you wanna go
is now a place that you never have to leave. So what are you gonna do?
Build a better product? Pay for your position at the top
or find a louder way to scream. That's why even though we said earlier
that it's not necessarily search engine companies fault
that they've gotten worse, it also kinda is.
You know how captures have gotten harder and harder over time? Well, that's in part because they were used
to train machine learning, which then led to the bots becoming more sophisticated,
which then led to the need for stronger and stronger captures and so on.
What we're seeing here is likely something similar, only internet-wide, where search engines are struggling
to distinguish between quality content and spam from AI systems trained
on traditionally trusted sources like Wikipedia.
The increasing cheapness of relatively sophisticated spam tools
has resulted in numerous odd trends. Some funny, like when oodles of products on Amazon
wound up being named, I'm sorry, I cannot fulfill that request.
It goes against open AI use policy. And others, disturbing.
Like the rise of procedurally generated clickbait obituaries,
often for private citizens, many of whom aren't even dead.
You used to have to be at least a D-list celebrity in order to get that kind of treatment.
Search engines have not lost the fight against spam,
at least not yet. But as machine generation continues to progress
and proliferate, your search experience probably isn't going to get any better.
Thanks for watching guys, if you liked this video, maybe you'd like another video we have about how streaming is basically becoming cable.
You can click on it somewhere.