A couple of days ago I wrote a post regarding some excellent analysis of SearchMetrics data. Since then I’ve read and heard other reports on the data, some drawing some pretty firm conclusions.
This got me thinking about how safe we are being so sure that we can make these connections.
Deadly Digital Marketing Facts… or Just Misinterpretation
To my mind, while the findings are broadly valid, they need to be tempered with the idea that the data set that SearchMetrics use is selected by hand, the key phrases monitored are not random, and there is a large skew towards pre-established products, markets and companies as a result.
My question was.
Social Media Correlation Does Not Equal Causation
I have some explaining to do I can see. My point is that established brands, selling house hold products, do, for the most part, have large advertising budgets, big marketing departments and in just about every case I can think of, a large, well known website that has been around in most cases for longer than Google. (which Searchmetrics’s is measured against).
It might be reasonable to assume that these well known, big brand, venerable sites with huge marketing behind them already had enormous social media presence before anyone realised that it was so important as a ranking factor in 2014.
In other words, if you measure a handpicked set of sites that are bound to have a pre-existing high social media presence, is it surprising to find a strong correlation between that presence and those sites being big and successful in Google?
Is it right to extrapolate that as a rule you can apply to smaller niche sites in less commercial or lucrative areas of the internet?
Here’s a very quick way of putting this logical fallacy;
- All elephants are grey
- My car is grey
- Therefore, my car is an elephant
Like the Chewbacca defence, it makes no sense does it? But this is what we are assuming.
- These handpicked sites from selected high-profile niches rank very highly
- They all use social media
- Therefore, social media is a very positive ranking factor
For all that, I do agree social media is important, both as a ranking factor and for customer engagement. I’m just not convinced (at all) that is is as important as the SearchMetrics data is leading some people to believe.
Some Well-Known Examples Of Logical Errors In The Real World
You’d have thought that we’d now be able to confidently state what’s causing what. But the reality of the situation is that the question of cause, which has continually plagued philosophy and science throughout history, is always dragging us backwards for a variety of reasons. We’ve evolved to spot patterns and we’re essentially programmed to harvest information that’s supportive of our pre-existing concepts, which is called “confirmation bias”. Time has proven that it’s difficult for humans not to confuse causality and correlation.
When you take these considerations in mind, it’s essential for scientists to meticulously control and design their experiments to ensure that there aren’t any hidden variables, and that there isn’t any bias or circular reasoning at work. They’ve always got to have full knowledge of the limitations imposed by the methods that they’re using and their requirements. Results can’t be overstated and representative samples should always be drawn from whenever it’s possible.
All you have to do is take a look back through history to understand the real importance of this issue. Underestimating the risk of confusing causation and correlation can have serious consequences.
Vaccinations
One of the biggest correlation/causation confusions in recent history has been parental worries about vaccination safety, which were brought up by the idea, made popular by many celebrities, that rubella, mumps and measles vaccinations were directly causing autism spectrum disorders. The whole idea had originally been influenced by the 1998 Andrew Wakefield paper, but the medical community quickly debunked it. Studies carried out after Wakefield’s didn’t show any causal relationship, even after many vaccinations. It’s true to this day that some parents are still concerned about a connection between vaccines and the development of autism.
These misunderstandings most certainly aren’t harmless. Time magazine reported that, in 2011, children’s vaccinations were delayed, split up or skipped by 13% of parents. That figure increased dramatically to between 20% and 50% in some rural areas. Now that we’re over 15 years on from the onset of this panic, outbreaks of measles and whooping cough are being reported by medical centres. And we don’t know if that correspondence is causal, correlative or coincidental.
Hormone Replacement Therapy
Observational studies are at risk of including hidden variables, risk bias and, the least desirable of all risks, having a study group that’s not actually reflective of the population you’re looking at. Ensuring that a representative sample’s being studied is essential in allowing researchers to use their results in real-world applications, rather than just for the people inside the study.
Take the example of HRT, or Hormone Replacement Therapy. Past its uses for treating the symptoms related to menopause, it was once praised for its supposed ability to lower people’s risk of CHD, or Coronary Heart Disease. A hugely praised and publicized observational study carried out in 1991 was to blame for the belief. Randomized controlled studies, such as the Women’s Health Initiative, later showed that there was actually either a statistically insignificant or negative relationship between hormone replacement therapy and coronary heart disease.
Well, if that’s the case, you might be wondering what caused the difference. One influence was that women that use hormone replacement therapy are generally of a higher socioeconomic status than men, meaning that they’ve got higher quality of exercise and diet – an explanation that wasn’t taken into account by the observational study.
Worker Productivity
It’s very difficult to study humans. They aren’t just going to react to the stimulus involved in your study, they’re also reacting to the experiment. The difference with animals is that they don’t really understand what’s going on. Experiments today are usually carefully designed to control for these kinds of factors, but it’s not always been that way.
A chain of experiments carried out between 1924 and 1932 studied worker productivity and how it was effected by changing a factory’s environment, including alterations to workstation positions, tidiness and light levels. Around about the time that they’d thought they were getting some interesting results, they realized that there was a pretty big issue. The productivity increases that’d been observed fell straight back down after the researchers left the factory, suggesting that the workers’ understanding of the experiment in progress, rather than the changes made by the researchers, was to blame for the increase. The location of the experiments was at the Hawthorne Works in Cicero, III. To this day, the effect is still known as the Hawthorne Effect.
A concept that’s related to this is called the John Henry effect, where a control group’s member(s) attempt to “beat” the experimental group by making much more effort. It doesn’t matter if they aren’t in knowledge of the experiment, they’ve just got to see a group getting additional instruction or new tools. It’s natural for people to be keen to demonstrate their capabilities to gain respect.
Gambling
Evolution’s hardwired people to notice and seek out patterns, but we’re seemingly unable to work on those when we’re gambling for long periods of time. We are able to accept that an event, such as flipping a coin, maintains the same odds despite the number of times they’ve been performed. But the issue is that we’re also seeing those events, not at all rationally, as being a streak. That leads to the creation of false mental correlations between events that are actually completely randomized. We see the past as our basis for current patterns and, as a result, continually think that the next flip’s going to be exactly what we thought it would be.
You’ll commonly here this principle being referred to as the Monte Carlo fallacy, or the gambler’s fallacy. Back in 1913, betters were amazed when a roulette wheel in a casino landed on black 26 times consecutively. They we entirely convinced that a red land was “due”, so they continued to throw down chips. It’s around about that time that you’d want to be the person running the casino, they’d have made an absolute fortune.
Originally published on demondemon dot com by Paul Rone-Clarke