Introduction

“Technology advancement has led to various uncertainties, but there is one thing which we can be certain of, which is the ever-increasing amount of data available on the web.”

While raw data itself might prove difficult to provide significant interpretations, data are usually sorted into meaningful statistics before publication. Statements like “9 out of 10 doctors recommend this product” and “Germany experienced 5% GDP growth in 2020 Q3” are statistical summaries. Industries like insurance and quality assurance are essentially built from statistical analysis. With how commonly statistics are used in our daily lives, unethical people can easily manipulate statistics due to a lack of emphasis on critically analyzing data.

Who’s lying?

Businesses want sales, and the media want views, governments want to support, while we as the public want whichever is the best for us. Note the conflict of interest between these parties and the public. Parties like these can easily falsify statistics, indirectly manipulating the public to behave in their favour. Sometimes the publishers themselves fall for confirmation bias, citing only data that supports their own argument or agenda. There are indeed many studies with opposing conclusions. Still, one should almost always lean towards the majority unless there exists an inherent flaw within underlying research methodologies, something which anti-vaccine and flat earth believers failed to do so.

Common techniques of statistical misuse

1. Leading questions

Let’s look at the following survey to obtain the public’s view on the U.S. military intervention.

  • Do you support the U.S. Military providing foreign aid to nations in need?
  • What do you think of the U.S. Military wasting taxes on aggressive intervention?

When these statements are contrasted against each other, we can see that the previous statement is more likely to gain support due to words like aid and in need, whereas the latter will achieve the opposite effect due to words like wasting and aggressive. Words that evoke emotions should be avoided in order to obtain an unbiased view, such as:

  • What is your view on the U.S. military intervention? If so, please provide a reason.”

These are known as suggestive/leading questions. It subtly influences the person answering with an implied answer. As a matter of fact, these questions have such a profound impact on the outcome on legal court cases where such questions are generally not allowed in Malaysia as stated under section 141 of the Evidence Act 1950.

2. Sample Quality

To understand what a sample is, we look at another relevant term, population. Say you want to find out the percentage of people that wear spectacles in Malaysia. To obtain an accurate depiction, one can conduct a survey on all Malaysians, which is not feasible in terms of time and cost. So, a realistic approach would be sending out say, 200 surveys to random Malaysians with your acquaintances’ help. In this example, the population would be all Malaysians, in contrast, the 200 participants are the sample, which is merely a subset of the population. It is also important to note that the sample should be somewhat representative of the population. For example, it is rather unlikely to generalize the average height of all elementary students in a nation by only taking samples from one particular school. Perhaps obtaining data from 10 schools from each state would prove to be appropriate, which brings us to the next issue of how many samples is considered sufficient?

One of the most important rules in statistics is the law of large numbers. The law of large numbers simply states the results approach the expected value as the number of trials or samples increases. Simply put, if you flip a coin five times and obtain four heads, one might conclude that the probability of obtaining heads is 80%. While repeating the trials for a sufficient amount, we can conclude that the probability of obtaining heads and tails is 50%. The Law of large numbers explains why investment returns are consistent over longer time frames as well as why insurance and casino related businesses are guaranteed a profit in the long run. Unfortunately, there is no strict cutoff in determining the ideal sample size. Generally, researchers tend to estimate the appropriate sample size by selecting the research setting’s confidence interval and variance.

More than often, a biased sample can heavily skew the conclusions. Back in the older ages, telephone surveys were a popular methodology to access public opinion. However, the sample is already biased in terms of income and age, where the majority of telephone owners belong in higher income groups, and are generally older. Other forms of biased sampling comes in the form of voluntary response bias, where the samples tend to be self-selected. A good example is a public opinion on controversial opinions, where individuals with strong opinions tend to get overrepresented since only this group would be bothered to participate in similar surveys/questionnaires.

Relatable examples of inadequate sampling are the formation of stereotypes, where conclusions are made regarding a category of people typically devised from a small sample. From a Malaysian perspective, things like the certain Chinese being stingy and the Malays being lazy are commonly heard, while it is rather unlikely that these views hold for the majority. These views are likely formed under an extremity bias. The Chinese that are neither overly stingy nor extravagant are less likely to turn into discussion topics because exaggerations make for an interesting conversation, hence not turning into stereotypes. Considering the fact where most people communicate with people who have similar demographics and similar acquaintances, it is even likely that these stereotypes are over-represented due to double counting, where one outlier is simply being reused as a topic.

3. Visualization

Data can be visualized in diagrams and graphs, allowing viewers to get a better overview while also leaving room for falsification.

Diagram 1: Gun deaths in Florida as of 2014 (Source: Reuters)

If any graphs deserve a spotlight for being misleading, it would be the graph above by the Florida Department of Law Enforcement. Clearly, an inverted y-axis and area “above” the curve only indicate that the original graph was inverted upside down. 

The diagram above demonstrates the difference when manipulating the axes. From the heights of the bar charts on the left, one might be quick to assume that group B is significantly higher than the other 2 groups when there is only a minor difference between the groups as portrayed on the right. While visual misdirections are relatively easy to spot, this is not precisely true, thanks to the auditory distractions and information bombarding, which are fairly prevalent when accessing information online.

4. Comparison and Context

In the light of recent events, people have been frantically comparing the Coronavirus with other diseases of different infection rate, mortality rate, etc. Most of their efforts are in vain without considering factors like testing frequency, demographics, population, etc. As an example, China has revised their reporting standards on Coronavirus more than once, as demonstrated here and here, yet we still see sensationalized headlines revolving around the number of daily Coronavirus cases. On the other hand, from a financial standpoint, investors would analyze prospective investments via metrics like valuation and profitability. Strong believers of value investing emphasize a low PE ratio and PEG ratio more than anything else. It is crucial to note that businesses across different industries have different profit margins and valuation metrics. Comparing the current values to historical data might make more sense assuming no changes in business model and economic trends.

5. Correlation doesn’t imply causation

“When A increases, B decreases.” A familiar statement to justify all sorts of cause and effect relationships. Unfortunately, just because two variables are correlated doesn’t mean there exist a cause and effect relationship. Generally, researchers would compare their findings in the form of experimental groups against control groups to confirm any underlying causation.

Ruling parties and politicians like to point to outperformance in stock markets as a form of superior administration, although market performance is affected by variables that are usually beyond the control of any affiliation – e.g., Obama’s election date being extremely close to the start of S&P500’s decade of the bull run. 

There are also scenarios whereby two variables are caused by a common factor, e.g., the inverse relationship between imports vs suicide rates. Do imports have a direct effect on suicide rates or vice versa? Perhaps. But a more likely explanation is economic performance. Higher imports and lower suicide rates tend to happen in tandem when the economy is good, making sense which people buying more imported goods and are less likely to experience financial difficulties during a strong economy. 

Confusion between the cause and effect is also misleading, e.g., The more firemen that are deployed to rescue a fire, the more damage is done. However, the inverse is in fact true. Critical fire conditions prompt higher deployment of firefighters. The damage being done is caused by the critical point of fire regardless of the number of firemen sent.

Conclusion

Over time, it has been increasing in difficulty to control our information consumption due to the boom of data, which has opened up room for manipulation. Certain things lie beyond our control as others are being incentivized to lie to achieve ulterior motives. While ethical development is not instantaneous, general awareness and maintaining scepticism towards the aforementioned techniques can be nurtured over time. I hope to see an increase in initiatives to increase statistical awareness conducted by the government and NGOs via education. In the end, it is up to the general public to up their game in their statistical savviness to detect trickery.


Researcher: Cheong Jian Yan

Reviewer: Millen Lau

Editor: Arivaasaran Arjunan

Download the article here: The-Misuse-of-Statistics.pdf