Why three-fourth is not always 75%?

One of my pet peeves is the fact that numerical ideas often (too often) get misused in a bid to convey the wrong impression. It is done by those who should know better and very often by those who indeed know better. Recently, Butttowood had a decent article on this. The financial sector is the biggest culprit. Especially, in the willfully misleading category.

When I was in my previous avatar as a flunky in a global investment advisory, the Economist at the firm in charge of global asset allocation released a report key insight was "Semiconductors lead the rest of technology in the recovery cycle in 75% of the recessions" (or some such tripe). Now, this godforsaken sector was one I was in charge of and therefore had to read more on.

Turns out our Global Guru had looked at the last 4 major global recessions over the last 80 years and found that in 3 out of those 4 semis semiconductor stocks bounced sharper and sooner than the rest of tech.

This is tripe. When you are doing empirical research and are looking at 4 cycles, you have no business describing anything in percentage terms. I stopped reading anything else published by the "Guru". But who am I to say anything? He was the top ranked Economist in the investment industry, and I was the guy who wasn't good enough to get fired when I desperately wanted to.

In the Indian context, Outlook published a cover story saying total scam amount in India Rs. 1.75 lakhs crores or some such. They detailed many scams in this one -

The list roughly goes like this
900 crores - Fodder scam
600 crores - Taj Corridor scam
23 crores - Railway placements scam
3 crores - Perhaps accepted in bribe by the first cousin once removed of some central Govt employee
etc etc.

The last scam number listed was

Total black money stashed abroad = 1.72 lakh crores (estimated).

So, this ginormous number that forms more than 95% of the amount put in the cover page is plucked out of the hat. So, why the $#*k should you go into details in the other scams?

There is this beautiful idea of significant digits in the art of measurement. I never really understood this while at school. Any measurement, be it with Vernier Callipers or Screw guage comes with a built-in error factor. (Least Count?). So, whenever you gave any measurement the number of significant digits must be determined keeping this in mind.

So, if the built-in error is 1 cm. We cannot give a measurement that says 223.5 cms (even if take 10 measurements and average this out). We can at best say 223 cms or 224 cms. The 223.5 suggests that we have confidence over that 0.5, which we can technically not have. Simple idea really. It is like saying if your measurement has some built-in error, do not convey more accuracy than there is. So, if you measure some length 18 times with Vernier Callipers and the average comes out to be 32.222cms. You should bite the bullet and say roughly 32cms. Conveying confidence beyond what the numbers tell you is a crime (or at least should be considered one). Finance and sports are the two fields where this gets done the most.

You might have seen something along these lines frequently

1. The best stock returns are seen from Thursday to Monday.
2. Left-arm bowlers have seen the most success in ODIs conducted since the 1990s.
3. Ricky Ponting really struggles against India as he has an average that is 6 runs than his overall average

Why are these absurd?
If you analysed stock returns over three-day window, some three window would have the best returns. This does not mean that that three-day window has something special going for it. This means something else. Something very special. Something that every statistician worth his salt must have the courage to say on 90% of the times he attempts some statistical analysis. This means nothing.

Ditto the other two inferences.

The idea of statistical significance
So, are all statistical inferences absurd? Of course not. This is where the term statistical significance comes into the picture. If the observation is statistically significant, the flag must be raised. And only then must the flag be raised. So, how do we wind our heads around statistical significance. Statisticians have fancy terms for this. But let us see if we can have an intuitive approach around this. Let us have a go at this with an example.

Let us say we want to test whether Ricky Ponting underperformance against a particular team, say, India is statistically significant. Let us further say his average against India is less than his overall average by about 15% (this looks significant).

Now, let us not take his average against his nemesis, India and keep it as a benchmark metric. Now, let us revisit the original sample and extract a sub-sample from this randomly. If the benchmark metric is lower than that observed in the sub-sample, say, 90% of the time, then let us say that the underperformance is statistically significant.

Let us build on this with numbers. Let us say, Ricky Ponting has scores of {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} in 10 innings. Further let us say, he has played 2 matches each against 5 teams. His overall average is 5.5. Now, let us say he has an average against India that is more than 15% less than his overall average. Or, an average of 4.5 or lesser. Is this statistically significant?

If we extract two scores that have to average 4.5 or lesser, we can have {1, 2}, {1, 3} {1, 4}, {1, 5}, {1, 6}, {1, 7}, {1, 8}, {2, 3}, {2, 4}, {2, 5}, {2, 6}{2, 7}, {3, 4}, {3, 5}, {3, 6}, {4, 5} - 16 possibilities. Totally, there are 45 possibilities. So, there is a nearly 40% chance that some random sample of 2 out of this 10 will have an average that is 15% lesser than the original. So, this 15% below average number means nothing. It is not statistically significant.

In reality, a great many quoted inferences derived from numbers are not statistically significant. If you are handed any statistical inference on a platter, you have hajaar grounds to suspect it is false. And we have not even come to the idea of bias. There are many ways in which we can bias a sample, Some biases creep in, while some others are introduced. Let me give a few examples.

The 2015 world cup stats counter states rather gleefully that the Indian batting unit has one of the highest strike rates in power plays. Now, we need to remember that this is largely because of 70% of India's cricket is played on subcontinental wickets, where the par score is 330-ish. England might play 50% of its matches on English wickets, where the par score might be 260-ish. So, unless the powerplay strike rates are at least 30% apart, we have no business making any inferences. These are the biases that the samples naturally carry.

There are some other biases that data-presenters can bring in. The most beautiful and one that most fudge-statisticians introduce with an unbearable holier-than-thou approach is the selection bias. Let me deal with this with an example.

Let us say, there are two stocks Alpha and Beta that, as an analyst I want to suggest are correlated heavily. I will draw the stock charts for Alpha and Beta and compute correlation numbers. But here is where I will be smart. I will choose the end date to be today's date and the start date to be any date from 2010 to 2013. I will find the correlation numbers for all 1000 or so possibilities and pick the date from which the correlation is the highest. If it is a Friday afternoon and I want to be really intellectually dishonest before my weekend, I will 'float' my end date also.

For any two stocks, if you have large enough database, about 40 minutes of time on your hands, and a moral compass pointing towards "bonus" there is a 50% chance of finding one set of dates where the correlation is more than 90%. You can even wear your best "Why are you looking at me like that. This is what the numbers are telling us". If you want to be thorough, you should find some pseudo-intellectual justification for having picked the date range that you did indeed pick. In case you are wondering how I know this scam with this much clarity, you should look for Business Objects Cognos 91% correlation on some research database. (In my defence, I am not proud of this).

A good statistician is one who can look at a lot of data and tell us why they do not mean much; and then pick up one nugget that actually means something. The statistician who cannot say "this means nothing" should be kicked out of his job. Lot of stats that we see online and on Television are the ones that should be binned. 

How to invent a new religion

Let us say we want to start a new religion. Why, you ask? It is a wonderful Friday morning; thats it.

Now, we are going to about it very scientifically. We are going to pick up the best practices that our predecessors followed, adapt them to the 21st century and then improvise from there.

We need a God. We have seen the ones without God, and they seem to struggle to grow beyond "weird cult". God cannot be a person, we have seen how that fails. He/She cannot be overly mythological as we are dealing with a 21st century audience. God has to be something omipresent but benign, something no one will have an issue with. Sparkling in its simplicity, wonderfully powerful with loads of uncontestable goodness.

We need some Godly material, some testament/veda kind of thing that gives messages from God. The usual guff about not killing anyone, not harming anyone will have to be improved upon a little, now that we have grand legal systems. The material guide must be cloyingly good-natured. So, in case some hyper-rational dudes get all wound up that the backstories do not fit, we can then say "We are saying do not kill your brother. You wouldnt want to disagree with that, would you?"

The material needs to have credibility and must come from some kind of powerful body. Again, as we are dealing with the 21st century, so something packaged as Science might be useful. Also, this must be refurbished every now and then. This is where I think the major religions lost out a bit and had to resort to "reform" and stuff. Let us be very Kaizen about this and institute continuous improvement as part of the original package.

We need a messenger for God. No one really reads the original stuff, so this messenger person must be phenomenal. People must go -  "If he says so, it must be true". The person must have credibility, must have made his money and earned his name in another field and must have wide reach. This is the place that gives room for a tweak. We can look at a model where there is more than one messenger. With scandal and temptation everywhere, and a feral media presence, we cannot take chances with just the one messenger. In my mind, we should look from within the ranks of Hollywood icons, wildly successful entrepreneurs. retired politicians.

We need guilt. This one is obvious. Whats the point of inventing an elaborate religion if we cannot control the masses with some kind of check-back mechanism. Whats the fun in that?  And where there is guilt, there must be repentance/penance.

We need a conversion mechanism. Without this, again we run the risk of numbers not being on our side. We should not take on the existing religions head-on. They tend to get all violent and prickly if we do that. We should slowly co-opt people from within the biggies. At first, they would worship both sets of gods; slowly we would acquire more mindspace, and within a century we should leave no trace of the original Gods. This is what happened to the Pagans. We should target one vulnerable "new Pagan" at a time and slowly create momentum around the conversion idea.

We need to have a modern version of "burning people at the stake". There needs to be some degree of fear. If not fear of death, at least fear of ostracism, (or fear of lack of funding).

We need congregations. Massive gatherings of people where we reaffirm our faith, have a good time and return to our original destinations with good cheer and more stuff to preach.

We should not be geographically constrained. We should have devotees everywhere, without any region feeling aggrieved or under-represented. (This is where our distributed-leadership, local icon model could be better). The moment we have devotees across nations, we can transcend nationalism and push agendas better.

This is the core framework. Because we get to start from scratch, we can try some new stunts also. May be we can pick up a flag, or specify attire; something that screams identity.

Now, if only we had thought of all this 20 years earlier. Someone has beaten us to this. There is already a religion that has been creeping into our existence over the past two decades. It started as a mere cult, but has now reached preposterous levels. Within years, it will be bigger than the big religions. From there on, it will be a small step to banish the existing religions.

God - Nature. If you want to get all polytheistic about it, add Environment, Earth, Water, Resource etc to it. In an interesting irony, these guys have gone all reverse-pagan on the existing religions.

Material: Stuff published by IPCC. No one really reads this. It is periodically updated, is suitably internationally-themed, has Scientific titles adorning its every orifice. Almost all its original fantastical claims have been re-jigged. Now, anyone who contests this is termed a "denier". Any time they come out with a report, everyone from the Economist to the Times of India must carry an update. Ticks all boxes really.

Messenger(s): They started with Al Gore, then they realized that retired or otherwise, politicians have only so much credibility, so went on a recruitment drive. Now, the largest icon is a Celebrity Economist, Paul Krugman. These days, Krugman does not start any article without berating the "deiners" for questioning climate change. If you have noticed, many of the articles will start with "if there was any doubt, that has been removed now". If doubt has been removed now, how credible were you when you said there was no doubt, way back in 2010? is a question you dare not ask.

Guilt: Man is the root cause of all environmental damage, repeat after me for I am IPCC (Imagine this as a rap). The beauty of the material is the fact that even if he/she does not understand any of the climate-model guff (No one does, by the way), some two bit nincompoop can still tell you "I do not understand why you keep talking about scientific evidence. I cannot imagine you want to destroy all forests and do not want to care about the environment."

For real-world purposes, you can now buy carbon offsets. People eat junk food all the time, lay their body to waste with all kinds of things, but are asked to feel bad if they travel in a car.

Penance: Voluntary carbon offsets. People track their carbon footprint, and then offset their "excess" carbon. Need I say more. I am not making this up

Burning at the stake: Scientists who contest anything said about his can lose funding. Republican senators will be called stupid, you or I would be called a "denier".This letter is brilliant. This piece of research is illuminating, wonderfully challenging some taboos. But I guess the people who wrote these are now without tenure.

Pan-national symbol: Green. This is brilliant. No one has adopted a color this favourably before. the other "red" buggers seem benign in comparison. Even climate-agnostics such as 2IIM offer a Green account for the products they sell.

Congregations: Davos, here I come. Of course, if I do make the trip, I must remember to voluntarily offset my carbon footprint.

We still do not know where the green brigade stand on iconoclasm, whether they will fight crusades, or create backtories to fill out the mythology. But make no mistake, this is a religion alright. Across national boundaries, never have a group of people been so singularly driven by a single ideology.

I must state that this article has nothing to do with my stand on climate change. I might be a denier or an aspiring high priest. So, kindly do not brand me as one or the other based on just this article.

Hired Guns can be fired too!

Guns hired for a specific task can and will be fired once task is fulfilled. If a 20-year old hired gun does not know this, it is a pity. If a 60-year old hired gun does not know this, the joke is on the hired gun.

My sympathies are with party-workers, but there is a part of me that goes "You had this coming."