Data Is Not Enough: Simpson’s Paradox

TLDR:
The total aggregate can show the opposite trend of the aggregate of its subdivisions. This statistical Gestalt explains how people can justifiably live in alternate realities. Data alone is not enough

Gestalt theory says that that the whole is different from the sum of its parts. In statistics this is called Simpson’s Paradox.

So when we hear people today say “listen to the data” or “I’m data driven” that doesn’t really mean much because data are susceptible to paradoxes, and can tell contradictory stories depending on how they are clustered.  And bad actors manipulate this fact to use data you accept to prove conclusion you do not, trying to persuade you that day is night.

As an example lets take the covid incidence rates for 3 states, and each state shows an increasing trend. But each state show lower overall rates than the preceding. So each individual state show increasing rates, but the total shows decreasing.

This dynamic also explain how one can win the electoral college yes loose the general.   Or vice-versa.  It’s why gerrymandering works.  You can manipulate the cluttering to achieve the effect you want, regardless of the total aggregate.

IOW, you can use the same data to tell contradicting stories. So data alone are not enough.

Economics Isn’t About Money

aerial view of people walking on raod

I think of economics as a set of actors each experimentally discovering the rules of their unevenly shared contexts where discovery destabilizes those contexts in unpredictable and varying ways, forcing all actors to adjust and repeat endlessly. Not limited to financial economies.

In this ways economics is a general dynamic that applies well beyond its familiar financial expression.  Money, status, power, intellectual advances, even computational automata.

The Venn-Inversion Fallacy

A Venn-Inversion is a fallacy where from a small Venn diagram overlap, you posit a complete capture.

The IDW, a favourite wellspring of head scratchers, personifies many fallacies.  This one in particular arises time and again.  But I haven’t seen anyone yet model it and give it a name.  Let Eric be my inspiration.

The pattern works list this… There are innovative people who recognize things that most cannot, and as a result advance their culture.  Dick Fosbury for example.  And there are barking mad people who still believe that the world is flat and vaccines will turn you into Bill Gates’ garden gnome.

Now, not all innovators are crazy.  And not all crazies are innovators.  But there is some overlap between these two sets.  Innovators so visionary, so far ahead of their culture, that their culture cannot understand them and so label them crazy.

The fallacy, come from thinking that because your culture calls you crazy you must be a true visionary, a prophet of the the truth.  The label alone is proof.  If they call you crazy its only because you are right, they are wrong, and they are scared.

From minor overlap, to complete capture and conspiracy thinking.

The IDW folks take every criticism as persecution.  They cultivate this into full persecution complexes and epic Manichean narratives.  And through Venn-Inversions, criticism alone becomes proof positive that they, only they, hold the truth.

Speak Like a Human

total lunar eclipse
Photo by Mark Tegethoff on Unsplash

(Originally Posted 01/04/2007)

In a world run by Google word choice is powerful. Colloquial “vaccination” turns up mostly anti-vaxx material. Clinical “immunization” tuns up mostly sane scholarship. Doctors don’t speak human so humans often don’t hear them.

There’s a post over at medgadget about how searching google for “vaccination” turns up mostly anti-vaccination “moonbats”. One of the reasons is the medical industry’s preference for the word “immunization”.

So doctors say “immunization” while regular people and moonbats use “vaccination”. The lesson here is in a world run by Google, if you don’t speak like regular people, regular people will never hear your voice–even if you’re a doctor.

I wonder how much of our current anti-vaxx problems has been because they never saw the sane information due to the vicissitudes of Google search strings.

No, We Are Not Living In a Computer Simulation

Sock Monkey plush toy on brown panel
Photo by Denisse Leon on Unsplash

We are not living a simulation. The computation required to simulate body momenta increases with number of bodies (N). The speed of action in our universe does not vary with N. Therefore our universe is no the output of computation. Therefore our universe is not a simulation.

I really would like to put an end to the intellectual masterbation of the simulation hypothesis.

The argument seems to go as follows…

First, we make simulations.  Simulations are built on computation.  Our computing power follows something like Moore’s law.  If we extend this in time indefinitely we have computing power increasing as infinitely as we have time.

Second, self-awareness and consciousness are inevitable consequences increased computational fidelity.  Such consciousnesses will eventually start to create their own internal simulations that will follow the same evolution, repeating the pattern fractaly.

We cannot know how many layers of this process there are.  But we can know that the number must be at least 1, and we no reason to think it is not greater.  This means that there is necessaily a greater chance we are living in a some nested layer of a universe simulator, than there is that this universe is the one and only.

The simulation hypothesis is just betting on the probabilities.

However the probabilities are wrong, because they are built on magic.

We can easily compute (or simulate) the momenta of any two bodies at any arbitrary point in the future.  Add a third and this system becomes chaotic and non-computable.  Add 10^82 (estimated number of atoms in the known universe) and, well, things get even trickier.

Let’s say we do eventually develop ways to compute the non-computable (there is no evidence to suggest this could ever be true, but let’s just give this to the simulators).  Even in this case the time to resolve a 10^82 body problem must still be greater than the time to resolve a 2 body problem, because of the differential computation required.

But the time it takes to resolve the motion of N bodies in the real word does not vary with N.  The time it take to compute N bodies in a simulation regardless of computing power, does necessarily vary with N.

In a simulation computing the motion of 2 bodies takes less time than 2+N bodies regardless of computing power.

In the real world resolving the motion of 2 bodies takes the same time as resolving the motion of 2+N bodies because no computation is taking place.  To only way to accomplish this in a simulation is with magic.  And if your hypothesis depends on magic, its fantasy not a hypothesis.

Old Stars Don’t Lead New Bulls

I’ve never been a fan of “What’s hot, what’s not” lists. However CNN money has a very interesting article about how the Jack Welch business dogma may be working its way over to the “not” list.

Now I will admit that Welch has probably forgotten more about running a business just this week than I’ll ever know. But I have to wonder if we’re nearing a sort of Copernican revolution in business. Has much of what we’ve held to be true, including Welch, only seemed true because of our limited perceptions? The west was quite sure the earth was the center of the universe before Copernicus and Kepler provided the theory with Galileo and Brahe providing the evidence that the sun and not the earth is the true center (of course with even a little more perspective we “know” that the universe’s center is everywhere and its circumference nowhere, making all of these guys right but for varying wrong reasons–but that’s a topic for another blog).

Daniel Scocco at Innovation Zen posted something that got me thinking about what the CNN articles really means, which reminded me of this chart from Barry Ritholtz.

I’ve looked pretty carefully, and despite Welch’s book being widely read, none of the stars from 89-99 continued to shine between 02-06. As Ritholtz says in an earlier post, old stars don’t lead new bulls.

Furthermore as Kaplan and Foster point out in Creative Destruction (and as Scocco points out in his post, and as you can read in this pdf):

In 1987, Forbes republished its original “Forbes 100” list and compared it to its 1987 list of top companies. Of the original group, 61 had ceased to exist. Of the remaining thirty-nine, eighteen had managed to stay in the top one hundred… They survived. But they did not perform. As a group these great companies earned a long-term return for their investors during the 1917-1987 period 20% less than that of the overall market. Only two of them, General Electric and Eastman Kodak, performed better than the averages, and Kodak has since fallen on harder times… Similarly, of 500 companies in the original S&P 500 list in 1957, only 74 remained on the list in 1997 and of these only 12 outperformed the S&P 500.

With this sort of long-term failure rate something must be wrong in our theories of prudent business management as wrong as Ptolemy was about the sun going around the earth, and as wrong as the Catholic Church for adhering to such dogma for over a millennium despite centuries of glaringly obvious contradictory evidence. Scocco suggests what’s wrong: MBA programs are about administration, not innovation. And I’d go a step further by adding that administration and innovation are not just different but antithetical (like puzzle problems and wicked problems).