Benford’s Law and the Hidden Geometry of All Numbers

When I came across Benford’s Law, also known as the Signifigant Digit Phenomenon, I was pissed off.  Because I recently turned 26, and that means I’ve spent a quarter century of my life completely unaware of what would appear to be one of the most important mathematical laws in existence.  Worse still, this particular mathematical law was discovered in 1881, and formally proven in 1935, so I am several lifetimes behind the curve.

Surprise is good—existiential shock is even better.  As we’ll see in the course of this article, being whumped upside the head by the utterly absurd and totally unexpected is the best possible way for a human being to learn—there are whole disciplines of mathematics that define Information as “the difference that makes a difference.”

And man...this is a difference that messed my head up good.

Dr. Theodore P. Hill asks his mathematics students at the Georgia Institute of Technology to go home and either flip a coin 200 times and record the results, or merely pretend to flip a coin and fake 200 results. The following day he runs his eye over the homework data, and to the students’ amazement, he easily fingers nearly all those who faked their tosses.

“The truth is,” he said in an interview, “most people don’t know the real odds of such an exercise, so they can’t fake data convincingly.”

Dr. Hill is one of a growing number of statisticians, accountants and mathematicians who are convinced that an astonishing mathematical theorem known as Benford’s Law is a powerful and relatively simple tool for pointing suspicion at frauds, embezzlers, tax evaders, sloppy accountants and even computer bugs.

The income tax agencies of several nations and several states, including California, are using detection software based on Benford’s Law, as are a score of large companies and accounting businesses.

So what’s going on? Well, it turns out that random is not random after all.  No matter where you assemble your numbers from—and dig the examples in the chart above, birth rates, census data, nearly anything—you will find that the distribution of digits will obey the curve depicted above.  The digit 1 will occur more than 30% of the time, and there’s an even curve down to the digit 9.

Is all this explainable by a simple tautology? It’s also interesting to note that Benford’s Law is true for nearly anything—observe how Lottery numbers do not conform to the curve. As we will see later, this is red flag evidence that the Lottery, much like presidential elections or Las Vegas card games, is in fact rigged.

News Is Not Information But Information Is News

In a recent Skilluminati Research ramble about Rethinking Information and the Attention Economy, I quoted a passage from Mark Taylors The Moment of Complexity: Emerging Network Culture which bears repeating (and expanding) here:

Information, we have discovered, is inversely related to probability...the more improbable a phenomenon or event, the less it is anticipated, and thus the more information it communicates when it occurs. 

Another source of confusion is that we generally do not think of information as a liability.  We pay to have newspapers delivered, not taken away. Intuitively, the record of past actions appears to be valuable (or at worst, useless) commodity.  Perhaps the increasing awareness of environmental pollution and the information explosion brought on by computers have made the idea that information can have a negative value seem more natural now than it would have seemed earlier in the century.

This is why so much of the progress in psychology this past century has come from the study of abnormal cases, and this is also why genetics and evolution theory are both deeply indebted to an obscure branch of science known as Teratology—the study of monsters.  Monsters in this case meaning the unworkable and spectacular mutations and birth defects that fill up Philladelphia’s uniquely memorable Mutter Museum.  When I was a little kid, I enjoyed reading my mom’s Oliver Sacks books, such as The Man Who Mistook His Wife for a Hat, because Sacks was an eminent neurologist who made startling discoveries and advances by studying the one-in-100-million rare disorders of the brain. 

Explanations vs. Solutions

As I might have observed once or twice before, humans are exceedingly talented at solving problems.  Explaining how our solutions work, though....yeah, not so much. Although we have abundantly proven that it exists and hold true for a dizzying array of phenomena at all levels of scale, nobody has provided an adquate explanation for why it happens.

Though empirically firmly established by Benford’s experiments (and by later findings), there has been no convincing physical or mathematical explanation for the Significant Digit Phenomenon. Several explanations have been proposed, including arguments based on scale- and base-invariance, extensions of the notion of natural density, and picking a probability distribution at random. However, none of these arguments have been fully convincing.

--courtesy of this rather sparse website

I cannot resist passing the microphone to the ghost of R. Buckminster Fuller for a second:

While it takes but meager search to discover that many well-known concepts are false, it takes considerable search and even more careful examination of one’s own personal experiences and inadvertently spontaneous reflexing to discover that there are many popularly and even professionally unknown, yet nonetheless fundamental, concepts to hold true in all cases and that already have been discovered by other as yet obscure individuals. That is to say that many scientific generalizations have been discovered but have not come to the attention of what we call the educated world at large, thereafter to be incorporated tardily within the formal education processes, and even more tardily, in the ongoing political-economic affairs of everyday life. Knowledge of the existence and comprehensive significance of these as yet popularly unrecognized natural laws often is requisite to the solution of many of the as yet unsolved problems now confronting society. Lack of knowledge of the solution’s existence often leaves humanity confounded when it need not be.

Why is this Dope, Hip, Fresh and/or Relevant?

I fully realize it’s a sign of moral weakness, or at least considerable marijuana abuse, to quote the classic Darren Aronofsky movie Pi, but I will do so now:

1. Mathematics is the language of nature.
2. Everything around us can be represented and understood through numbers.
3. If you graph these numbers, patterns emerge.
Therefore: There are patterns everywhere in nature.

After all, the best part about about restarting Brainsturbator is knowing exactly who I’m writing for—I have no need whatsoever to convince you that it’s a Very Good Idea to pay attention to reality, look for patterns, make connections, and do independent research.  That’s why you’re here, and that’s a beautiful thing. 

Speaking as a proud criminal, Benford’s Law has obvious and immediate applications in the realm of money laundering and data theft.  Remember the NYT excerpt that began this article: Benford’s Law is a “powerful and relatively simple tool for pointing suspicion at frauds, embezzlers, tax evaders, sloppy accountants and even computer bugs”.  As Professor Hill taught us (without even knowing he did it) if you’re going to fake the funk, fake the funk right.  A recent Physorg article provides this invaluable hint:

For example, because a year’s accounting data of a company should fulfill the law, economists can detect falsified data, which is very hard to manipulate to follow the law. (Interestingly, scientists found that numbers 5 and 6, rather than 1, are the most prevalent, suggesting that forgers try to “hide” data in the middle.)

“Election Forensics”

Here’s an immediately useful application: determining wether or not the results of a democratic election are legitimate.  Considering I live in a country where the past two—TWO—Presidential elections have been stolen in plain sight and I still have people telling me I should be excited about Barack Obama, this is pretty hip, dope, fresh and relevant to me.  We have covered electronic voting machines and the delicate art of stealing elections once before, so here’s a hopeful update:

Fraudulent elections and disputes about election outcomes are nothing new. Gumbel (2005) reviews the sorry history of deceit and electoral manipulation in America, going back to the dawn of the republic. Throughout the world, in old and new democracies alike, allegations of vote fraud frequently occur (Lehoucq 2003). One new element is voting technologies that make some familiar methods for physically verifying the accuracy of vote totals impossible to use. The advent of electronic voting machines means that often now there are no paper ballots to be recounted. To steal an election it is no longer necessary to toss boxes of ballots in the river, stuff the boxes with thousands of phony ballots, or hire vagrants to cast repeated illicit votes. All that may be needed nowadays is access to an input port and a few lines of computer code. To detect such manipulations is a difficult and urgent problem. In terms of legitimacy it is not clear whether the worse problem is that erroneous election outcomes may occur or that many may not believe that correct outcomes are valid.

Long story short, according the numbers, Mexico’s 2006 election was rigged, Venezuela’s 2004 election was crooked, and the US 2004 election was completely fraudulent in the state of Florida.  You can read the rest here.

Further Reading for Curious Primates

In 1986, Theodore Hill presented a “rigorous proof” of Benford’s Law which is surprisingly readable for the most part:

Proof of Benford’s Law

The white paper that introduced me to this concept—very readable:

How Do Numbers Begin?

Walter Mebane paper on using Benford’s law to analyze election results:

Election Forensics