Black Swan – Are All Swans White?
All the swans are white – Is It the only truth?
The Black Swan refers to those events which are considered unpredictable in the normal course of business. They are random, unexpected, but highly impactive. These events are considered outliers, because there is no past data which can point towards their occurrence soon. There is not any Hollywood’s scenario, film, or idea about them.
The philosopher John Stuart Mill wrote in A System of Logic in 1843 “all swans are white” to show how large numbers of consistent observations can encourage a wrong induction.
Bertrand Russell was yet another leading philosopher to invoke black swans in this way, in his 1912 book The Problems of Philosophy.
Karl Popper used the “black swan fallacy (error)” to show that scientific ideas can never be proven true, only falsified.
all swans in Australia were black
Before European settlement, all swans in Australia were black. The first white swans were introduced to Australia during the 19th century, in 1896 the white swan was introduced into Western Australia by British colonists. In the early 1900s, it is believed a Russian settler and the town’s mayor, Oscar Bernard, introduced white swans to Northam in Western Australia.
Surprisingly, the Avon River in Northam became the only place in Australia where the newly introduced bird survived and today it is still the only place in Australia where white swans breed naturally in the wild.
In adult Black Swans, the body is mostly black, except for the broad white wing tips which are visible in flight. The bill is a deep orange-red, paler at the tip, with a distinct narrow white band towards the end. Younger birds are much greyer in color and have black wing tips. Adult females are smaller than the males.
Definition
The Black Swan refers to those events which are considered unpredictable in the normal course of business. They are random, unexpected, but highly impactive. These events are considered outliers, because there is no past data which can point towards their occurrence soon. There is not any Hollywood’s scenario, film, or idea about them.
The Black Swan Theory refers to unpredictable events of massive scale. There is no scientific model which can predict these events. These events occur not just in business, politics, and nature but in stock markets as well.
Western knowledge systems are limited
Now, in public policy circles “black swan theory” is used to highlight the fragility of assumptions, but the theory also highlights the arrogance of western knowledge systems. Fragility = the quality of being easily broken or damaged
The black swan therefore highlights the need to weave (combine, mix) together indigenous (native, natural) and non-indigenous knowledge. Non-indigenous knowledge = scientific knowledge, based on numbers
You only get the full picture and a deeper understanding of the world with both. Public services and other organizations can learn so much from indigenous knowledge holders, and indigenous knowledge can make things better for everyone.
Black Swans – Know unknowns
The term, Unknown Unknowns, attributed to Donald Rumsfeld, refers to those situations where you don’t know you have a problem, so you don’t know that you need to apply resources to solve it.
Black swans and risk management
Fundamental assumptions in risk management: a) You can identify hazards in an exhaustive manner, b) the set of potential future events is delimited and known, c) no “unknown unknowns”.
You can estimate the probability of future events (p) and the magnitude of their consequences (r). If the risk is not acceptable to you, you can act to modify the probabilities or the magnitude of consequences. You cannot deal with this assumption in black swan’s events and situations with high uncertainty.
Typical examples of thin-tailed probability distributions: a) normal (Gaussian) distribution, and b) exponential distribution.
For a normal distribution: a) probability of an observation which is more than 6 standard deviations from the mean (a “six-𝜎 event”) is 9.9 ⋅ 10−10 = event is less frequent than one day in the life of the universe.
The “tail” of a probability distribution is the part which is far away from the mean. They represent extreme events. They carry significant weight. Together with high probability of something “unusual” occurring they follow a power law, like the Pareto distribution.
Pareto distribution
If X is a random variable with a Pareto (Type I) distribution, then the probability that X is greater than some number x, i.e., the survival function (also called tail function), is given by the formula.
Black swan payoff on prediction
Examples:
- Drilling for oil and gas exploration
- Portfolio risk on stock market, credit risk for big projects
- R&D in medicine
Note that:
- Despite its outlier status, it is often easy to produce an explanation for the event after the fact
- A black swan event may be a surprise for some, but not for others; it’s a subjective, knowledge-dependent notion
- warnings about the event may have been ignored because of strong personal and organizational resistance to changing beliefs and procedures
Winner takes all effects
Less than 0.25% of all the companies listed in the world represent around half the market capitalization. Less than 0.2% of books account for half their sales. The top 1% of bands and solo artists now earn 77% of all revenue from recorded music. Less than 0.1% of drugs generate a little more than half the pharmaceutical industry’s sales. “The rich get richer” (Add Bill Gates to your sample of thirty random people and mean wealth will jump by a factor of 100 000).
Winners convert more often to “golden goose” events (extreme upside) than to “black swan” events (extreme downside).
Mediocristan and Extremistan
Let’s play the following thought experiment. Assume you round up a thousand people randomly selected from the general and have them stand next to each other in a stadium….
Imagine the heaviest person you can think of and add him to that sample. Assuming he weighs three times the average, between four hundred and five hundred pounds, he will rarely represent more than a ridiculously small fraction of the weight of the entire population (in this case, about a half a percent.) … You can get even more aggressive. If you picked the heaviest biologically human on the planet (who yet can still be called a human), he would not represent more than, say, 0.6 percent of the total, a very negligible increase (p.32).
Consider by comparison the net worth of the thousand people you lined up in the stadium. Add to them the wealthiest person to be found on the planet — say Bill Gates, the founder of Microsoft. Assume his net worth to be close to $80 billion — with the total capital of the others around a few million. How much of the total wealth would he represent? 99.9 percent?
Suppose one randomly chooses a thousand authors and adds up the total number of books they have sold. Now, add the bestselling author in the world, J.K. Rowling, the author of the Harry Potter books. Her book sales vastly exceed the total of the other thousand authors
In Extremistan, inequalities are such that one single observation can disproportionately impact the aggregate, or the total.
Grey swans
Grey swan is a term used to describe a potentially significant event whose occurrence may be predicted beforehand but whose probability is considered small. In other words, it is a risk with a potentially significant impact but a low perceived likelihood of happening. Because there is a slight chance the event will occur it should be anticipated, particularly as it could shake up the world economy and stock market.
Grey swans can be positive or negative and significantly alter the way the world operates – which is why we should take them seriously. Examples of grey swans include climate change, population growth, and rising debt. They can also be called the small signals.
Black swan paradox in ML
The black swan paradox represents the event of something unexpected happening. In machine learning, this highlights the key difference between a frequentist and a Bayesians. More explicitly, it emphasizes the fact that a frequentist does not consider the prior in their probability computations whereas a Bayesian does.
Explanation:
- A hunter wakes up in the morning, looks out the window, and sees a black swan. The hunter is baffled (surprised and mixed) because this has never happened before! By the time he reaches for his gun, bang bang … the black swan has disappeared. Fascinated by his discovery, he runs into town to tell everyone.
- When the hunter explains this rare sighting, everyone begins to laugh at him and question his sanity = ability to think and behave in a normal and rational manner. They call him crazy and confused because black swans do not exist! “They have never been seen before!”. “It’s impossible!” they argue. The hunter knew no one would ever believe him unless he brought the proof. Thus, he went to hunt the swan.
- The hunter went out, found the black swan again, killed it, and brought it back to the village. Everyone was in shock because black swans are impossible.
This is the black swan paradox. The takeaway from this story is that we cannot rule out some hypothesis because we have never witnessed it before. That is why Bayesian statistics (in contrast to frequentism) have become so vital in machine learning today.
The Limits of Data Science
Data is about the past. “It’s important to remember that big data all comes from the past. A ‘black swan’ event can throw the most perfectly calibrated models into chaos. There are lots of examples. It’s not the models who failed by themselves, necessarily it’s that something unprecedented happened that changed everything. Unexpected changes are, by definition, rare, … but, as Nassim Taleb points out in Black Swan, they aren’t as rare as we think.
The signs of changes usually come from data points that we are or are not collecting, because they didn’t used to matter. It may be true that intuitions from people with their finger on the wind will be better at predicting Black Swan events than the smartest data scientist.
Are Black Swans Swimming in Big Data?
References: Based on a lot of internet resources. Thank you very much to you all.
