Benford’s Law

Posted on February 28, 2022 by memoriesandsuch

Numbers are all around us and with numbers come number patterns. And when we research number patterns, we come across something very interesting. Also known as the Newcomb–Benford law, the law of anomalous numbers or the first-digit law, Benford’s Law is a statistical statement about the occurrence of digits in lists of data and is an observation that in many real-life sets of numerical data, the leading digit is likely to be small.

According to the law, in sets that obey the law, the number 1 appears as the leading significant digit about 30% of the time, while 9 appears as the leading significant digit less than 5% of the time. If the digits were distributed uniformly, they would each occur about 11.1 % of the time. Benford’s Law also makes predictions about the distribution of second digits, third digits, digit combinations, and so on. The law is named after physicist Frank Benford, who stated it in 1938 in a paper titled “The Law of Anomalous Numbers”, although it had been previously stated by Simon Newcomb in 1881 and is similar in concept, though not identical in distribution, to the Zipf’s law. So according to Benford’s Law, the finding that the first digits or numerals to be exact of the numbers found in series of records of the most varied sources do not display a uniform distribution, but rather are arranged in such a way that the digit “1” is the most frequent, followed by “2”, “3”, and so in a successively decreasing manner down to “9”

The discovery of Benford’s law goes back to 1881 when the Canadian-American astronomer Simon Newcomb noticed that in logarithm tables the earlier pages that started with 1 were much more worn than the other pages. Newcomb’s published result is the first known instance of this observation and includes distribution on the second digit, as well. Newcomb proposed a law that the probability of a single number N being the first digit of a number was equal to log(N + 1) − log(N). The phenomenon was again noted in 1938 by the physicist Frank Benford, who tested it on data from 20 different domains and was credited for it. Benford’s data set included the surface areas of 335 rivers, the sizes of 3259 US populations, 104 physical constants, 1800 molecular weights, 5000 entries from a mathematical handbook, 308 numbers contained in an issue of Reader’s Digest, the street addresses of the first 342 persons listed in American Men of Science and 418 death rates. The total number of observations used in the paper was 20,229.

It has been shown that this result applies to a wide variety of data sets, including electricity bills, street addresses, stock prices, house prices, population numbers, death rates, lengths of rivers, and physical and mathematical constants. Like other general principles about natural data – for example, the fact that many data sets are well approximated by a normal distribution — some illustrative examples and explanations cover many of the cases where Benford’s law applies, though there are many other cases where Benford’s law applies that resist a simple explanation. It tends to be most accurate when values are distributed across multiple orders of magnitude, especially if the process of generating the numbers is described by a power-law, which is common in nature.

Benford’s law tends to apply most accurately to data that span several orders of magnitude. As a rule of thumb, the more orders of magnitude that the data evenly covers, the more accurately Benford’s law applies. For instance, one can expect that Benford’s law would apply to a list of numbers representing the populations of UK settlements. But if a settlement is defined as a village with a population between 300 and 999, then Benford’s law will not apply.

In general, it has been seen a series of numerical records follows Benford’s Law when they
represents magnitudes of events or events, such as populations of cities, flows of water in rivers or sizes of celestial bodies; do not have pre-established minimum or maximum limits; are not made up of numbers used as identifiers, such as identity or social security numbers, bank accounts, telephone numbers; and have a mean which is less than the median, and the data is not concentrated around the mean

This law can be utilised to detect patterns or the lack thereof in naturally occurring datasets. This can lead to important applications in data science such as catching anomalies or fraud detection. It’s expected that a large set of numbers will follow the law, so accountants, auditors, economists and tax professionals have a benchmark what the normal levels of any particular number in a set are.

In the latter half of the 1990s, accountant Mark Nigrini found that Benford’s law can be an effective red-flag test for fabricated tax returns; True tax data usually follows Benford’s law, whereas made-up returns do not. Ponzi schemes can be detected using the law. Unrealistic returns, such as those purported by the Maddoff scam, fall far from the expected Benford probability distribution.

In 1972, Hal Varian suggested that the law could be used to detect possible fraud in lists of socio-economic data submitted in support of public planning decisions. Based on the plausible assumption that people who fabricate figures tend to distribute their digits fairly uniformly, a simple comparison of first-digit frequency distribution from the data with the expected distribution according to Benford’s law ought to show up any anomalous results. In the United States, evidence-based on Benford’s law has been admitted in criminal cases at the federal, state, and local levels.

Walter Mebane, a political scientist and statistician at the University of Michigan, was the first to apply the second-digit Benford’s law-test (2BL-test) in election forensics. Such analyses are considered a simple, though not foolproof, method of identifying irregularities in election results and helping to detect electoral fraud. Benford’s law has been used as evidence of fraud in the 2009 Iranian elections. An analysis by Mebane found that the second digits in vote counts for President Mahmoud Ahmadinejad, the winner of the election, tended to differ significantly from the expectations of Benford’s Law and that the ballot boxes with very few invalid ballots had a greater influence on the results, suggesting widespread ballot stuffing. Another study used bootstrap simulations to find that the candidate Mehdi Karroubi received almost twice as many vote counts beginning with the digit 7 as would be expected according to Benford’s law, while analysis from Columbia University concluded that the probability that a fair election would produce both too few non-adjacent digits and the suspicious deviations in last-digit frequencies as found in the 2009 Iranian presidential election is less than 0.5%. Benford’s Law has also been applied for forensic auditing and fraud detection on data from the 2003 California gubernatorial election, the 2000 and 2004 United States presidential elections, and the 2009 German federal election.

Benford’s law has also been misapplied to claim election fraud. When applying the law to Joe Biden’s election returns for Chicago, Milwaukee, and other localities in the 2020 United States presidential election, the distribution of the first digit did not follow Benford’s law. The misapplication was a result of looking at data that was tightly bound in range, which violates the assumption inherent in Benford’s law that the range of the data is large.

Macroeconomic data the Greek government reported to the European Union before entering the eurozone was shown to be probably fraudulent using Benford’s law, albeit years after the country joined the EU. In genome data, the number of open reading frames and their relationship to genome size differs between eukaryotes and prokaryotes with the former showing a log-linear relationship and the latter a linear relationship. Benford’s law has been used to test this observation with an excellent fit to the data in both cases. The law has also been used successfully in scientific fraud detection. A test of regression coefficients in published papers showed agreement with Benford’s law. As a comparison group subjects were asked to fabricate statistical estimates. The fabricated results conformed to Benford’s law on first digits but failed to obey Benford’s law on second digits.

So if you want to test Benford’s Law yourself, it’s very simple. Just pick up a random book or magazine and list or sort the numbers. You will find about 30% of the numbers collected from any issue will start with the number 1. Let me know in the comments section if the law fit in your experiment.

Valentine’s Day

Posted on February 14, 2022 by memoriesandsuch

Today is Valentine’s Day which worldwide is celebrated as the day of love. Also called Saint Valentine’s Day or the Feast of Saint Valentine, Valentine’s Day is celebrated annually on February 14 and originated as a Christian feast day honouring one or two early Christian martyrs named Saint Valentine and, through later folk traditions, has become a significant cultural, religious, and commercial celebration of romance and love in many regions of the world.

There are several martyrdom stories associated with the various Valentines connected to February 14, including an account of the imprisonment of Saint Valentine of Rome for ministering to Christians persecuted under the Roman Empire in the third century. According to an early tradition, Saint Valentine restored sight to the blind daughter of his jailer and numerous later additions to the legend have better related it to the theme of love. An 18th-century embellishment to the legend claims he wrote the jailer’s daughter a letter signed “Your Valentine” as a farewell before his execution while another addition posits that Saint Valentine performed weddings for Christian soldiers who were forbidden to marry.

The Feast of Saint Valentine was established by Pope Gelasius I in 496 AD to be celebrated on February 14 in honour of Saint Valentine of Rome, who died on that date in 269 AD. The day became associated with romantic love in the 14th and 15th centuries when notions of courtly love flourished, apparently by association with the lovebirds of early spring. In 18th century England, it grew into an occasion in which couples expressed their love for each other by presenting flowers, offering confectionery, and sending greeting cards, known as valentines. By the 1900s printed cards began to replace written letters due to improvements in printing technology. Ready-made cards were an easy way for people to express their emotions in a time when direct expression of one’s feelings was discouraged. Cheaper postage rates also contributed to an increase in the popularity of sending Valentine’s Day greetings. The Valentine’s Day symbols that are used today include the heart-shaped outline, doves, and the figure of the winged Cupid. In Italy, Saint Valentine’s Keys are given to lovers as a romantic symbol and an invitation to unlock the giver’s heart, as well as to children to ward off epilepsy, also known as Saint Valentine’s Malady. Americans probably began exchanging hand-made valentines in the early 1700s. Today, according to the Greeting Card Association, an estimated 145 million Valentine’s Day cards are sent each year, making Valentine’s Day the second largest card-sending holiday of the year, after Christmas.

An official feast day in the Anglican Communion and the Lutheran Church, the day is not a public holiday anywhere. Many parts of the Eastern Orthodox Church also celebrate Saint Valentine’s Day on July 6 in honour of the Roman presbyter Saint Valentine, and on July 30 in honour of Hieromartyr Valentine, the Bishop of Interamna in modern-day Terni, Italy.

While the European folk traditions connected with Saint Valentine and St. Valentine’s Day have become marginalised by the modern Anglo-American customs connecting the day with romantic love, some remaining associations connect the saint with the advent of spring. While the custom of sending cards, flowers, chocolates and other gifts originated in the UK, Valentine’s Day remains connected with various regional customs in England. In Norfolk, a character called Jack Valentine knocks on the rear door of houses leaving sweets and presents for children with many children scared of this mystical person. In Slovenia, Saint Valentine or Zdravko was one of the saints of spring, the saint of good health and the patron of beekeepers and pilgrims as the belief is that plants and flowers start to grow on this day and has been celebrated as the day when the first work in the vineyards and the fields commences. The day is also said to mark the beginning of spring.

The oldest known valentine still in existence today was a poem written in 1415 by Charles, Duke of Orleans, to his wife while he was imprisoned in the Tower of London following his capture at the Battle of Agincourt. (The greeting is now part of the manuscript collection of the British Library in London, England.) Several years later, it is believed that King Henry V hired a writer named John Lydgate to compose a valentine note to Catherine of Valois.

Cupid is often portrayed on Valentine’s Day cards as a naked cherub launching arrows of love at unsuspecting lovers. But the Roman God Cupid has his roots in Greek mythology as the Greek god of love, Eros. Accounts of his birth vary; some say he is the son of Nyx and Erebus; others, of Aphrodite and Ares; still others suggest he is the son of Iris and Zephyrus or even Aphrodite and Zeus, who would have been both his father and grandfather. According to the Greek Archaic poets, Eros was a handsome immortal who played with the emotions of Gods and men, using golden arrows to incite love and leaden ones to sow aversion. It wasn’t until the Hellenistic period that he began to be portrayed as the mischievous, chubby child he’d become on Valentine’s Day cards.

Giving red roses may be an obvious romantic gesture today on Valentine’s Day, but it wasn’t until the late 17th century that giving flowers became a popular custom. The practice can be traced back to when King Charles II of Sweden learned the language of flowers which pairs different flowers with specific meanings on a trip to Persia and subsequently introduced the tradition to Europe. The act of giving flowers then became a popular trend during the Victorian Era, including on Valentine’s Day, with red roses symbolising deep love.

So here’s wishing everyone a very Happy Valentine’s Day. Remember, where there is love, there is life!

A VUCA World and how it impacts us

Posted on February 11, 2022 by memoriesandsuch

The past few years have shown us in no uncertain words how volatile our world is. Every week brings new changes and most of us are unable to make any plans because we don’t know what next week will bring us.

This is encapsulated very well in the acronym VUCA which stands for Volatile, Uncertain, Complex and Ambiguous which stands for what our world is today. First used in 1987, the acronym draws on the leadership theories of Warren Bennis and Burt Nanus, to describe or to reflect on the volatility, uncertainty, complexity and ambiguity of general conditions and situations. The U.S. Army War College introduced the concept of VUCA to describe the more volatile, uncertain, complex and ambiguous multilateral world perceived as resulting from the end of the Cold War and more frequent use and discussion of the term VUCA began from 2002. It has subsequently taken root in emerging ideas in strategic leadership that apply in a wide range of organizations, from for-profit corporations to education.

The deeper meaning of each element of VUCA serves to enhance the strategic significance of the VUCA foresight and insight as well as the behaviour of groups and individuals in organisations. It discusses systemic failures and behavioural failures, which are characteristic of organisational failure.

V stands for Volatility which is the nature and dynamics of change, and the nature and speed of change forces and change catalysts. It refers to the speed of change in an industry, market or the world in general. It is associated with fluctuations in demand, turbulence and short time to markets and it is well-documented in the literature on industry dynamism. The more volatile the world is, the more and faster things change.

U is Uncertainty or the lack of predictability, the prospects for surprise, and the sense of awareness and understanding of issues and events. Uncertainty refers to the extent to which we can confidently predict the future. Part of the uncertainty is perceived and associated with people’s inability to understand what is going on. Uncertainty, though, is also a more objective characteristic of an environment. Truly uncertain environments are those that don’t allow any prediction, also not on a statistical basis. The more uncertain the world is, the harder it is to predict.

C means Complexity which is the multiplex of forces, the confounding of issues, no cause-and-effect chain and confusion that surrounds organisations. It refers to the number of factors that we need to take into account, their variety and the relationships between them. The more factors, the greater their variety and the more they are interconnected, the more complex an environment is. Under high complexity, it is impossible to fully analyse the environment and come to rational conclusions. The more complex the world is, the harder it is to analyse.

And lastly, A stands for Ambiguity which encompasses the haziness of reality, the potential for misreads, and the mixed meanings of conditions and the cause-and-effect confusion. These elements present the context in which organizations view their current and future state, present boundaries for planning and policy management and come together in ways that either confound decisions or sharpen the capacity to look, plan and move ahead. It points to a lack of clarity about how to interpret something. A situation is ambiguous, for example, when information is incomplete, contradicting or too inaccurate to draw clear conclusions. More generally it refers to fuzziness and vagueness in ideas and terminology. The more ambiguous the world is, the harder it is to interpret.

The particular meaning and relevance of VUCA often relate to how people view the conditions under which they make decisions, plan forward, manage risks, foster change and solve problems. In general, the premises of VUCA tend to shape an organisation’s capacity to anticipate the issues that shape, understand the consequences of issues and actions, appreciate the interdependence of variables, prepare for alternative realities and challenges and interpret and address relevant opportunities. For most organisations, VUCA is a practical code for awareness and readiness.

So how can we try and navigate a VUCA World? Though it may seem inescapable in certain situations and industries, one can use it to advantage. The key to managing is to break VUCA down into its parts and to identify volatile, uncertain, complex, or ambiguous situations. Each type of situation has its causes and resolutions, so one should aim to deal with one at a time.

Counter volatility with vision. Accept and embrace change as a constant and don’t resist it.
Create a strong, compelling statement of objectives and values, and develop a clear, shared vision of the future. Have flexible goals which can be amended quickly.
Meet uncertainty with understanding which can help understand and develop new ways of thinking and acting in response to VUCA’s elements.
Make investing in, analysing and interpreting business and competitive intelligence a priority, so that one doesn’t fall behind. Stay up to date with industry news, and listen carefully to find out what others want.
Review and evaluate performance. Consider what one did well, what came as a surprise, and what one could do differently next time.
Simulate and experiment with situations, so that one can explore how they might play out, and how one might react to them in the future. Aim to anticipate possible future threats and devise likely responses. Gaming, scenario planning, crisis planning and role-playing are useful tools for generating foresight and preparing responses.
Communicate clearly because in complex situations, clearly expressed communication help to understand direction.
Develop and promote collaboration. VUCA situations are often too complicated for one person to handle, so strong teams that can work effectively in a fast-paced, unpredictable environment is essential.
Fight ambiguity with agility by promoting flexibility, adaptability and agility. Plan, but build in contingency time and be prepared to alter plans as events unfold.
Hire, develop and promote people who thrive in VUCA environments as these people are likely to be collaborative, comfortable with ambiguity and change, and have complex thinking skills.
Encourage people to think and work outside of their usual functional areas, to increase their knowledge and experience. Job rotation and cross-training can be excellent ways to improve agility.
Lead teams, but don’t dictate to or control them, instead develop collaborative environments and work hard to build a consensus. Encourage debate, dissent and participation from everyone.
Embrace an ideas culture. Reward team members who demonstrate vision, understanding, clarity, and agility.

When one is affected by VUCA, one has a choice. Either one allows VUCA to manage, overload and overwhelm them, or they accept and manage it so that they can mitigate its effects. When one decides to accept VUCA, they choose to make themselves and others less vulnerable and empower everyone to deal with uncontrollable, unpredictable forces.

World Wetlands Day

Posted on February 2, 2022 by memoriesandsuch

Today is the World Wetlands Day. Dating back to 1971, World Wetlands Day is an environmentally-related celebration that reaffirms the protection and love for wetlands, which are the small environments of plant life and organisms found within water bodies that bring about ecological health in abundance to not only water bodies but environments as a whole. First celebrated in 1997, World Wetlands Day serves to recognise the influence and positive production that Wetlands have had on the world and in terms brings communities together for the benefit of Mother Nature. This day also raises global awareness because wetlands play a significant role in people and on the planet. The day was formally acknowledged by the United Nations on 30 August 2021as World Wetlands Day. 2 February each year is World Wetlands Day to raise global awareness about the vital role of wetlands for people and our planet. This day also marks the date of the adoption of the Convention on Wetlands on 2 February 1971, in the Iranian city of Ramsar on the shores of the Caspian Sea.

A patch of land that develops pools of water after a rainstorm would not necessarily be considered a wetland, even though the land is wet. Wetlands have unique characteristics: they are generally distinguished from other water bodies or landforms based on their water level and on the types of plants that live within them. Specifically, wetlands are characterised as having a water table that stands at or near the land surface for a long enough period each year to support aquatic plants. A wetland is a distinct ecosystem that is flooded by water, either permanently or seasonally. Flooding results in oxygen-free or anoxic processes prevailing, especially in the soils. Wetlands are considered among the most biologically diverse of all ecosystems, serving as home to a wide range of unique plant and animal species and occur naturally on every continent, except for Antarctica. The water in wetlands is either freshwater, brackish or salt water and the main wetland types are classified based on the dominant plants and/or the source of the water. Wetlands contribute several functions that benefit people and are called ecosystem services and include water purification, groundwater replenishment, stabilization of shorelines and storm protection, water storage and flood control, processing of carbon including carbon fixation, decomposition and sequestration, other nutrients and pollutants, and support of plants and animals. Wetlands also place a role in climate change mitigation and adaptation. The world’s largest wetlands include the Amazon River basin, the West Siberian Plain, the Pantanal in South America, and the Sundarbans in the Ganges-Brahmaputra delta. According to the UN Millennium Ecosystem Assessment, wetlands are more affected by environmental degradation than any other ecosystem on Earth.

Over time, human construction has led to various ecological problems affecting wetlands. Overpopulation and construction have led to a decrease in environmental conservation and total has brought issues to these lands. Many wetlands are being lost and ecologists claim that humans should recognise this dilemma before the loss of a natural filter and conserver of the world.

The theme for the 2022 World Wetlands Day is Wetlands Action for People & Nature and today the Convention on Wetlands and its Contracting Parties will launch the next World Wetlands Day campaign to scale wetlands conservation actions for people and planetary health.

Mumbai Memories

Posted on January 26, 2022 by memoriesandsuch

A couple of days back, I was struggling to write something when I chanced upon some writing prompts. One of them stood out for me which said to write about a time when the community came together and displayed solidarity. And since I am in Mumbai right now, this prompt seemed apt to showcase the Mumbai spirit that I have witnessed.

The first experience was during the bomb blasts that shook the city in 1993. This was the first time something like this had happened. On March 12, 1993, a series of bomb blasts brought Mumbai, then known as Bombay, to a wailing stop. Twelve bombs went off within two hours and ten minutes that rocked several parts of the city, killing over 250 people. The explosions were the first large-scale coordinated terror attack to be carried out in the world and were the first terror attack where RDX was used as an explosive. The first bomb exploded in the basement of the Bombay Stock Exchange (BSE), followed by blasts in the Fisherman’s Colony in Mahim causeway, the Air India Building, Zaveri Bazaar, Hotel Juhu Centaur, Plaza Cinema, Hotel Sea Rock, Century Bazaar, Katha Bazaar, Worli and the Passport Office, over the next few hours.

My sister and I were in college then with my sister’s classmates coming from different parts of the city as it was a specialised college. There was chaos in the city and public transportation was shut down and if I remember correctly the city came under curfew. Because of the blasts, telephone exchanges were disabled or rather restricted. We could make calls to only numbers from the same exchange or possibly the next one. Educational institutions and offices all shut down and everyone was asked to go home and stay home. My sister called home in a fix because two of her classmates stayed in different parts of the city, and not at a walkable distance, so they were worried they could not get home. So we asked them to come home and my mum prepared extra food for the girls. They stayed in our home for the next two or three days until their families could pick them up. So how did the community come into the picture? Since we could not call their homes to let them know they were fine in our home, we started a phone chain. We called as many numbers as we could dial and in turn, asked them to try the girls’ numbers or if they could not get through, try to call as many people as they could and ask them, in turn, to try and call the girls homes. Everybody we called and they in turn called did this phone chain and by the end of the day, the parents of the girls knew about their whereabouts. This was the first time I saw what everyone always talks about the spirit of the city. Years later when I was working at my first job, I learnt from the old-time staff there that a bomb had blasted just meters away and they told me how they found burnt and charred pieces of flesh just outside the office complex. Everyone ran outside when heard the blast and tried to help as much as possible, both in terms of getting medical aid to those who needed it as well as food and water.

The next episode is a compilation of how people helped each other during the monsoon floodings that affect Mumbai every year. Since Mumbai is more or less reclaimed low lying land with an antiquated sewage system, inherited from the British during their rule, flooding during the monsoon season is par on course. The first one was in one of my first years of working. It had been raining for the whole day and by the time it was time to go home, the situation had become dire. I got on the bus and for some reason, fell asleep. When I woke up about 30 minutes later, I found that we were barely 10 minutes into a 30-minute journey and the bus had completely stopped. After waiting for another 10 minutes, I decided to just get off the bus and start walking. After walking for a while on a bridge, we realised that the bottom of the bridge was flooded and to add insult to the injury, there were open potholes. But then as we reached the bottom of the bridge, we found people from the nearby buildings helping those walking and walked them through the potholes, holding the hands of the elderly and those who had difficulty in walking. There were also women from the area with pots of hot tea and biscuits passing them around to the people. This was the pur Bombay spirit in action.

Another time was yet again when the rains played havoc in the city. This was either in 1999 or 2000 and I was working out of a small office of our company in south Mumbai. As before, it had started raining and the others in the office wanted to go back home. As the seniormost person in the office, I had to call the head office to get permission to leave around 2 pm. They gave us that permission after an hour after checking the situation. My sister who was also working close by suggested we take a taxi to get home and so we did. But because many areas of the city were waterlogged, we could only get as far as Dadar West, near Plaza to those who know the place. The taxi refused to move further because of the flooding and so we had to get out and walk. I remember at Khodadad Circle, the water was as high as my waist and I am not a very short person. Walking or rather wading in waist-deep water is pretty hard and by the time we got out of that area, we were exhausted. But there were so many people looking out for others during that hour-long walk that it didn’t feel so bad. A cousin who was doing an internship was stuck in Dadar station because the trains stopped running. After waiting for hours for the trains to restart, she finally called my mother and came to our home because there was no way she would make it to her house in one of the northern suburbs. Another friend who also lived in a far-flung suburb tried to get a lift from a colleague to get to a point from where she could go home was stuck in traffic for almost 7 hours with the car not moving an inch. Some people in the buildings nearby then came down and asked women and children to come up to their homes to spend the night because it would not be safe and that’s how she spent the night.

I could go on and on about stories about how Mumbai’s people are super resilient and always come together, especially when there is a tragedy, but I think these stories showcase this wonderful city with a heart of gold. Mumbai is my hometown and even though I no longer live here, it will always be home and have a very special place in my heart.

Memories and Such

Memories of a very ordinary life

Tag Archives: Writing