The probability puzzle that makes your head melt

Scott Lazarus our consultant managing the role
Posting date: 1/21/2015 1:38 PM

A reference in a recent Magazine article to the Monty Hall problem - where a contestant has to pick one of three boxes - left readers scratching their heads. Why does this probability scenario hurt everyone's brain so much, asks maths lecturer Dr John Moriarty.

Imagine Deal or No Deal with only three sealed red boxes.

The three cash prizes, one randomly inserted into each box, are 50p, £1 and £10,000. You pick a box, let's say box two, and the dreaded telephone rings.

The Banker tempts you with an offer but this one is unusual. Box three is opened in front of you revealing the £1 prize, and he offers you the chance to change your mind and choose box one. Does switching improve your chances of winning the £10,000?

Each year at my university we hold open days for hordes of keen A-level students. We want to sell them a place on our mathematics degree, and I unashamedly have an ulterior motive - to excite the best students about probability using this problem, usually referred to as the Monty Hall Problem.


This mind-melter was alluded to in an AL Kennedy piece on change this week and dates back to Steve Selvin in 1975 when it was published in the academic journal American Statistician.

It imagines a TV game show not unlike Deal or No Deal in which you choose one of three closed doors and win whatever is behind it.

One door conceals a Cadillac - behind the other two doors are goats. The game show host, Monty Hall (of Let's Make a Deal fame), knows where the Cadillac is and opens one of the doors that you did not choose. You are duly greeted by a goat, and then offered the chance to switch your choice to the other remaining door.

Most people will think that with two choices remaining and one Cadillac, the chances are 50-50.

The most eloquent reasoning I could find is from Emerson Kamarose of San Jose, California (from the Chicago Reader's Straight Dope column in 1991): "As any fool can plainly see, when the game-show host opens a door you did not pick and then gives you a chance to change your pick, he is starting a new game. It makes no difference whether you stay or switch, the odds are 50-50."

But the inconvenient truth here is that it's not 50-50 - in fact, switching doubles your chances of winning. Why?

Let's not get confused by the assumptions. To be clear, Monty Hall knows the location of the prize, he always opens a different door from the one you chose, and he will only open a door that does not conceal the prize.

For the purists, we also assume that you prefer Cadillacs to goats. There is a beautiful logical point here and, as the peddler of probability, I really don't want you to miss it.

In the game you will either stick or switch. If you stick with your first choice, you will end up with the Caddy if and only if you initially picked the door concealing the car. If you switch, you will win that beautiful automobile if and only if you initially picked one of the two doors with goats behind them.

If you can accept this logic then you're home and dry, because working out the odds is now as easy as pie - sticking succeeds 1/3 of the time, while switching works 2/3 of the time.

Kamarose was wrong because he fell for the deception - after opening the door, the host is not starting a new 50-50 game. The actions of the host have already stacked the odds in favor of switching.

The mistake is to think that two choices always means a 50-50 chance. If Manchester United play Accrington Stanley in the Cup then, with the greatest respect to proud Stanley, it's more likely that United will progress to the next round.

Still not convinced? You are in good company. The paradox of the Monty Hall Problem has been incredibly powerful, busting the brains of scientists since 1975.

In 1990 the problem and a solution were published in Parade magazine in the US, generating thousands of furious responses from readers, many with distinguished scientific credentials.

Part of the difficulty was that, as usual, there was fault on both sides as the published solution was arguably unclear in stating its assumptions. Subtly changing the assumptions can change the conclusion, and as a result this topic has attracted sustained interest from mathematicians and riddlers alike.

Even Paul Erdos, an eccentric and brilliant Hungarian mathematician and one-time guest lecturer at Manchester, was taken in.

So what happens on our university's open days? We do a Monty Hall flash mob. The students split into hosts and contestants and pair up. While the hosts set up the game, half the contestants are asked to stick and the other half to switch.

The switchers are normally roughly twice as successful. Last time we had 60 pairs in 30 of which the contestants were always stickers and in the other 30 pairs always switchers:

Among the 30 switcher contestants, the Cadillac was won 18 times out of 30 - a strike rate of 60%Among the 30 sticker contestants, there were 11 successes out of 30, a strike rate of about 36%So switching proved to be nearly twice as successful in our rough and ready experiment and I breathed a sigh of relief.

I had calculated beforehand the chances of ending up with egg on my face and the team of 30 stickers beating the 30 switchers. It was a risk worth taking, but one shouldn't play Russian Roulette too often.

Next year we will need something different, perhaps Simpson's Paradox. Imagine that 1% of people have a certain disease.

A diagnostic test has been developed which performs as follows - if you have the disease, the test has a 99% chance of giving the result "positive", while if you do not have the disease, the test has 2% chance of (falsely) giving the result "positive".

A randomly chosen person takes the test. If they get the result "positive", what is the probability that they actually have the disease? The answer, 1/3, is perhaps surprisingly low.


Click here for the article on the web.

Related blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our Blogs & News portal or check out the related posts below.

Weekly News Digest: 10th - 14th May 2021

This is Harnham’s weekly news digest, the place to come for a quick breakdown of the week’s top news stories from the world of Data & Analytics.       Personnel Today: Mental Health Awareness Week: Concerns up 24% from last year It was Mental Health Awareness week this week, and this year, the focus was on the theme of nature. Personnel Today revealed some worrying statistics on the back of research from Close Brothers into the state of the population’s wellbeing in 2021.  Reports of mental ill-health has increased by nearly a quarter since this time last year as a direct consequence of the stresses and strains of COVID-19. From yo-yoing in and out of lockdowns to extended periods of isolation, job uncertainty and illness, this year has been like no other and it’s most certainly taken its toll.  63 per cent of 16–34-year-olds report mental health worries, up a seventh from last year.For those who are 55+, this worry has risen by a third. In this piece, it is made clear that the underlying issue lies not only with COVID-19, but the lack of support given by employers. The research revealed that 70 per cent of employers don’t have a wellbeing budget in place, and only 8 per cent of firms invest more than £126 per employee each year in health and wellbeing.  To read the full research, visit Personnel Today here.  Towards Data Science: 5 unique skills every Data Scientist should know We know that career tip articles for Data Scientists can all feel pretty ‘samey’. But this article in Towards Data Science mixes up the usual advice, looking at how those in, or aiming to be in, the industry need to brush-up on their softer skills if they are to be successful.  Tips include: Cutting down the jargon in order to communicate effectively with stakeholders. Don’t be hasty to overpromise, or you’re at risk of seriously under-delivering. Become friendly with your team’s software engineer, they’ll only be able to help you be more efficient and effective in your role.  Of course, there has to be some mention of coding in there – it wouldn’t be a data-based article without it. Make sure you’re mastering your SQL Optimisation. Don’t leave your Git out in the cold, become familiar with the practice to ensure you can update your model code quickly.  To read the full article, click here.  Analytics India Mag: What SMBs can learn from Big Tech’s AI playbook? AI has come on leaps and bounds in a short space of time, and its popularity has boomed. For the monster-sized companies, where budget is of no question and innovation can happen overnight if need be - embracing AI has been a total no-brainer. Workflows become more efficient, technology becomes smarter, and the scope of growth seems infinite.  However, despite all the benefits of AI that are so regularly shouted about, it’s been clear since the birth of the technology that there’s a huge divide in those who can and those who cannot afford to implement this innovation.  Up until now.  In this piece from Analytics India Mag, author Ritka Sagar, highlights how SMEs are finally finding ways to become ‘inventive’ with how they implement and use AI systems without breaking the bank.  To read how SMEs are managing this, click here. Silicon Republic: For smart cities to work, they need to be neutral and objective The concept of a smart city seems like something out of a futuristic, sci-fi film but, in fact, they are closer to becoming a reality than we may think.  The idea being that urban areas use sensors and other electronic methods to collect data. From citizens to traffic, water supply networks to crime detection, all of these assets of life, and more, are monitored, data collected, and insights given to make ‘life’ more efficient.  On the surface, it’s all very cool, but there are, of course, worries that come with it. In this Silicon Republic article, Computer Scientist, Larissa Suzuki, discusses the importance of ‘neutral and objective’ smart cities if they are to work.  She says; “Data and services in smart cities must be neutral and objective when reporting information about the city environment. They should encompass the entire population and respect data licences, regulation and privacy laws,” she said. “In a similar fashion, the digital services and the backbone technology – including algorithms – should be free from any ideology or influence in their conception, operation, integration and dissemination.” To read more on the future of smart cities, visit Silicon Republic here. We've loved seeing all the news from Data & Analytics in the past week, it’s a market full of exciting and dynamic opportunities. To learn more about our work in this space, get in touch with us at info@harnham.com.     

Using Data Visualisation To Bring Data & Analytics To Life

The majority of the human population are visual learners. Our brains are wired in such a way where we can register 36,000 visual messages per hour, and visuals are processed 60,000 times faster than text. In short, one of the best ways to truly assimilate and understand new-found knowledge is through clear and digestible imagery.  Because of this valuable insight, we are now witnessing the fast-growing trend of Data Visualisation. Over the next six years, the value of Data Visualisation tools is expected to reach $19.2 billion, over double what it was in 2019.  Data & Analytics is one key area where data visualisation is used continuously. The raw data collected on a daily basis by Data Analysts can be incredibly time-consuming to sift through, not forgetting near-impossible to form palatable findings from. However, through the use of data visualisation tools such as graphs, heat maps, charts and infographics, confusing, text-based data can be transformed and brought to life. So, how can Data Visualisation help your business? Greater understanding of your data As Lydia, our Senior Recruitment Consultant, stated in her most recent article – data insights have the capability of not only improving decision-making, but also allow you to spot key trends, errors and predict future challenges. Nevertheless, all of these brilliant capabilities of data insights can only occur when teams can garner an in-depth understanding of the data being presented to them.  Without a background in statistics, which very few members of any team would possess, the raw data simply wouldn’t mean anything, and key insights could be missed. Utilising data visualisations not only makes data more tangible, but it also allows every team member to understand the data, make decisions and implement changes more efficiently. Standing out from the competition The effectiveness of Data Visualisation is no secret, and time and time again it’s been proved that this way of presenting data is far more likely to produce results than simply reviewing text.  Research within Analytics Insight reported that businesses using data discovery tools are 28 per cent more likely to find timely information compared to their dashboard-using counterparts, and 48 per cent of business intelligence users at companies with visualisation tools are able to find the information they need without the help of a specialist team.  Nevertheless, despite the incredible benefits, only 26 per cent of businesses globally are using data visualisation tools.  While the reasons for this slow uptake are varied, it’s clear that those companies who are willing to invest in Data Visualisation are far more likely to stand a head above their competitors. It can improve customer experience 98 per cent of companies will use data to help drive a better customer experience, but it doesn’t always mean that this data is collected, managed or presented well.  Data is, and should be, used as a way to back up what brands are saying, especially if they’re shouting from the rooftops about how fantastic they are.  When a business or brand uses accurate Data Visualisation to tell this story – for example, the percentage of consumers who report high levels of customer satisfaction, or the amount of money donated to CSR projects – audiences will respond much better than if the claim appears to be empty words without any evidence.  Data Visualisation is undoubtedly one of the most effective ways to communicate data, both internally and externally. The comprehensible formats available enables information to be processed with ease, and for learnings and understandings to be absorbed and implemented with much more efficiency than text-based raw data. It’s clear that this trend is only going to grow in popularity as businesses begin to put more investment behind it in order to reap the benefits and watch the positive impact on their bottom lines prosper.  For examples of how Harnham uses Data Visualisation, head over to our recent research reports.  If you're looking to take the next step in your career or build out your Data & Analytics team, we can help. Take a look at our latest opportunities or get in touch with one of our expert consultants to find out more. 

RELATED Jobs

Salary

US$155000 - US$175000 per annum

Location

New York

Description

Do you have a broad, in-depth knowledge of Machine Learning? Have you developed Media Mix Models and Attribution Models from scratch?

Salary

£80000 - £100000 per annum + Flexible Working

Location

London

Description

Lead a team of enthusiastic passionate optimisation specialists as well as build strategies that take optimisation to the next level!

Salary

US$100000 - US$120000 per annum

Location

Pittsburgh, Pennsylvania

Description

An autonomous mining company is developing robots to keep workers safe and mine for coal. They are looking to add a Computer Vision Engineer to their team.

recently viewed jobs