The probability puzzle that makes your head melt



A reference in a recent Magazine article to the Monty Hall problem - where a contestant has to pick one of three boxes - left readers scratching their heads. Why does this probability scenario hurt everyone's brain so much, asks maths lecturer Dr John Moriarty.

Imagine Deal or No Deal with only three sealed red boxes.

The three cash prizes, one randomly inserted into each box, are 50p, £1 and £10,000. You pick a box, let's say box two, and the dreaded telephone rings.

The Banker tempts you with an offer but this one is unusual. Box three is opened in front of you revealing the £1 prize, and he offers you the chance to change your mind and choose box one. Does switching improve your chances of winning the £10,000?

Marcus du Sautoy explains probability and the Monty Hall problem to Alan Davies on Horizon

Each year at my university we hold open days for hordes of keen A-level students. We want to sell them a place on our mathematics degree, and I unashamedly have an ulterior motive - to excite the best students about probability using this problem, usually referred to as the Monty Hall Problem.

This mind-melter was alluded to in an AL Kennedy piece on change this week and dates back to Steve Selvin in 1975 when it was published in the academic journal American Statistician.

It imagines a TV game show not unlike Deal or No Deal in which you choose one of three closed doors and win whatever is behind it.

One door conceals a Cadillac - behind the other two doors are goats. The game show host, Monty Hall (of Let's Make a Deal fame), knows where the Cadillac is and opens one of the doors that you did not choose. You are duly greeted by a goat, and then offered the chance to switch your choice to the other remaining door.

Most people will think that with two choices remaining and one Cadillac, the chances are 50-50.

The most eloquent reasoning I could find is from Emerson Kamarose of San Jose, California (from the Chicago Reader's Straight Dope column in 1991): "As any fool can plainly see, when the game-show host opens a door you did not pick and then gives you a chance to change your pick, he is starting a new game. It makes no difference whether you stay or switch, the odds are 50-50."

But the inconvenient truth here is that it's not 50-50 - in fact, switching doubles your chances of winning. Why?

Let's not get confused by the assumptions. To be clear, Monty Hall knows the location of the prize, he always opens a different door from the one you chose, and he will only open a door that does not conceal the prize.

For the purists, we also assume that you prefer Cadillacs to goats. There is a beautiful logical point here and, as the peddler of probability, I really don't want you to miss it.

In the game you will either stick or switch. If you stick with your first choice, you will end up with the Caddy if and only if you initially picked the door concealing the car. If you switch, you will win that beautiful automobile if and only if you initially picked one of the two doors with goats behind them.

If you can accept this logic then you're home and dry, because working out the odds is now as easy as pie - sticking succeeds 1/3 of the time, while switching works 2/3 of the time.

Kamarose was wrong because he fell for the deception - after opening the door, the host is not starting a new 50-50 game. The actions of the host have already stacked the odds in favor of switching.

The mistake is to think that two choices always means a 50-50 chance. If Manchester United play Accrington Stanley in the Cup then, with the greatest respect to proud Stanley, it's more likely that United will progress to the next round.

Still not convinced? You are in good company. The paradox of the Monty Hall Problem has been incredibly powerful, busting the brains of scientists since 1975.

In 1990 the problem and a solution were published in Parade magazine in the US, generating thousands of furious responses from readers, many with distinguished scientific credentials.

Part of the difficulty was that, as usual, there was fault on both sides as the published solution was arguably unclear in stating its assumptions. Subtly changing the assumptions can change the conclusion, and as a result this topic has attracted sustained interest from mathematicians and riddlers alike.

Even Paul Erdos, an eccentric and brilliant Hungarian mathematician and one-time guest lecturer at Manchester, was taken in.

So what happens on our university's open days? We do a Monty Hall flash mob. The students split into hosts and contestants and pair up. While the hosts set up the game, half the contestants are asked to stick and the other half to switch.

The switchers are normally roughly twice as successful. Last time we had 60 pairs in 30 of which the contestants were always stickers and in the other 30 pairs always switchers:

Among the 30 switcher contestants, the Cadillac was won 18 times out of 30 - a strike rate of 60%
Among the 30 sticker contestants, there were 11 successes out of 30, a strike rate of about 36%
So switching proved to be nearly twice as successful in our rough and ready experiment and I breathed a sigh of relief.

I had calculated beforehand the chances of ending up with egg on my face and the team of 30 stickers beating the 30 switchers. It was a risk worth taking, but one shouldn't play Russian Roulette too often.

Next year we will need something different, perhaps Simpson's Paradox. Imagine that 1% of people have a certain disease.

A diagnostic test has been developed which performs as follows - if you have the disease, the test has a 99% chance of giving the result "positive", while if you do not have the disease, the test has 2% chance of (falsely) giving the result "positive".

A randomly chosen person takes the test. If they get the result "positive", what is the probability that they actually have the disease? The answer, 1/3, is perhaps surprisingly low.


Click here for the article on the web.



<< Click here to see more recent news articles >>

 

 

Harnham blog & news

With over 10 years experience working solely in the Data & Analytics sector our consultants are able to offer detailed insights into the industry.

Visit our News & Blogs portal or check out our recent posts below.

MACHINE LEARNING ENTERS BIOINFORMATICS AND ITS FUTURE IS BRIGHT

Machine Learning Enters Bioinformatics and its Future is Bright

Ever wondered how your email system knows which emails to show you and which to put in your junk or spam folder? Enter Machine Learning. It learns what you open and read and after a time can differentiate what you ignore, toss, or move to spam. Now imagine that same type of learning in the life sciences. As scientific advances move toward Data and Machine Learning to scale their knowledge, you can imagine the possibilities. After all, as you read this, trends in the life sciences, specifically with an eye toward bioinformatics showcase machine learning such as genome sequencing and the evolutionary of tree structures. Human and Machine Learning with a Common Goal There has been so much data provided over the past few decades, no mere mortal could possibly collect and analyze it all. It is beyond the ability of human researchers to effectively examine and process such massive amounts of information without a computer’s help.  So, machines must learn the algorithms and they do so in any number of ways. For the most part, it’s a comparison of what we know, or is already in a databank, with the information we have and don’t yet know. Unrecognized genes are identified by machines taught their function. The Future is Bright Machine Learning is giving other fields within the life sciences both roots and wings.  Imagine scientists being able to gain insight and learn from early detection predictions. This type of knowledge is already in play using neuroimaging techniques for CT and MRI capabilities. This is useful on a number of levels, not the least of which is in brain function; think Alzheimer’s Research, for example.  The hurdle? It isn’t the availability of such vast amounts of data, but the available computing resources. Add to that, humans will be the ones to check and counter-check validity which can in turn become more time-consuming and labor intensive than the computer’s original analysis. And it’s this hurdle which leads to a caveat emptor, or “buyer beware” of sorts. Caveat Emptor: Continue to Question Your Predictions In other words, how much can you trust the discoveries made using Machine Learning techniques in bioinformatics? The answer? Never assume. Always double check. Verify. But as you do so, know this. Work is already in progress for next-generation systems which can assess their own work.  Some discoveries cannot be reproduced. Why? Sometimes it’s more about asking the right question. Currently, a machine might look at two different clusters of data and see that they’re completely different. Rather than state the differences, we’re still working on a system that has the machine asking a different kind of question. You might think of it as a more human question that goes a bit deeper.  Imagine a machine that might say something noting the fact that some of the data is grouped together, but if different, it might say while it sees similarities, but am uncertain about these other groups of data. They’re not quite the same, but they’re close.  Machine Learning is intended to learn from itself, from its users, and from its predictions. Though a branch of statistics and computer science, it isn’t held to following explicit instructions. Like humans, it learns from data albeit at a much faster rate of speed. And its possibilities are only getting started. Want to see where Bioinformatics can take your career? We may have a role for you. If you’re interested in Big Data and Analytics, take a look at our our current vacancies or contact one of our recruitment consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

How to lead a Data team

How To Lead A Data Team

Dream teams from sports to business are an ideal everyone aspires to live up to. But what is it every basketball or football dynasty has which makes them a dream team? What is it that brings individuals together to overcome odds, set examples, find solutions, and create the next best thing? Good management.  The need for good management is no different in the Data Science world. Yet according to our latest Salary Guide, poor management is one of the top five reasons Data professionals leave companies. So, let’s take a look at what poor management is, what causes it, and how businesses can better retain Data talent. What’s Your Data Science Strategy? Most businesses know they need a Data team. They may also assume that a Data Scientist who performed well can lead a Data team. But that isn’t necessarily the case. Managers have to know things like P&L statements, how to build a business case, make market assessments, and how to deal with people. And that’s just for a start.  The leader of a Data team has a number of other factors to consider as well such as Data Governance, MDM, compliance, legal issues around the use of algorithms, and the list goes on. At the same time, they also need to be managing their team with trust, authenticity, and candor. The list of responsibilities can be daunting and if someone is given too much too soon and without support, it can be a recipe for disaster. Other businesses might believe that a top performing Data Scientist would make a good manager. Yet these are two different fields. Or you might look at it this way. If you are willing to upskill a top performing Data professional and train them in managerial skills, giving them the education and support they need, that is one solution. Another solution is to create a Data Science strategy which brings in people with business backgrounds. Data Science is a diverse field and people come from a number of backgrounds not just Computer Science or Biostatistics, for example.  Now that you’ve seen what might cause a manager to fail, let’s take a look at a few tips to help you succeed. Seven Tips for Managing a Data Team Managing a team is about being able to hire, retain, and develop great talent. But if the manager has no management training, well, that’s how things tend to fall apart. Here a few tips to consider to help ensure you and your team work together to become the dream team of your organization: Build trust by caring about your team. Help define their role within the organization. Ensure projects are exciting and that they’re not being asked to do project with vague guidelines or unrealistic timeframes.Be open and candid. Remember, Data Scientists are trained in how to gather, collect, and analyze information. If anyone can see right through a façade, it will be these Data professionals. Have those “tough” conversations throughout every stage of the hiring, onboarding, and day-to-day, so that no one is caught unaware.Offer consistent feedback. And ask for it for yourself as well from your team.Ensure your team understands the business goals behind their projects. Let them in on the bigger picture. Think long-term recruitment for a permanent role, not short-term. If you have an urgent project, consider contracting it out. Prioritize diversity to include academic discipline and professional experience. Does the way this person view the world expand the knowledge of your team’s knowledge? Dream teams don’t always have to agree. Sometimes, the best solutions are found when there are other opinions. Finding the perfect, “Full Stack” Data Scientist or Data Engineer or Analyst is not impossible, and retaining them can be even easier. If you’ve done your job well, your team will trust you, have a balanced skillset, and understand how their work supports the organization and its goals. For more information on how to be a great manager, check out this article from HBR.  Ready for the next step?  Check out our current vacancies or contact one of our recruitment consultants to learn more. For our West Coast Team, call (415) 614 - 4999 or send an email to sanfraninfo@harnham.com.  For our Mid-West and East Coast Teams, call (212) 796 - 6070 or send an email to newyorkinfo@harnham.com.

AI and Predictive Analytics Check-in to Hospitality

With summer in full swing, many of us are either planning our vacations, or have already enjoyed them and are thinking of where to go next. Regardless of location, we’re all looking for the same thing; a great experience to remember for years to come. No matter how exciting our trip, we all want our plans to run smoothly and, luckily, AI is here to help.  Today we have more options and more buying power than ever before. The ease with which we can search and select via our phones has kept businesses on their toes and driven them to look beyond traditional service.  By incorporating AI, the hospitality sector is implementing new ways to serve their customers more easily and efficiently. Fueling the AI Hospitality Experience The hospitality industry has a notoriously high turnover rate, relying heavily on seasonal workers, and those early in their careers. But, with AI, digital analysis, and predictive analytics entering the industry, new technologies are providing alternative customer service solutions: Predictive Analysis  Automation  Smart Domotics Advertising Predictive Analysis  As ‘the customer is always right’, the best way to create a smooth and memorable experience is to know what they want and then give it to them.  Given that difficulties can arise when there are too few staff for the number of guests, there is a need to be proactive when planning, in order to be reactive on the day. Utilizing Machine Learning, facilities can predict staffing and supply needs, planning for a more streamlined and, ultimately, better service.  Automation  Automating repetitive operations such as check-ins and check-outs, room assignments, and housekeeping deliveries gives staff more time to focus on the customer. As small and large hospitality businesses compete with the growing success of home sharing platforms, such as AirBnB, AI can give traditional facilities a fresh edge. In addition, rapid and efficient responses lead to greater customer satisfaction which, in turn, leads to a healthier bottom line.  Smart Domotics  More and more hotels are looking to the Internet of Things and Linked Technologies as they evolve into ‘smart’ destinations. With devices that can measure everything from room temperature to customer preferences, facilities can adapt in order to create an optimal environment. Furthermore, interaction with these ‘smart’ technologies can help hotels evolve over time, placing a greater emphasis on elements that prove to be the most popular with customers.  Advertising  From targeted Social Media campaigns to personalized gifts on arrival, Analytics can enhance the entire customer experience. When booking, users can engage with Chatbots 24/7, adding an element of humanity to the online booking experience.  When customers engage with resort apps and website, AI technologies cross-check their interactions and adapt their recommendations accordingly. With more people travelling than ever, the effort of keeping up with travelers the world over, night and day, is shifting to AI, thereby allowing the workforce more freedom to tend to customer needs.  AI in the Cloud The world of digital is transforming our lives, and the rise of Cloud technologies has taken digital analysis to the next level. With the advancements in AI, the hotel industry needs professionals who can create apps, collect and translate data, and, of course, build rigid infrastructures.  If you want to help hotel owners get a leg up on their competition and have a hand in creating a memorable travel experience for someone, we may have a role for you. To learn more, check out our current vacancies.  For the West Coast team call us at (415) 614-4999 or email us at sanfraninfo@harnham.com. For our Mid-West and East Coast Teams call us at (212) 796-6070, or email newyorkinfo@harnham.com.

Why Texas is the place to be for technology jobs

Why Texas is the place to be for technology jobs The big data market is heating up the world over, and perhaps no more so than in Texas. The Dallas, Austin and Houston areas in particular are experiencing a massive boom in big data jobs, with many large tech companies making the move from Silicon Valley to enjoy all that Texas has to offer. But why the shift towards the southern state, and what does it mean for candidates looking big data jobs and broader technology roles? Tax-free Texas The Texan market is looking increasingly lucrative for both young start-ups and established tech companies alike.  One of the most significant factors in this rapid growth is the favourable tax conditions in the state. There’s no corporate or individual income tax, with Texas ranking 47 out of 50 states when it comes to taxes paid per $1,000 of personal income. As California tax rates hitting up to 10.84 for corporations and 12.3% for individuals, it’s understandable that entrepreneurs and big business alike are looking to the southern state for bigger breaks on tax day. On top of this, Texas offers favourable funding and regulatory conditions for young and growing businesses, providing a ‘pro-business’ environment for corporations to thrive. Texas State offers billions of dollars in incentives to businesses every year, providing all the more reasons for those in the technology industry to think hard about making the move. With a state government that celebrates business and provides easy to navigate laws and regulations, many businesses find the transition from Silicon Valley to Austin smooth and seamless. As organisations in San Francisco are priced out of the area, some of the nation’s top talent are moving to pastures greener – and for many, that means Texas. The living is easy On top of the tax breaks gained when moving to Texas, many movers and shakers experience a favourable quality of life. The cost of living is low – for example, the median home value in Austin is $321,600 compared to San Francisco’s $1,1943,300 – with relatively cheap utilities and the second-largest GDP in the nation. The market is robust, which has resulted in money being poured back into cities and communities to make them more attractive to businesses and young families. People can move to Austin and get more bang for their buck than they can in many other parts of the country, enjoying not only a booming technology market, but also superior housing and affordable living. Add in a comfortable climate and famously friendly locals and you’ve got a part of the country that is becoming increasingly appealing to even the most seasoned technology professionals. Technology is taking off Texas is huge when it comes to the technology industry. There was a 41.4% jump in technology industry employment between 2001 and 2013, resulting in large numbers of jobs being taken up across Austin and the wider state. And in 2016 alone, Texas added a huge 11,000 new technology jobs to its market, ranking it second of the 50 states in tech industry employment. The tech hub of Austin alone is home to employers such as Dell, Apple, Microsoft and Samsung, plus an increasingly significant number of start-ups peppering the landscape with innovation. There are a range of incubators and universities that feed into the city’s talent pool, with Austin ranking third in the list of US cities providing the most technology jobs in 2017. However, such growth doesn’t stop Austin and its other Texan counterparts from being a friendly and accessible place to work. There is less of the cut-throat nature that comes with tech in Silicon Valley, and more of a community, collaborative approach. Meanwhile, Dallas-Fort Worth is enjoying being the second-largest data center market in the country, offering an abundance of big data jobs to savvy business people.  Working in Texas Much of the Texas technology market is geared towards candidates currently, with more jobs than skilled employees to fill them. Companies are doing more to attract top talent to Texas, including offering generous benefits packages, relocation allowances and flexible work conditions, and the expectation is that this market will only continue to grow. If you’re looking for technology jobs in Austin or further afield in Texas, we might have just what you’re looking for. Take a look at our US data and technology jobs here.

Recently Viewed jobs