Sunday, December 29, 2013

CitiBike share--what are the chances?

I have been working with Joe Jansen on the Citibike data in the R Language.  Citibike is New York's bike sharing program, which started in may and currently has more than 80,000 annual members.  The R Language is a freely available object oriented programming language designed originally for doing statistics at Bell Labs.

Joe has downloaded all the data and done an extensive analysis, which you can find here.  I did a simpler analysis predicting trips using a statistical regression model and graphed it using the function ggplot2 in R.  I found that maximum temperature, humidity, wind, and amount of sunshine to be significant factors in predicting the number of trips that will be taken on any given day.  While rain was not a significant factor, it is likely confounded with sunshine, so it is only not a factor after accounting for amount of sunshine.  Also, keep in mind that a number of days with rain, especially in the summer, are generally sunny days with an hour or two of rain or thunderstorms.  The day of the week, surprisingly, was not an important factor influencing number of trips.  The R-squared, which is a typical measure of predictive power and is on a scale from 0 to 100%, was more than 70%.

Here is a graph of the results that shows the predicted number of trips per 1,000 members versus the actual number of trips.  The day of the week is indicated by the color of the point.
I am an amateur with the function ggplot, and so the legend for day of the week has the days of the week in alphabetical order rahter than Monday , tuesday, etc.  Help on that and other aspects of ggplot for this graph would be welcome (please comment accordingly).

If day of the week made a difference, for any given point on the x-axis (predicted trips) you would have more of a certain color that is high on the y-axis than other colors.  For example, if more trips occurred on weekends, you would have more of the green colors (Saturday and Sunday) on top.  However, no such affect seems to exist.  I guess people are enjoying Citibike every day of the week, or casual riders on the weekends are roughly making up for weekday commuting riders.

Monday, November 25, 2013

Highest property taxes in America?

I read on CNN's money website today that Westchester County, NY has the highest property taxes in America (see Nov 25 Money website). Moreover, the New York area in general seems to have the highest taxes.  That surprises me, because, as an owner of a co-op in Brooklyn, I know that my property taxes, and property taxes in general in the city, are extremely low.

So what's the problem?  If you click on the "interactive graph" you find that you can display results in two ways.  The headline and accompanying map refers to the taxes in dollars.  This type of information is little more than a graph of housing prices in the US, because expensive houses have higher taxes than cheaper houses.  Sure, tax rate comes into play, but the owners of a $10 million mansion in a low tax district still generally pay more property taxes per year than the owners of a $200,000 house in a high tax district.

Here's an example.  Click on Brooklyn on the interactive map and you will see taxes of $3,050.  Click on Richland County, South Carolina (where my parents live), and you will see that taxes average $1,129, nearly one-third the "high" taxes of Brooklyn.  Yet this belies the fact that housing prices are much higher in Brooklyn.

How much higher?  Well, to see this, go to the interactive map that shows taxes as a percentage of home prices.  This map accounts for different housing costs and shows taxes in the familiar manner, as a rate.  In this map, you can see that Brooklyn property taxes are 0.53% of housing prices and Richland County's are 0.75%.  (By the way Westchester County is 1.76%, which is high but certainly not the highest).

Thus, while taxes on the map shown in the headline are nearly three times higher in Brooklyn than in Richland County, S.C., taxes are actually 30% lower in Brooklyn, when looked at as a percentage of home prices.

Monday, August 12, 2013

What are the chances of different "splits" in bridge?

If you know how to play bridge, skip to the fourth paragraph!
In bridge, 13 cards are dealt to each of 4 players (so all 52 cards are dealt).  Players sitting across from each other are partners, so we could think of the two teams positions as North and South and East and West on a compass.  A process of "bidding" ensues, in which the team with the highest bid has selected a "trump" suit and a number of rounds, or "tricks" that they have contracted to take.

Suppose North-South had the highest bid and North is playing the hand.  Then East "leads" a card, meaning East places a card (any card he/she wants) face up on the table.  The play goes clockwise, East-> South-> West -> North.  South, West and North must play a card of the same suit that East played.  When four cards are down, the highest one wins the "trick" and that winner puts any card of his/hers down, in order to begin a new trick.  Play continues until 13 rounds of 4 cards each have been played.

Suppose that West wins a trick and thus gets to play a card.  He plays the Ace of Hearts.  North, who is next and otherwise required to play hearts, is out of hearts.  North can play any other suit, but if he chooses to play the "trump" suit (say Spades are trump), then he automatically wins the trick unless East or South is also out of hearts and play a higher card in Spades (the trump suit).  In other words, trumps are very valuable.  In the bidding process, the teams try to bid in such a way that the trump suit is one in which they have a lot of cards.  Generally, the team with the winning bid (the "contract") will have at least 7 of the 13 trumps between the two of them, meaning the other team will have 6 or fewer.  Whatever the number the opponents have, it is generally advantageous to the contract winners if they have the same number each rather than them being skewed to one or the other opponent.

Bridge players begin here:
So here is the probability piece.  Suppose you and your partner hold 7 trumps between you, what are the chances the opponents each have 3?  have 4 and 2?  have 5 and 1?  have 6 and 0?  To solve this sort of problem, we use combinations.  See my earlier post for some detail (and more odds of bridge hands).

The opponents have 26 cards altogether and we want to know the number of different groups of six among those 26 cards.  Think of this process as a process of picking six cards from the 26.  You have 26 choices for the first card, 25 for the second, and so on, and thus there are 26*25*24*23*22*21 total 'permutations' of size 6.  However, we do not care what order they are in so for each first card, there are 6 possible positions, for each second card, 5, etc., and thus we need to divide these permutations by 6*5*4*3*2*1, in order to get the number of unique sets when order does not matter. Again, see my earlier post for a more detailed explanation of this concept.

The R language allows for calculation of this combination of 6 out of 26 with the command "choose(26,6)." This is the denominator when we calculate probabilities, because it gives the total number of equally likely combinations of 6 cards.  The numerator is split into the two bridge hands of 13 cards each.   The number of combinations with an even 3-3 split are "13 choose 3" for both hands.
To calculate that probability in R, we write:   choose(13,3)*choose(13,3)/choose(26,6) and get 35.5%

How about hands with a 4-2 split?  That is the chance that Opponent 1's hand has 4 trumps multiplied by the chance that Opponent 2's hand has 2 trumps PLUS the chances that Opponent 2's hand has 4 trumps multiplied by the chance that Opponent 1's hand has 2 trumps.  Since the chance that either Opponent has 4 are the same, we can just double the probability of Opponent 1 having 4 and Opponent 2 having 2.  We get: choose(13,4)*choose(13,2)*2/choose(26,6) = 48.4% of one opponent having 4 and the other having 2 trumps.

Continuing this calculation, we get the following chances for hands with 6 trumps in the opponents hands( 6 trumps "out"):
3-3 split : 35.5%
4-2 split: 48.4%
5-1 split: 14.5%
6-0 split:  1.5%

For hands with 5 trumps out, we get:
3-2 split: 67.8%
4-1 split: 28.3%
5-0 split: 3.9%

For hands with 4 trumps out:
2-2 split: 40.7%
3-1 split: 49.7%
4-0 split: 9.5%

For hands with 3 trumps out:
2-1 split: 78%
3-0 split: 22%

For hands with 2 trumps out:
1-1 split: 52%
2-0 split: 48%

I find it interesting that the even split (for 2, 4, or 6 trumps out) is only the most likely scenario when 2 trumps are out.  When 4 trumps are out, a 3-1 split is more likely.  When 6 are out, a 4-2 split is more likely.

Monday, April 29, 2013

Simpson's Paradox

A North Slope real estate broker (named North) is trying to convince you that North Slope is a more affluent neighborhood than South Slope.  To prove it, he explains that professionals in North Slope earn a median income of $150,000, versus only $100,000 in South Slope.  Working class folks fare better in North Slope also, with hourly workers making $30,000 a year to South Slope's $25,000.

The South Slope real estate broker (named South) explains that North is crazy.  South Slope is much more affluent.  The median income in South Slope is $80,000 versus the North Slope median of $40,000.

Question: Who is lying, North or South?
Answer: It could be neither.
Consider the breakdown of income shown below.

We can see that North is not lying.  Half the hourly South Slope workers earn $20K and half $30K, for a median of 25K.  A similar calculation for the North Slope workers yields an hourly median of 30K.  For professionals in the South Slope, the median is $100K, with half earning $80K and half earning $120K.  In the North Slope, a similar calculation yields the median of $150,000.

South is not lying either.  For the South Slope, the median is $80,000, since more than half of the workers make less than or equal to $80,000 and more than half make $80,000 or more (according to the definition of median, at least half must be above the median and at least half must be below).  For the North Slope, the median is $40,000.

What happened here?  The problem, and the reason for the conflict between the wages according to type of work and the overall wages, is that the percentage of residents in each category does not match.  Thus, though professionals and hourly workers make more in the North Slope, there are far more hourly workers in the North Slope than in the South Slope.  Thus, the overall median (or mean) income is lower in the North Slope.

While Wikipedia has an entry for Simpson's Paradox, a specific example of which I described above, it seems that most people are unaware of it.  My motivation for writing about it is not the made-up example I present above but the fact that I encounter it so much in my everyday work.  I either make my clients very happy by explaining that the 'bad' effect they have found may well be spurious or, anger them when I explain the interesting relationship they have found is a mere statistical anomaly.

Monday, November 12, 2012

The Worst Graph

One reason for quotes like there are "there are lies, damn lies, and statistics" is because of graphs like these:

This was on the front of this morning with the caption: "Huge US Oil Boom ahead: The U.S. will overtake Saudi Arabia to become the world's biggest oil producer before 2020."

I was shocked at first glance, because I thought oil production was going to go up 10 or 20-fold from the tiny amount in 2011 to the huge amount in 2015.  That does indeed sound huge.  Then I looked at the left y-axis, where I can see it is only going from 8 million to 10 million barrels a day, an increase of about 25%.  

Fine, you say, but you can still easily see that the light blue bar is above the dark blue bar starting in 2025, showing the US overtakes Saudia Arabia.  

I'm afraid not.  The two bars are not Saudia Arabia versus US production but oil versus gas production, and it is not even clear whose production is depicted.  Is the the whole world, the US, Saudi Arabia?  The article puts US production at 5.8 million barrels a day in 2011, so it appears not to be US production, but other sources put it at closer to 9 million, so maybe it is the US.  

Ok, you say, despite the poor caption, at least you can clearly see that gas production begins to top oil production (in whatever country the graph is depicting) around 2025.  

Not really.  Since oil is in millions of barrels per data and the gas is in billions of  cubic meters (per day, per month, per year, who knows?), this is actually not the case.  The year 2030 shows oil at about 10 million barrels per day and gas at nearly 800 billion cubic meters.  Which is more?  Maybe the readers of money can quickly translate these figures into BTUs or some useful measure of production output, but I sure can't tell you.

Fine, you say, but since they start at about the same level, we at least know that gas increases more than oil over the time period.  

Sorry, even that is incorrect.  Look at the scale on the left axis (oil), which starts at 8 and goes to 12, a 50% increase.  The right axes starts at 600 and goes to 800, a 25% increase. Thus, oil goes from 8 to just over 10 (more than a 25% increase) while gas goes from a little over 600 to just little under 800 (less than a 33% increase--maybe a little more than oil but maybe not).

The only thing that appears to be correct about this graph is the year, until you realize that in the first period, there are only four years (2011-2015) while in the other periods, there are five year differences.

Friday, September 21, 2012

Election Polls

With the upcoming election, I have been following my favorite prediction site:  That site has a big map showing current predictions state-by-state as well as the overall electoral vote prediction.  It also shows the senate predictions.  It has been amazing accurate in the past (though, of course, this doesn't mean that the sites predictions wont change considerably between now and the election). The predictions are all based on some sort of averaging of polls, and the site shows the results of each poll.  What I have found interesting (and it has been noted on the site) is that some polls appear to lean toward Obama while others lean toward Romney.  In other words, the polls appear to have biases.

Why?  Theories abound about this, and much of it comes down to the polling methodology.  The most compelling reason I have seen comes from Nate Silver's blog on the New York Times site.  Silver's blog compares traditional polls, which call only land-line phones, with more modern polls, which call cell phones along with land-lines.

As shown by a chart in Silver's blog,  there is a clear and consistent difference in every swing state between the two types of polls, with modern polls leaning toward Obama.  This is consistent with the idea that younger people are both likely to vote for Obama and also more likely to not have landlines.  This issue has been pointed out before, and a Pew Research Report in 2010 noted substantial differences in party affiliation between voters who had a landline and those who only had a cell phone.

There is no doubt that the percentage of homes without landlines is rising rapidly.  See, for example, the CDC Report from last year, showing that about 30% of adults did not have a landline in 2011, about twice the percentages as 2008.  This increase in wireless-only homes does not necessarily mean an increase in bias (more and more Republicans may be shedding their landlines, and thus the bias could fall even as wireless only usage increases).  Still, the departure in the polls indicates that a bias persists.


Friday, November 11, 2011

Born to Run?

About a year ago, I read a book called "Born to Run," by Christopher McDougall, who last week wrote an article in the New York Times Magazine on the same subject.

McDougall's basic premise is that we were faster and less injury-prone before we started wearing all these fancy running shoes and that they are what's causing running injuries. For example, in the New York times article:

"Back in the ’60s, Americans 'ran way more and way faster in the thinnest little shoes, and we never got hurt,' Amby Burfoot, a longtime Runner’s World editor and former Boston Marathon champion, said during a talk before the Lehigh Valley Half-Marathon I attended last year. 'I never even remember talking about injuries back then,' Burfoot said. 'So you’ve got to wonder what’s changed.'"

Statistics frowns on such anecdotal evidence, though it does make a good story. Did we really run faster? There are a lot of facts that we can look at though average times aren't among them. Marathon records (shown in Wikipedia) for men have indeed only downticked a little since the sixties. In 1970, Ron Hill of the UK (close enough, runnig-shoe wise, to be considered american?) set a record of 2:09:29. This year, a new record of 2:03:38 was set (the most recent US record was 2:05:38 in 2002). Six minutes in 40 years doesnt seem like much, but is it because of the shoes or because the sport has matured? And are Americans seen less because running isnt really a big competitive sport here?

When you look at women's times, the changes are much more dramatic. Women more recently began running marathons and fewer participated in the sport in general until relatively recently. In 1970, the women's marathon record was 3:02:53 (set by an american). In 2003, Paula Radcliffe (England) ran it in 2:15:25. That's a 47 minute improvement, or nearly 2 minutes per mile. In the 2011 New York Marathon, 40 women from the US bested the 1970 record time (see marathon site here for results).

So, I can't agree that we ran "way faster" 40 years ago. This doesn't mean that bare foot runners are slower than shoed runners because changes over the last fourty years in the level of competition, and improvements in training and fitness, rather than shoes, might have been the factors contributing to improved times.

How about injury rates? Do people get more injuries with running shoes than without? Unfortunately, any data on injury rates is tainted by the changes in the makeup of the population that runs (from a small, highly fit population to a large more population more varied in fitness--think of the then-overweight President Clinton running with a stop at McDonalds post-jog), and there haven't been any studies that directly compare injuries over time for barefoot running against running shoe running. A good summary article is here.

A recent article in Nature, while not looking at historical data, supports McDougall's contention that running shoes can be more harmful than bare feet when running. The article is lead-authored by Daniel Lieberman, a big advocate of barefoot running, so his bias may have been to look at things he believed were helpful about barefoot running and not at aspects of barefoot running that may be harmful. The article looks at impact forces and not at injuries, and doesn't consider that runners with shoes may be able to change their stride to reduce the impact forces (McDougall says this is hard to do with running shoes, and, from my own experience, I tend to agree, though I don't think it is impossible).

The statistical net-net is that there is no direct evidence either way right now. I admit some bias but I would say that the lack of evidence, given the power and money behind the shoe industry, tends to make me believe that, at best, fancy shoes are no better than bare feet, because if there were an effect in favor of shoes, I would certainly think we'd have seen a study by now (this is something correctly pointed out by McDougall and other advocates of barefoot running). Therefore, don't be surprised if you see me running with feet au-naturel someday soon.