Mining social data: craigslist w4m
This was an experiment I thought of and hacked together yesterday. I was curious about whether cultural differences between cities would show up if I did a Bayesian analysis of words in personal ads across different cities.
So here’s what I did:
- Downloaded 500 ads each from the w4m section in craigslist in Boston, New York, San Francisco, Los Angeles and Chicago.
- Split each ad into individual words using a basic \W regex split
- Trained a Bayesian filter using the words as features, and the city as the result
- Tested the filter against a training set to ensure that it made better predictions than random
- Had the filter calculate the words that best discriminate which city an ad came from
- Removed all the words that were geographically specific, e.g. manhattan, cambridge, mission
The list is shown below, with words in descending order of predictive ability. There are a lot of random words, perhaps some could have been eliminated with a larger dataset, but I noticed some pretty clear themes emerging:
- Not surprisingly, given the lovely climate compared to the other cities on the list (I just moved here, I get to rave, ok?), San Francisco ads had a lot of “outdoor” words, such as picnic, hikes, and cycling
- The New York ads were much more likely to contain words indicating sexual tastes
- People in LA like bright adjectives like excellent, fantastic and lame. They also refer to the “industry” a lot (the movie industry)
- In New York, Boston and Chicago, local sports teams appeared very high. In LA and SF they didn’t appear at all
- I was a bit confused by the appearance of the word pink in Boston. As it turned out it often preceeded Floyd or red sox hat, a pink red sox hat being something that Boston women like to make fun of.
- In SF, people use the word tee as an abbreviation for T-Shirt. I had never heard of this, and no one in any other city uses it ever.
(of course, this is not representative of the whole city, just people who post personal ads on craigslist in the last few days. A very self-selecting group, of course, and one that would exclude anyone who was at Burning Man)
Anyway, here’s the list so you can make your own silly generalizations: