Categories
Planning

Craigslist and U.S. Rental Housing Markets

This is a summary of our JPER journal article (available here) about Craigslist rental listings’ insights into U.S. housing markets.

Small map of 1.5 million Craigslist rental listings in the contiguous U.S., divided into quintiles by each listing's rent per square footRentals make up a significant portion of the U.S. housing market, but much of this market activity is poorly understood due to its informal characteristics and historically minimal data trail. The UC Berkeley Urban Analytics Lab collected, validated, and analyzed 11 million Craigslist rental listings to discover fine-grained patterns across metropolitan housing markets in the United States. I’ll summarize our findings below and explain the methodology at the bottom.

But first, 4 key takeaways:

  1. There are incredibly few rental units below fair market rent in the hottest housing markets. Some metro areas like New York and Boston have only single-digit percentages of Craigslist rental listings below fair market rent. That’s really low.
  2. This problem doesn’t exclusively affect the poor: the share of its income that the typical household would spend on the typical rent in cities like New York and San Francisco exceeds the threshold for “rent burden.”
  3. Rents are more “compressed” in soft markets. For example, in Detroit, most of the listed units are concentrated within a very narrow band of rent/ft² values, but in San Francisco rents are much more dispersed. Housing vouchers may end up working very differently in high-cost vs low-cost areas.
  4. Craigslist listings correspond reasonably well with Dept of Housing and Urban Development (HUD) estimates, but provide up-to-date data including unit characteristics, from neighborhood to national scales. For example, we can see how rents are changing, neighborhood by neighborhood, in San Francisco in a given month.

Housing trends on Craigslist

I’ll run through the findings and explain the methodology at the end. The data set contains 11 million Craigslist rental listings from summer 2014. Nationwide, the median rent is $1,145, the median square footage is 982, the median rent/ft² is $1.11, and the average number of bedrooms is 2. This map depicts 1.5 million geolocated Craigslist rental listings in the contiguous U.S.:

Map of 1.5 million Craigslist rental listings in the contiguous U.S., divided into quintiles by each listing's rent per square foot
Map of 1.5 million Craigslist rental listings in the contiguous US, divided into rent/ft² quintiles, summer 2014

Craigslist conforms to our geographic expectations of the U.S. housing market: large cities on both coasts have high rents, particularly through the Boston-Washington corridor and along the coast of California. Other hot-spots exist in south Florida and the metro areas of large, affluent cities like Chicago, Denver, and Seattle.

The interior of the U.S. has a sprinkling of less-expensive listings punctuated by mid-priced clusters around major cities and regional centers. Interestingly, rental listings in North Dakota generally have extremely high rents/ft², reflecting unmet demand for housing in oil-producing areas. In fact, North Dakota has the 2nd-highest median rent/ft² of any Craigslist region:

The most expensive Craigslist regions by median rent per square foot: New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta
Median rent/ft² in the most expensive Craigslist regions. Horizontal line depicts the nationwide median.

By region I just mean Craigslist subdomain, like “Boston” in http://boston.craigslist.org. Overall, New York, North Dakota, San Francisco, Boston, and Santa Barbara are the most expensive Craigslist regions. The other usual suspects from Southern California, Hawaii, and the Eastern Seaboard also pepper this list. But among the 15 most populous metro areas, the large cities elsewhere in the Sunbelt (plus Detroit) are much cheaper:

The most most populous Craigslist cities, by median rent per square foot: New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta
Median rent/ft² in the most populous metro areas on Craigslist. Horizontal line depicts the nationwide median.

New York’s median rent/ft² is more than 3.5x higher than Atlanta’s, reflecting underlying differences in land values which capitalize variation in amenities, incomes, supply, and demand. Median rent figures for all cities appear in the article so I won’t belabor them here, but suffice it to say that the median rental unit in metro areas like New York, Boston, and the Bay Area costs well over $2,300/mo. That’s across the whole metro area and its exurbs – not just the typical pricey urban core.

Rents also vary widely within cities. To examine this, we estimated the distribution of rents/ft² for each of the 415 Craigslist regions in the U.S.:

Distribution of rents per square foot in each of the 415 Craigslist U.S. regions, with the 15 most populous broken out for detail: New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta
Distribution of rents for all 415 US Craigslist regions (left), with the 15 most populous metros broken out for detail (right). Each has its own line, colored by its median rent/ft². The area under the curve between any two points represents the probability for that interval.

I know that this rainbow spaghetti is a bit much, but bear with me for a moment. The color gradient just shows that metros with lower median rents/ft² tend to peak at lower values and tend to be more peaked. This “compression” of rents in soft markets is significant. In Detroit, most of the listed units are concentrated within a very narrow band of rent/ft² values, but in San Francisco rents are much, much more dispersed.

Metropolitan fair market rent based housing vouchers – designed to unlock neighborhoods of opportunity to the poor – may work differently in high-cost versus low-cost areas. Wide variations in rents within a city might make it very difficult for voucher holders to “move the needle” and upgrade neighborhoods in a statistically dispersed market like San Francisco’s, especially when considering household size.

Housing affordability on Craigslist

The article looks at three simple but telling indicators of affordability: the proportion of income spent on rent, the proportion of listings below fair market rent, and a “rental power” indicator. I’ll summarize each briefly:

a.) Rent burden

The standard definition of “rent burden” is rent exceeding 30% of household income. This chart shows the share of its income that a typical (i.e., median) household would spend on a typical Craigslist rent, in each of the 15 most populous metro areas:

Rent burden: ratio of median Craigslist rent to median monthly household income for the 15 most populous metropolitan areas in the U.S. New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta.
Ratio of median rent to median monthly household income for the 15 most populous metro areas in the US. Horizontal line depicts the standard 30% definition of rent burden.

At their median rent and income values, New York, Los Angeles, San Francisco, Miami, Boston, and San Diego all exceed the rent burden threshold. In other words, the typical household renting the typical apartment on Craigslist would experience (what’s supposed to be) atypical rent burden. Details for all the other metro areas are available in the journal article.

b.) Fair market rent

HUD generally defines fair market rent as the region’s 40th percentile rent. 37% of the Craigslist listings are below the corresponding HUD fair market rent – quite close to the expected value of 40%. However there is considerable variation. More than two-thirds of the listings in Phoenix and Las Vegas are below the fair market rent, but New York and Boston have only single-digit percentages of listings below this threshold:

Proportion of Craigslist listings at/below HUD FMR, per Craigslist region: New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta
Proportion of Craigslist listings at/below HUD fair market rent. Horizontal line depicts the standard 40th percentile fair market rent.

This is troubling for housing advocates. As discussed earlier, fair market rents might be insufficient for households trying to upgrade neighborhoods in metros with highly dispersed rents; they also appear to limit housing seekers in New York and Boston (among others) to very narrow slices of available housing units.

Furthermore, the HUD fair market rent for a 2-bedroom unit in Alameda County (in the San Francisco Bay Area) dropped from $1,585 in 2015 to $1,580 in 2016 (despite the region’s skyrocketing rents) because HUD extrapolated it from the 2013 5-year rolling-average ACS estimate. Craigslist data offer a real-time alternative to easily take the pulse of local housing rental markets at fine scales to inform current estimates.

c.) Rental power

Finally, the “rental power” indicator represents an estimate of how many square feet someone can rent on Craigslist in each metro area for the nationwide median rent of $1,145. It simply divides the nationwide median rent by each regional median rent/ft²:

Rental power indicator: how many square feet one can rent on Craigslist in each of the 15 most populous metropolitan areas for the nationwide median rent of $1,145, given each's median rent per square foot: New York, Boston, Miami, San Francisco Bay Area, Los Angeles, Chicago, Philadelphia, Seattle, Washington D.C., Dallas, Houston, Detroit, Phoenix, Atlanta
Rental power: how many square feet one can rent on Craigslist in the 15 most populous metro areas, for the nationwide median rent. Horizontal line depicts the nationwide median square footage.

Memphis, TN offers the greatest value among large metros at 1,659 square feet, while New York offers the least value at 398 square feet. Details for all the other metro areas are in the journal article.

Metropolitan patterns

Urban planners need current data at the local scale. We calculated census tract-level median rents/ft² from 100,000 San Francisco Bay Area listings and mapped the results. Such visualizations enable local planners to quickly take the pulse of changing neighborhoods:

Census tract-level map of median rents per square foot on Craigslist in the San Francisco Bay Area
Census tract-level median rents/ft² in the Bay Area

San Francisco, Berkeley, and Silicon Valley are very expensive, while East Oakland and Vallejo are less so. This is a powerful tool for understanding housing characteristics, trends, and affordability: we can see how rents are changing, neighborhood by neighborhood, in San Francisco in a given month.

In southern California, the greater Los Angeles area similarly shows a gradient in rents from expensive coastal areas towards cheaper inland areas. Although some inland urban areas like San Bernardino and Palm Springs have moderate rents, others like Victorville are heavily represented by the lowest quintiles:

Detail of Craigslist rental listings, divided into nationwide quintiles by rent per square foot in the greater Los Angeles area (top) and the Midwest between Chicago and Detroit (bottom)
Craigslist rental listings in the greater Los Angeles area (top) and the Midwest between Chicago and Detroit (bottom)

The Craigslist data reveal different spatial patterns elsewhere in the country. In the Midwest, Chicago has a high-priced urban core with mid-priced suburbs, but Detroit inverts this metropolitan model with a low-priced core and more expensive exurbs. This is consistent with Detroit’s history of capital flight and white flight over the past 50 years. Likewise, small “rust belt” cities in Indiana, Ohio, and Michigan are dominated by listings in the lowest quintiles.

Usage trends

On Craigslist, the most rental listing activity takes place on Mondays and Tuesdays, while Sundays have the fewest listings posted. From Monday onward, the number of daily listings posted declines, before repeating all over again the next week. Median rents/ft² are about 11.5% higher on Sundays (the most expensive day) than they are on Mondays (the least expensive day). Housing advocates might target housing searches to take advantage of rhythms in volume, cost, and unit characteristics.

Methodology and validation

This project was conducted in Paul Waddell’s Urban Analytics Lab, which collected 11 million rental listings from Craigslist across the U.S. between May and July 2014 using a web scraper built with Python. For a full discussion of the web scraping methodology and copyright law, check out the journal article. This study cleaned the data, extracted useful elements, organized them, and analyzed them, all with Python and its pandas, numpy, and matplotlib libraries.

The study validated the listings data against HUD’s median rents for this time period by calculating correlation coefficients to assess co-variation for a sample of metros. The correlations between the Craigslist and HUD median rents are positive, strong, and statistically significant:

Scatterplot of Craigslist median rents versus HUD median rents, for each metropolitan area. The simple regression lines represent the relationship between the two datasets disambiguated by number of bedrooms.

On average, median rents are 7% higher for 1-bedroom, 3% higher for 2-bedroom, 7% lower for 3 bedroom, and 1% higher for 4-bedroom units than the corresponding HUD median rents. However, the bias varies between regions: New Orleans and Oklahoma City have very similar median rents across the two data sets, while other regions like Las Vegas have considerably lower median rents in the Craigslist data set, while yet other regions like New York have considerably higher median rents in the Craigslist data set.

Conclusion

Long story short: there are incredibly few rental units below fair market rent in the hottest housing markets. This (along with the compression of rents) tells us there might be a big disconnect between housing vouchers and the largest rental housing information exchange (Craigslist). This problem doesn’t exclusively affect the poor: the median household in cities like New York and San Francisco would experience rent burden when paying the median rent.

These national, regional, and local patterns conform to general expectations, yet offer far fresher and finer-grained data. However, Craigslist listings provide advertised rents, not final negotiated rents in legal contracts, and some rental markets like New York’s are dominated by brokers. Craigslist is not representative of the entire housing market: individuals vary in levels of Internet access and technical savvy to list and search for housing online as a function of wealth, employment, education, language, and other socio-demographic traits. Nevertheless, Craigslist presents an invaluable data source for housing research.

Download the journal article

Citation info: Boeing, G. and P. Waddell. 2016. “New Insights into Rental Housing Markets across the United States: Web Scraping and Analyzing Craigslist Rental Listings.” Journal of Planning Education and Research. Online first. DOI: 10.1177/0739456X16664789

27 replies on “Craigslist and U.S. Rental Housing Markets”

Interesting analysis.

Are you aware of any studies that explores the relationship between price and the availability of “social housing” (that which is controlled to subsided by the government or non-profits)?

For example: “Denmark has a total housing stock of 2.5 million housing units. 19 percent of housing units belong to social housing associations, and about 1 percent belongs to public authorities.”
http://urbanlifecopenhagen.weebly.com/housing.html

If the goal of social housing is to make it affordable – does it work?

Along those same lines…I was reading about how Sweden tries to support “cohousing” for seniors. Apparently, a group of seniors can approach the government for financing of a property to set up cohousing. In addition, the Swedish government is involved with subsidizing cohousing in additional ways which aren’t clear to me. The Swedish government doesn’t believe that “market” considerations should determine whether people are able to get housing since they might be forced or influenced to move based on market prices. They want to protect their society from such malicious influences.

Another vote for making it public! Or, alternatively, would you consider working with a local civic tech org to walk us through your methodology so that we could reproduce your work at a local level? Also, did you get any pushback from Craigslist? We have a Craigslist scraper, but weren’t sure if we would face legal risk publishing data for it, even as a nonprofit.

A more interesting association might be housing/jobs. The markets/locales experiencing the booms are also experiencing economic growth. Is the housing crunch in those areas experiencing an associated housing boom in proportion to the economic growth?

NYC real estate broker here. There are a few trends that – at least according to my clients – are unique to the New York rental market and Craigslist specifically, and that I thought could be of interest, if not necessarily applicable to the datasets here.

As a note, I’ve been working as a real estate agent for the past four years – the first 1.5 years primarily in the Brooklyn and queens before switching over to Manhattan. The following points are based on my experience and apply to Manhattan and prime neighborhoods in the outer borough rental markets.

As you mentioned, a significant portion of Craigslist posters in NYC are brokers. As brokers have to pay to advertise listings ($10 per ad, $8 if ad space is purchased in bulk .. I believe my brokerage purchases bundles of 500 ads). Compared to other listing services that cater to NYC rentals this is fairly steep price per ad and so brokers tend to limit their Craigslist posts to the very best deals, listings that are sure to rent quickly.

In other cases though, because there is virtually no oversight to enforce accuracy on Craigslist, bait and switch flourishes when brokers attempt to amass a lot of renter inquiries in a short time. Listed prices are often off by hundreds of dollars and its not uncommon to find fabricated listings with photos that can be traced back to buildings in other cities entirely by simply reverse image searching.

Perhaps most sneaky thing that could skew craigslist data is the ubiquitous ‘flex’. I’m not sure if this exists in other major cities, but its particularly popular in the summer market here (lots of students / post grads moving who are willing to sacrifice space for location). Flexing, or converting an apartment to create additional bedrooms is extremely commonplace. While the rules and regulations put out by the city attempt to address this, fire safety being the main concern, many landlords and management firms will look the other way when tenants put up walls. Frequently tenants will sign leases with a no-wall policy after having seen flexed apartments in the building. While there are companies that provide legal room dividers (not quite walls but that comply with fire codes by leaving a 12-inch space between the divider and the ceiling) private handymen are often employed to build a full floor to ceiling wall for around the same price ($1000-1200/wall). How this relates to Craigslist? Many buildings have large two bedrooms that can convert to four bedrooms (for instance a large living room and dining alcove can be turned into two separate bedrooms with a remaining living room). Sometimes one bedroom units can flex into three bedroom units. If a listing like this is going to be posted on Craigslist by brokers they will almost always be posted with the higher bedroom count (4BR/2BA, not 2BR/2BA).

I’ve got a lot to say about how all of this relates to affordable housing but too much to get into in a comment thread. In short, the process of finding a rental that accepts a rent voucher is a special kind of hell. I think about this a lot when my particularly out of touch clients do things like test pressure of every faucet or analyze the exact direction of sunlight in each room of an apartment.

Streeteasy is the most accurate website for current NYC rentals anymore. They strive for accuracy and really do their best to keep ads on the site legitimate. I wonder how different the markets would look between site sites.

[…] Craigslist and U.S. Rental Housing Markets The UC Berkeley Urban Analytics Lab collected, validated, and analyzed 11 million Craigslist rental listings to discover fine-grained patterns across metropolitan housing markets in the United States. I’ll summarize our findings below and explain the methodology at the bottom… […]

Awesome to see this wide data set being mined.

Regarding the median income vs median rent, though, this seems like an over-simplification. I’m not sure how the direct comparison of these numbers can tell you much within a given market.

I’m assuming the income distribution shows significant skew, as does the rent distribution. Would it not make more sense to look at this data by quartile or decile?

Also, were you able to extract other salient details from the listings with any consistency? E.g. Number of rooms, square footage, proximity to city centers, other amenities. I would think that would be amazing fodder for further segmenting the available inventory. For example, it’s well known NYC has a glut of luxury apartments right now. As a result, the average rent on luxury apartments has been decreasing lately–though that’s hardly good or useful news for the typical renter.

Yes, I think segmenting the market would be an excellent way to examine this dataset. Comparing medians is just one reference point: the typical household and the typical rental listing. Also – yes the number of bedrooms, square footage, and geolocation (when available) were collected. See the paper for full details.

I’ve been pulling some stuff for New York State’s affordable housing research…I’m not seeing longitude/latitude for listings. Are you taking addresses and geocoding them?

I think they moved that data to the individual listing’s page. It’s not displayed, but buried in the HTML they have an element with attributes like data-latitude=”40.745900″ data-longitude=”-74.002061″, corresponding to the coordinates of the pin on the map. Just gotta parse it out.

Great article regarding rental prices on craigslist VS HUD stats. With a site that large, the stats they can pull out very accurately reflect those of the region. Beyond rentals, other markets can use CL data the same way.

[…] a dataset of Craigslist postings collected by the Urban Analytics Lab in 2014 and were provided by Geoff Boeing. We combined these listings with commute data, using a simulated dataset of commute destinations […]

[…] Craiglist and the US housing market.  Geoff Boeing of the University of California, Berkeley has assembled data from Craigslist to measure rental housing markets across the US. Craigslist has become the de facto listing service for rental housing. Boeing and his colleagues have scraped rental listings data from their website to develop price indices for major US markets.  Their work is summarized in this heatmap showing relative rent levels: […]

Leave a Reply to gboeingCancel reply