Twitter at Night

I found a dataset of Twitter user locations and wanted to see what it looked like as a map. In particular, I wanted to make it look something like an “Earth at night” image and compare global light pollution with global Twitter activity. The idea is to separate industrialization from “informatization”. If a luminous place lacks proportional web activity, the characteristics of that city or country could tell us something about the nature of the digital divide, including non-technical aspects like politics.

A dataset like this could also be a useful addition to Digital Earth, as a location's web activity would be an interesting and increasingly important characteristic.

The dataset, found on infochimps.org, contains 3.8 million locations for tens of millions of Twitter users. It was gathered by mining data from user profiles between 2006 and 2010. For the uninitiated, Twitter profiles have an open text field where users can enter their location. This means it may contain any number of things, including country names, street addresses, coordinates, or “woudnt u like 2 no ;D”. While addresses and text like “Nap Town All Day Babyyyy” could have been geocoded, for this project I decided to only keep coordinates. Furthermore, I only kept coordinates that appeared to have been automatically entered by third-party applications or location-aware devices. These were prefixed by something like “iPhone:”. To clean the data, I used regular expressions in Notepad++, then did some manual cleansing in Excel.

When all was said and done, the dataset was significantly reduced, but still contained over 700,000 unique locations. QGIS and Photoshop helped produce an interesting visualization:

Twitter at Night: The globe illuminated with Twitter user locations
10000 x 3900 1.2MB jpeg*

You can see some imperfections in the data on the political map, e.g., points in the water, but we get the idea:

716,412 points plotted from coordinates taken
from the location field of Twitter user profiles
15000 x 6630 1.3MB gif*

Here is an “Earth at Night” composite image from NASA and the Defense Meteorological Satellite Program with the Twitter data overlaid. Not surprisingly, places like North Korea have few lights and also few Twitter points. But Eastern Europe, Russia, China, North Africa, and India are fairly well-lit with very sparse Twitter coverage. And take a look at Cuba – there isn’t a single point.

Twitter user locations plotted over a map of the world's lights
10000 x 3900 2.8MB jpeg*

Some things to keep in mind:

• These are mostly locations of mobile users, and the data are not necessarily a representative sample of Twitter users (or Internet users).
• In some countries, other microblogging services may be popular
• In mainland China, Twitter has been blocked since 2009. The lack of Twitter activity in China isn’t a good indicator of Internet access, but it can say something about a different kind of digital divide, maybe an information divide.

Here is a higher quality PNG of the Twitter at Night image (3.9 MB, 10000 x 3900).

*Note that all of these images have really high resolutions. If your browser has trouble displaying them, try right-clicking on the links and saving to your computer.

Putting North Korea on the map

If you've happened to pan over northeastern Asia in Google Maps, you may have noticed something odd. The roads, water bodies, markers, and labels that cover the extent of South Korea, Japan, Russia, and China are curiously absent over North Korea.

North Korea blank on Google Maps

In searching for an explanation I've found a lot of speculation, mostly about North Korea's authoritarian government, the country being cut-off from the West, the poverty, and so on. But these factors shouldn't necessarily prevent Google from mapping features in the DPRK, especially since high-resolution satellite imagery is available on Google Maps. After all, Bing has labeled the most basic features. Instead, the [partial?] answer seems to be that Google simply hasn't published the data. From a topic on the Google Maps Help Forum: 

"We never launched coverage in some countries because we simply weren't satisfied with the map data we had available. We're constantly searching for the best map data we can find, and sometimes will delay launching coverage in a country if we think we can get more comprehensive data." 

There's some good news, though, for those who want to browse detailed maps of North Korea. Google Map Maker has opened up the DPRK for contribution (along with 187 other countries as of this writing) and it looks like there is already a lot of detail.

In contrast to Google Maps, North Korea has more features in Map Maker

Features added to Google Map Maker in Pyongyang

Content from Map Maker may eventually lead to a satisfactory map of North Korea and publication on Google Maps. Until then, Map Maker, satellite view on Google Maps, and Panoramio photos on Google Earth are pretty good resources. Also have a look at the massive amount of data available from the North Korea Uncovered project. There you can download a KML file with layers for infrastructure, palaces, artillery, and many other interesting sites.There was a WSJ article about the project.

Together, all of these resources should help take some of the mystery out of the blank spot on the map.