Categories
Data

Visualizing Summer Travels Part 6: Projecting Spatial Data with Python

This post is part of a series on visualizing data from my summer travels.

I’ve previously discussed visualizing the GPS location data from my summer travels with CartoDB, Leaflet, and Mapbox + Tilemill. I also visualized different aspects of this data set in Python, using the matplotlib plotting library. However, these spatial scatter plots used unprojected lat-long data which looked pretty distorted at European latitudes.

Today I will show how to convert this data into a projected coordinate reference system and plot it again using matplotlib. These projected maps will provide a much more accurate spatial representation of my spatial data and the geographic region. All of my code is available in this GitHub repo, particularly this notebook.

Categories
Data

Using geopandas on Windows

projected-shapefile-gps-coordinatesThis guide was written in 2014 and updated slightly in November 2020.

I recently went through the exercise of installing geopandas on Windows. Having learned several valuable lessons, I thought I’d share them with the world in case anyone else is trying to get this toolkit working in a Windows environment. It seems that pip installing geopandas usually works fine on Linux and Mac. However, several of its dependencies have C extensions that can cause compilation failures with pip on Windows. This guide gets around that issue.

Categories
Data

Visualizing Summer Travels Part 5: Python + Matplotlib

This post is part of a series on visualizing data from my summer travels.

I’ve previously discussed visualizing the GPS location data from my summer travels with CartoDB, Leaflet, and Mapbox + Tilemill. Today I will explore visualizing this data set in Python, using the matplotlib plotting library. All of my code is available in this GitHub repo, particularly this notebook.

Categories
Data

Visualizing Summer Travels Part 4: Mapbox + Tilemill

This post is part of a series on visualizing data from my summer travels.

I’ve previously discussed my goals in visualizing GPS data from my summer travels and explored visualizing the data set with CartoDB and with Leaflet. The full OpenPaths location data from my summer travels is available here and I discussed how I reverse-geocoded it here.

Mapbox is a major provider of online web mapping services such as tiled web maps, the Tilemill cartography IDE, and the mapbox.js javascript library. Today I’ll run through how to create an interactive data map in Tilemill’s design studio, export the map as a set of tiles, upload the tileset to Mapbox, and then use a javascript client to display the map on a web page. Our final result will look something like this:

Categories
Data

Visualizing Summer Travels Part 3: Leaflet

This post is part of a series on visualizing data from my summer travels.

I’ve previously discussed my goals in visualizing GPS data from my summer travels and explored visualizing the data set with CartoDB. The full OpenPaths location data from my summer travels is available here and I discussed how I reverse-geocoded it here.

Lastly, I reduced the size of this spatial data set so Leaflet can render it more quickly on low-power mobile devices. I discussed why this is important and how to do it with the DBSCAN clustering algorithm and also with the Douglas-Peucker algorithm. The final data set I’ll be working with is available here.

Categories
Data

Reducing Spatial Data Set Size with Douglas-Peucker

In a previous post I discussed how to reduce the size of a spatial data set by clustering. Too many data points in a visualization can overwhelm the user and bog down on-the-fly client-side map rendering (for example, with a javascript tool like Leaflet). So, I used the DBSCAN clustering algorithm to reduce my data set from 1,759 rows to 158 spatially-representative points. This series of posts discusses this data set in depth.

Categories
Data

Clustering to Reduce Spatial Data Set Size

Read/cite the paper here.

In this tutorial, I demonstrate how to reduce the size of a spatial data set of GPS latitude-longitude coordinates using Python and its scikit-learn implementation of the DBSCAN clustering algorithm. All my code is in this IPython notebook in this GitHub repo, where you can also find the data.

Traditionally it’s been a problem that researchers did not have enough spatial data to answer useful questions or build compelling visualizations. Today, however, the problem is often that we have too much data. Too many scattered points on a map can overwhelm a viewer looking for a simple narrative. Furthermore, rendering a JavaScript web map (like Leaflet) with millions of data points on a mobile device can swamp the processor and be unresponsive.

Categories
Data

Reverse Geocode a Set of Lat-Long Coordinates to City + Country

This tutorial demonstrates how to reverse geocode a set of latitude-longitude coordinates to city and country using Python and the Google Maps API.

I have previously written about my GPS location data from this summer’s travels. The data set, gathered with the OpenPaths app, contains lat-long coordinates and timestamps. Without city or country data, any visualizations would be very simplistic because all I have is coordinates and timestamps. It would be nice to reverse geocode these coordinates to add city and country data to each point. Then, I could create richer, more informative marker popups that include this new geographical information.

Categories
Data

Visualizing Summer Travels Part 2: CartoDB

This post is part of a series on visualizing data from my summer travels.

I recently discussed OpenPaths and my goals in visualizing location data from my summer travels. In this post, I’ll explore visualizing this dataset with CartoDB. The OpenPaths data from my summer travels, which I’ll be working with in these examples, is available here and I discuss how I reverse-geocoded it here. CartoDB is a simple cloud-based tool for building web maps. You can import data through their web-based dashboard and quickly turn it into a dynamic map or visualization.

Categories
Tech

Pattern Unlock an Encrypted Android Phone

We’re all familiar with the possibility of data security breaches. Web sites get hacked, passwords get compromised, laptops get stolen. To mitigate these risks, we (try to) use strong passwords, keep our computers under lock and key, and encrypt our personal data. But what about our phones? They are increasingly relied on as mini-computers in our pockets, replete with email accounts, banking apps, and sensitive Dropbox files. Many apps store usernames and passwords in plain text.

What happens if your phone gets stolen? Many people don’t have any security or lock screen enabled at all. Others simply use a pattern or short PIN that is easily cracked in minutes. Android offers encryption, but it’s turned off by default. It’s also very inconvenient. To be effective, encryption requires a strong password, and Android (4.x) requires that you enter this password to unlock your phone when it boots-up, and also every time you unlock the screen.

The problem is that once you encrypt your phone, Android (again, versions 4.x – maybe this will change in a future release!) disables the ability to lock/unlock it with a pattern (annoyingly) or with a different, shorter PIN (perhaps understandably). Having to type in a long password every time you want to use your phone makes this is a non-starter for most users.

Ideally, we would enter a strong password to unlock and decrypt the phone at boot-up, and then use a simpler, user-friendly security mechanism (such as a pattern) to unlock the phone throughout the day. This would balance the benefits of strong-password encryption with the practicalities of making the phone accessible throughout the day.