How FOSS4G had a profound impact on my geospatial technology path

As some of you will know, the FOSS4G 2011 conference is coming up in Denver and I am the conference chair. I have only been to one previous FOSS4G, which was in Victoria in Canada in 2007. That event had a profound impact on my perspective on the geospatial industry, and on the software platforms I've chosen to implement geospatial applications on since then. And it has saved my companies a lot of money! So I wanted to share some perspectives about my experience at FOSS4G and what I've learned about open source software, to explain why you should come to FOSS4G, especially if you've never been before.

Back in 2007 I had just left my job as CTO of Intergraph and was weighing up what to do next, and looking at ideas for a geospatial startup company. I'd spent 20 years in the geospatial industry working with closed source products, and knew very little about open source geospatial products. Towards the end of my time at Intergraph, I'd been getting quite a few questions from sales guys, in Canada in particular, about the fact that their customers were showing interest in open source software, which was free, and how should they sell against that?  So I'd done a bit of research and had become interested in particular in PostGIS, the open source spatial database, as a possible platform for applications I was looking at in my new (yet to be created) startup, Spatial Networking. The fact that it was free was obviously attractive to a new startup owner, especially as I was looking at a system that (I hoped) would need to be deployed on many servers to cope with large numbers of users.

Open source software tends not to have such flashy marketing material as closed source software, so after a bit of digging around online and not finding all the information I was after, I got in touch with Paul Ramsey, who is one of the main people behind PostGIS. He suggested I should come up to FOSS4G to find out more. I did and was really impressed by the whole experience, both the event in general and what I found out about PostGIS - you can read my writeup at the time here. As I said in that writeup, there was much more energy and buzz than I had seen at other geospatial conferences I was used to attending.

I went ahead and used PostGIS at Spatial Networking and on other projects, including my current project Ubisense myWorld. I continue to be very impressed with PostGIS - it does all the core things you expect a spatial database to do (I had a lot of previous experience with Oracle Spatial and other systems), and it's FREE! I have never hit a bug during my time using it.

So what have I learned about open source geospatial software in those four years? First of all let me say that I have no strong predisposition to open or closed source development approaches, per se. I am happy to chose either open or closed source products depending on what I need in a given situation. What I do have s strong predisposition to is FREE. Obviously a product needs to meet your requirements, but assuming it does then free is rather attractive compared to having to pay for something. This is especially true in a cloud environment, where you may scale up to running many servers, and traditional per server licensing costs can really hurt you financially.

People in the closed source world often raise concerns about support in the open source world. My experience hasn't borne out this concern. With PostGIS, I've never needed support, it just works. With MapFish, another open source product we're using for myWorld, we needed a few enhancements. Some were addressed by the community within a month or two, others we were able to do ourselves as we had access to the product source code. I very much doubt that we could have got enhancements made in a mature closed source product in that timeframe. There are also more and more options in the open source geospatial world to pay people to do enhancements or fixes for you. Again it's a bit dangerous to make generalizations, you can get good support or poor support on different closed source products, and you can get good support or poor support on different open source products. But my experience with the open source products we've chosen has been very good.

I think that after price, perhaps the aspect of open source that I value most is longevity and predictability. Many times during my career, I have seen projects suffer because a vendor has decided to stop development of a product (or feature). I have also seen dramatic changes in terms of service or costs of online services. Google App Engine is one example of the latter - many people put significant effort into developing applications that were running for free, then Google changed the pricing model and people found themselves facing large costs they hadn't planned on. With an open source product that has a strong community behind it, there is much more long term stability. You know it's not going to go away tomorrow. Even if some developers leave, others are there to cover for them. And in the worst case you have access to the source code so could continue to maintain it yourself (though that's a very unlikely scenario as long as the community of developers has a certain critical mass). I have been moving more components of myWorld towards open source because of this predictability.

So anyway, if you are still paying for geospatial software you owe it to yourself and your company to come to FOSS4G and find out what all this open source software is about. There is a parallel universe out there with software products that have great capabilities and are FREE! It's also worth saying that in general there is good interoperability between open and closed source systems, so it's not an all or nothing proposition. In general open source web and database products are very strong, and they may well be able to complement your investment in existing applications.

This is the first time that FOSS4G has been in North America for 4 years, and it is not likely to be here again for another 3, so this is a rare opportunity if you are based here to meet a wide range of people developing and using these products. So I hope to see you in Denver in September at FOSS4G!

To finish up, check out this video featuring members of the organizing team talking about why you should be there:

FOSS4G 2011 Denver from Tyler Mitchell @ OSGeo on Vimeo.

Speaking at OpenStreetMap Meetup in Denver tomorrow

As mentioned previously, I'll be speaking at tomorrow's OpenStreetMap meetup in Denver tomorrow, at the cool new MapQuest offices downtown. My attempt to crowdsource the presentation got a great response, and I got lots of suggestions for interesting content. I plan to have something for everyone, along the following lines:

• A quick intro to OpenStreetMap for any newcomers
• Some tips on using the Potlatch 2 map editor, which provides some cool new features - this should be good for both newcomers and experienced mappers
• Examples of how government agencies from various parts of the world are using OpenStreetMap
• Cool new tools for developers using OpenStreetMap, like Leaflet and Kothic
• Some interesting applications using OpenStreetMap
• Last but not least, how OpenStreetMap was used to show that you can in fact walk across Dublin without passing a pub (pubs have always been a mainstay of OpenStreetMap!)

We'll also have some discussion about the state of OpenStreetMap in Denver and Colorado, and brainstorm on things that we might want to focus on before State of the Map and FOSS4G are in Denver in September! You can get full details on the meetup and sign up here.

Looking for content for "State of OpenStreetMap" presentation

I'm doing a couple of upcoming presentations on OpenStreetMap, the first one next week at the very cool MapQuest office in downtown Denver, so I encourage you to come along to that if you're in the neighborhood ... and this may well evolve into a presentation for State of the Map in Denver too!

So in the best OpenStreetMap tradition I thought I'd try a little crowdsourcing to help me pull this together. I'd be interested if you could send me links (or other info) about things that you think are (reasonably) new and interesting in the OpenStreetMap world including:

• Cool applications using OpenStreetMap data
• New (or improved) tools for creating / editing OpenStreetMap data
• Examples of businesses or government organizations using OpenStreetMap
• Anything else you think is interesting!

Please just drop me an email, or comment below. In fact if anyone has presentation slides covering interesting stuff that would be relevant, those would be great too. Happy to give credit to all contributors, of course!

A couple of discussions on location privacy and the iPhone

So my recent posts analyzing the iPhone location data log have gained a lot of traffic and attention over the past few days, from places including the Toronto Star, the Wall Street Journal, TUAW, PC World, MacDailyNews, Apfeltalk (German), Cisco, Pete Warden at O'Reilly, Business Insider, and more.

This led to me being invited to participate in a discussion on the Brian Lehrer Show yesterday on radio WNYC, the NPR affiliate in New York, together with Jennifer Valentino-Devries of the Wall Street Journal. We had a good sensible discussion, in contrast with a lot of the hysterical reporting that has been going on.

Today I will be taking part in a longer discussion on the KQED Forum discussion show in San Francisco. Also participating will be Congresswoman Jackie Speier, Jim Dempsey from the Center for Technology and Democracy, and Kara Swisher of All Things Digital. This will be at 10am mountain time, there should be a recording here sometime after that.

Apple issues Q&A on "Locationgate", and addresses key issues

Apple rather belatedly issued a Q&A on the whole "LocationGate" saga. This confirms what I said about the data being a cache of cell tower and wifi locations. The fact that this was kept for up to a year was a bug. Within the next few weeks they will reduce this to 7 days, they will not back up the cache any longer, and they will turn off the cache when you turn location services off, which addresses the issue reported by the Wall Street Journal and widely re-reported.. These are all good actions to take, and address the key issues in my opinion. It does reinforce the importance of developers being careful about location security, and Apple was slack in this case, even though the potential risks were much less dire than widely reported.

Note that in the short term if you are concerned, you can encrypt your iPhone database backup just by checking a box on the front page in iTunes (after plugging in your iPhone). If you do this, the current location log cannot be accessed by someone who hacks into your computer.

The scoop: Apple's iPhone is NOT storing your accurate location, and NOT storing history

The Summary
So in my previous two posts I discussed how the data I was seeing in my iPhone location logs was actually not very accurate, and certainly didn't reveal where I lived or worked or had stayed on my travels - beyond showing the cities I had been to, including general areas I had visited, as well as some I hadn't. There had been some discussion that the data appeared to be, in a number of cases, the location of cell towers you had been in communication with, although in some cases locations were a long way from where you had been.

The quick summary: I believe I have confirmed that Apple is not storing your location, but the (actual or estimated) location of cell towers (and WiFi access points) that are close to you, to help locate you as you move (these are not necessarily towers that you have been in communication with). In the data I have examined there is nothing that is based on the accurate location of the iPhone. For a good example, see my previous post showing the location of cell equipment in Coors Field baseball stadium, and not revealing the location of my home which is very close to there. In my opinion, if Apple was storing this data in order to know where you had been, they would be storing different, more accurate location data that they have access to.

And, importantly, they are not storing history - the only thing that can be found from the files is when you last visited a general area, not if you made repeat visits. This is especially important as it means that many of the concerns expressed about this data are simply not valid: it cannot be used to determine where you live, or work, or go to school, or who your doctor is.

Here is a report of what Al Franken said:

Sen. Al Franken, a Minnesota Democrat, said it raises “serious privacy concerns,” especially for children using the devices, because “anyone who gains access to this single file could likely determine the location of a user’s home, the businesses he frequents, the doctors he visits, the schools his children attend and the trips he has taken — over the past months or even a year.”

The only part of this that is correct is that the data will show what cities you've visited, with some indication of which parts of a city you may have visited, though nothing definite - there will be records in areas you didn't visit. And it doesn't show repeated visits to the same location, only the last one.

Update: see below for a very interesting comment from "Anonymous", who includes a link to a document submitted by Apple to Congress in July 2010. This includes the following:

"When a customer requests current location information ... Apple will retrieve known locations for nearby cell towers and Wi-Fi access points from its proprietary database and transmit the data back to the device" ... "The device uses the information, along with GPS coordinates (if available), to determine its actual location. Information about the device's location is not transmitted to Apple, Skyhook or Google. Nor is it transmitted to any third-party application provider, unless the customer expressly consents". 

The data under discussion in this whole debate is clearly (in my opinion) a cache of the data mentioned here of nearby cell towers and Wi-Fi access points. I guess the remaining valid concern is that this cache is not stored as securely as it could be, and a fairly large amount of data is stored in the cache. But still this data provides only relatively coarse information as discussed here, and is stored only on the user's own computer, so the risks are relatively minor compared to many of the more dramatic scenarios that have been raised.

Update April 27: Apple has issued a Q&A document about all this, which confirms the conclusions I had drawn, and talks about changes they will make. See my thoughts here.

Read on to find out how I reached these conclusions.

The details
Last night someone called Jude commented on my last post, saying:

My Guess?

It's not a list of cell phone locations that you've been to, but the opposite, a list of cell phone locations near you downloaded to the iPhone from Apple in case you move into range of one of them. i.e. At a guess what is happening is location services identifies a cell tower and asks for its location, and is replied to with the list of locations that contains that cell tower, that list is then cached so that it does not need to be requested again.

Of course, this is only a guess based on the wide range of addresses people are seeing and how its near to, but not exactly where, the people have traveled.

Good thinking Jude! I thought this could explain a lot, so I investigated further. First I looked at some data from my fairly recent New York trip. I looked at the timestamps on some locations and did a query to display all the locations with the same timestamp. I found out that in general, quite a number of records shared the same timestamp, and they would be clustered in the same area. For example, this screen shot shows a set of records that were all loaded at exactly the same time:

This cluster of points is some way above where I drove, I was driving along the Long Island Expressway going east from LaGuardia Airport. The timestamp appears to be in seconds and has 7 decimal places, so it is apparent that this set of data must have been downloaded in a single transaction, it was not obtained by communicating with cell towers at each of these locations independently. It seems reasonable to assume that this data was downloaded to help locate me in the event that I drove into this area (which I didn't). You can observe similar clusters by clicking a dot at random, copying the timestamp, and running a filter in Google Fusion Tables to display all dots with the same timestamp.

What I really wanted to do now was to animate my data, to more easily visualize what was happening. I couldn't figure out an easy way to do this in Google Fusion tables - although it has some capability for this, it wasn't recognizing the timestamp field as a date-time. So I went to look at the data that Sean Gorman had posted of his logs at GeoCommons (my original file had been too large to visualize there without me doing a little more work). GeoCommons has a cool animation capability, which you can try out on Sean's map by dragging the sliders at the bottom left.

I found something really interesting when I zoomed in around the geoIQ office in Arlington, where Sean works. This screen shot shows that between November 11, 2010 and April 20, 2011, there is no record of Sean being at his office.

Now I know that Sean likes to escape for a spot of skiing in Colorado now and then, but that's a pretty long absence for a company President :) ! And I know I have met with him in the office during that time period.

If you drag the time slider a little further, then at the same instant, about 20 more locations appear on the map, covering a general area around the office, roughly half a mile square:

So from this data I can tell that Sean was somewhere in the general area of this half mile square (not necessarily inside it) on April 20. I know nothing about whether he was there before that, and I don't know anything about exactly where he went.

So, this data stored in the iPhone logs is much less revealing than it may initially seem. At a quick glance it does look like it is recording your location history, and I think that Pete Warden and Alasdair Allan were quite right to raise the concerns that they did. It takes some digging in the data to realize that the concerns are not nearly as bad as they appeared at first sight. By publicizing it as they did, and providing their tools and documentation on how to examine the data, they made it easy for others like myself, Sean Gorman and Will Clarke to analyze the data and figure out more about what is going on.

It's still not clear exactly what the data is for, but my guess, as Jude suggested, is that it is to aid in fast location determination - once the iPhone figures out that you're in an area, it downloads data for surrounding cell towers (and Wifi hotspots, a detail I haven't gone into here but the data is available for those too, as discussed in my previous post), so it can quickly locate you as you move around that area (update: see the first comment below, and my addition to the initial summary, which reference a document from Apple that confirms that this is the case).

So to summarize again, there are still some concerns with this data - it does give an approximate indication of places you've been, but not good enough to identify specific buildings or businesses. It doesn't record history - there is no way to tell if you've visited a location multiple times, you can just tell the last time you visited a general area (though there might be clues about multiple visits - for example data showing you visited a neighboring area on a different date, but nothing definitive or detailed about repeat visits). But it definitely doesn't reveal the sort of detailed information that many people have been concerned about.

More on Apple recording your iPhone location history

In my previous post I discussed how the location data being recorded from my iPhone actually wasn't very accurate, and certainly not accurate enough to tell where I live or work (based on the data I've examined so far, which is in a table called CellLocation in the iPhone backup, and is the data discussed by Pete Warden and displayed by his iPhoneTracker app, which is what I used for the visualizations in my previous post). Pete's app aggregated data to a regular grid, partly to provide additional security.

However, I was sufficiently intrigued to follow Pete's instructions to get at the raw data. My investigations with this reinforced the conclusion of the previous post, that the data does not accurately represent your location. But it did show up some interesting new patterns. I loaded the data into Google Fusion Tables and have made it public, you can view it here (and feel free to play around with it).

Here is an interesting map of downtown Denver, where I live.

This shows all the raw point data, with no aggregation or changes. There are actually no dots at all in the block where I live. However, there is a noticeable cluster in Coors Field, the Colorado Rockies baseball stadium which is 3 blocks away from where I live. I haven't been to the Rockies stadium over the time period that this data was recorded. There's also a strong cluster in Mile High Stadium, home of the Denver Broncos.

I would assume that there is additional cell phone infrastructure in these stadiums, to help cope with the heavy concentration of people. A quick Google search found this article about AT&T infrastructure at Coors Field

This reinforces the notion that at least some of these locations are the locations of cell equipment that your phone is communicating with. But I'm not sure that's the whole story.

Here's a map of Cropston, where I spent most of my time on my last two visits to England. It's a small village in a fairly rural area.

Here there are no locations shown in the village itself where I spent most of my time. A lot of the locations are clustered in towns or along streets, but some seem to be more in the middle of nowhere. Hard to draw any definite conclusions.

I just received a suggestion from Jonathan Barnes, via Pete Warden, that the HorizontalAccuracy field may be significant, with lower values indicated accurate locations via GPS. However, I did a quick test, for example this map filters to only show records with this field set to 500.0, the minimum value I found from a quick skim (Fusion tables seems to treat this as a string rather than a number), and while this reduces the number of records it doesn't offer any noticeable change in accuracy - it still includes all the readings from Coors Field and Mile High Stadium where I haven't actually been.

Pete also pointed me at another table in the backup called WifiLocation, which in my case was about 5 times larger than CellLocation. I have loaded this to Fusion Tables here. One interesting thing about this table is that data only shows up in North America (with one random exception in Munich, where I haven't been recently). It seems a little more focused on areas I've been to, but no more revealing in terms of showing specific locations where I've spent time.

As I said, feel free to play with the tables I uploaded, and let me know if you find anything interesting! But my conclusion remains that this data doesn't reveal where you've been with any degree of accuracy.

Update: see my latest post where I conclude that this is data being downloaded from Apple, rather than uploaded, and that detailed history is not stored - thanks to Jude in the comment below for his suggestion about this.