ArcGIS for iPad: one small step for usability!

I just did an "upgrade all" on my iPad applications, and happened to notice that ArcGIS for iPad v1.5 includes the following enhancement:

We have simplified the experience of identifying locations on the map. A simple tap on the map replaces the Identify Location tool.

This is nice to see, since I'd pointed out in a couple of my recent usability presentations that it took a minimum of six clicks, and more typically 9-10 clicks, to display feature information on ArcGIS for iPad, and suggested that perhaps a single tap on the map would be simpler. My voice has been heard!! Or maybe it was just a coincidence :), but either way a big usability improvement for the product, well done ESRI folks!

Don't make me think: Ignite Spatial NoCo 2 from Peter Batty on Vimeo.

Update: hmmm, I just tried it out briefly (rather than just reading the description), and looks as though it still takes 3 clicks to actually see the attributes of a feature, and still has some significant issues distinguishing between features that are close together. So some improvement but not as good as I had hoped.

Talk on "The Geospatial Revolution" in Minnesota

Here is a video of my recent keynote talk at the Minnesota GIS/LIS conference in Duluth, which was an excellent event. There were about 500 people there, which is great in the current economic climate. It was mainly a "traditional GIS" audience, and I got a lot of good feedback on the talk which was nice.

I talk about current trends in the industry in three main areas: moving to the mainstream (at last!); a real time, multimedia view of the world; and crowdsourcing. There's a lot of the same material that I presented in my talk with the same title at AGI GeoCommunity (which doesn't have an online video), but this one also has additional content (~50 minutes versus 30 minutes).

Click through to vimeo for a larger video, and if you click on "HD" you will get the full high definition version!! I used a different approach to produce this video compared to previous presentation videos, using a separate camera and a different layout for combining the slides and video. I like the way this came out - I'll do a separate blog post soon with some tips on how to video presentations, I think.

The Geospatial Revolution (Minnesota) from Peter Batty on Vimeo.

You can also view the slides here:

Minnesota GIS/LIS The Geospatial Revolution Peter Batty

View more documents from Peter Batty.

Shakespeare on GeoDesign

I've been trying to figure out why ESRI is suddenly trying to make such a big deal of "GeoDesign" - this announcement talks about introducing "the first generation of geodesign concepts, technologies, and tools". But people have been doing design in GIS for decades - the first release of Smallworld in 1991 was really all about design, with features like version management to enable you to analyze multiple alternative designs, etc etc. In the 1980s IBM GFIS was heavily used for design applications. I've come to the conclusion that maybe they've been trying to think of something they can say that they do which those pesky neogeographers don't do (yet!), and trying to present design as something new and cool is their attempt to do this. Anyone else have any theories? It all just seems a bit odd to me.

Anyway, I was browsing a little Shakespeare in preparation for my talks in Stratford upon Avon next week, as you do, and discovered that actually GeoDesign is even older than this. It goes back to the 16th century at least - here is a surprisingly detailed description of the design and construction process from Lord Bardolph in King Henry IV, Part 2 (maybe there's a reason why this wasn't one of his greatest hits!):

"... When we mean to build,

We first survey the plot, then draw the model;

And when we see the figure of the house,

Then we must rate the cost of the erection;

Which if we find outweighs ability,

What do we then but draw anew the model

In fewer offices, or at least desist

To build at all? Much more, in this great work,

Which is almost to pluck a kingdom down

And set another up, should we survey

The plot of situation and the model,

Consent upon a sure foundation,

Question surveyors, know our own estate,

How able such a work to undergo,

To weigh against his opposite; or else

We fortify in paper and in figures,

Using the names of men instead of men;

Like one that draws the model of a house

Beyond his power to build it; who, half through,

Gives o'er and leaves his part-created cost

A naked subject to the weeping clouds

And waste for churlish winter's tyranny."

(First posted in the comments on a previous post, but I decided it deserved its own entry!)

Traditional GIS vendor market share for 2008-2009

Daratech has published its annual analysis of the GIS industry. I thought there were a few interesting things worth commenting on there. First I should say though that I am generally somewhat skeptical of these type of reports - there is a lot of subjectivity in what gets included and what doesn't. One illustration of this was when Bentley suddenly jumped to the number 2 position ahead of Intergraph a couple of years ago - and apparently kept that for a second year. While I was clearly not in an impartial position at that time (being CTO of Intergraph), this went against the intuition of myself and others I talked to in the industry - the general feeling was that somehow Bentley had persuaded Daratech to count a lot of "CAD" revenue as "GIS". And to be fair, there is a lot of debate about how much of Intergraph's revenue is "GIS" and should be counted here. For example, Intergraph does a lot of business providing 911 call taking systems in the Public Safety space, which have a strong geospatial element to them - do you or don't you count those in this number? There is a good argument in favor, but it might not be what everyone calls "GIS". In this year's version, Intergraph's revenue is twice that of Bentley (which is hard to reconcile with Bentley being second last year).

Anyway, all those caveats notwithstanding, the top three vendors for 2009 are projected to be ESRI with 30%, Intergraph with 16%, and GE Energy (Smallworld) with somewhere around 8% (exact figure not stated in this summary, approximate figure deduced from chart). No surprise to see ESRI and Intergraph in the top two spots, which has been the generally accepted state of affairs for most industry observers for a long time. From a personal perspective it is nice to also see Smallworld (GE) still in the number 3 spot, where they have been on and off - there are several others vying for that (so my former companies get 2 of the top 3 slots!). The report also says that GE/Smallworld has top position in utilities with 24% of the market - almost as strong as ESRI's position in the overall market.

I think it's interesting though that ESRI's share (which is consistent with previous reports) is actually lower than many people perceive. ESRI clearly is the dominant player in the traditional GIS space, they enjoy an effective monopoly in many markets (for example see Andrew's recent post on a US Air Force sole source bid), and their "mindshare" is pervasive. Many people I talk to, especially those in the "neo" world, assume that "everyone" doing "traditional GIS" is using ESRI. But what this report says is that 70% of people are not using ESRI (well 70% of the revenue comes from people not using ESRI, which is not necessarily the same thing - but nevertheless, a lot of people are not using ESRI). But you have to give ESRI credit for the way they have achieved such "thought domination" with "only" a 30% market share. I often think that there are several interesting comparisons to draw between the dominant positions of ESRI and Microsoft in their respective markets (maybe there is more for a future post there). Microsoft has operating system market share of 90%-ish, and Apple somewhere around 8%, but Apple has arguably more thought leadership and buzz around its offerings than Microsoft does. But there is very little of that sort of alternative thought widely seen in the traditional GIS space, even though Intergraph and Smallworld still have certain areas where they have technical advantages and/or alternative approaches to offer compared to ESRI.

However, we are now seeing more diversity of thought reaching a broader audience from the "neogeography" side of the house, which is a good thing for the industry. And this leads into the other area that I wanted to comment on, which is how it is increasingly hard to do this kind of market share analysis on the geospatial market, as geospatial technology becomes more imbedded in mainstream IT. As geospatial data becomes just another data type and becomes an element of many different applications, how do you say that a given application is "geospatial" or not? There is no place on this list for Google, Microsoft or Oracle, for example, all of whom clearly play a major role in the geospatial market these days. And there is nothing that captures the strong growth that I perceive in open source geospatial software (by definition, software revenue is not going to be a good metric for measuring the market share of free software!).

So overall, while there are certainly some interesting tidbits in this Daratech report summary, it is not at all reflective of the overall state of the broader geospatial industry, where it is increasingly hard to even define the market, let alone to measure market share in a quantitative way.

For those that are interested in quantitative analysis of the "traditional geospatial" market, I would also suggest considering the annual Geospatial Technology Report from GITA, which uses an alternative approach based on survey responses from GITA member organizations. There are potential flaws in that methodology too, but in many ways it is less subjective than something based on vendor revenues, and it provides a lot of additional detail on various aspects of how people are using the technology, which I have found interesting in the past. It's also a lot cheaper! (I should add for full disclosure that I am a former member of the board of directors of GITA, but I receive no remuneration for sales of GITA reports!).

Update: Matt Ball has a good post discussing the relevance of these type of industry reports.

Smallworld's 20th birthday

It was brought to my attention the other day that we have just passed the 20th anniversary of the founding of Smallworld (December 3rd 1988 was its first day). Many readers of my blog these days may not be familiar with Smallworld, which is now part of GE, who acquired us in 2000. I joined Smallworld in 1992 in the UK when it was still a fairly small startup (about 30 people or so I think), was the first person to move from the UK to the US when we started here in 1993 (working with our sales guy Steve Bonifas), and I left in 2002. Smallworld grew to become the global market leader in GIS for utilities and communications companies, and almost certainly still holds that title today, depending on which market survey you believe. It certainly remains very strong in the high end of the utility and telecom market.

Smallworld really revolutionized the GIS market back in the early nineties, introducing several radical new ideas, and several ESRI insiders have since told me that ArcInfo 8.0 (now ArcGIS) was developed in order to respond to Smallworld's impact on the market - it clearly copied a number of ideas from Smallworld, as well as missing a number of important aspects too.

Some of the key innovations introduced by Smallworld at the beginning of the nineties included:

• Handling of long transactions using a new technique called version management (which became an industry standard approach some ten years later, being adopted in some form by ESRI, Intergraph and Oracle, among others)
• Outstanding performance and scalability, with the ability to interactively pan and zoom around very large continuous databases, and immediately edit features without having to extract anything, which was revolutionary at the time (most comparable systems required you to check out data from a database into a local file, which typically took minutes to do).
• Both the preceding features were due to the Version Managed Data Store (VMDS), a specialized database management system developed by Smallworld (specifically by Mark Easterfield) which was highly optimized for long transactions and spatial data - interestingly the high performance characteristics came from an elegant caching approach that was specific to the way that long transactions were handled. You can read more about this in a white paper called GIS Databases are Different, which I co-wrote with Dick Newell. There's an interesting debate of course on the extent to which the title of this paper is true - it is in some ways but not in others, but that could take up several posts in its own right so I won't go there just now! In the late nineties as use of mainstream relational databases for spatial data became more common, VMDS was really a two-edged sword from a sales perspective. The market didn't like the idea of a "proprietary" database, even though in many ways it was and still is a superior technology for interactive graphical applications in a long transaction environment.
• An extraordinarily flexible and productive development environment, based on Smallworld's own object-oriented language called Magik (initially developed by Arthur Chance, with significant contributions from Nicola Terry), which was way ahead of its time and very similar to Python, which has of course become very popular in the last few years and recognized for being an extremely productive environment. Smallworld was a pioneer in the use of object-orientation for large scale systems. The system had a unique approach which made almost any aspect of the system incredibly customizable, yet in a way which minimized support and upgrade issues. Getting into detail on that is beyond the scope of this post but maybe that's another item I'll return to in future.
• A data model based on spatial attributes rather than spatial objects. This is one of the really important features that ESRI didn't get into ArcGIS, which is still based around the idea that a feature is a point, line or polygon (with a few minor extensions, but still the concept is that a feature has a specific spatial type). In Smallworld, an object (feature) could have any number of spatial attributes, each of which could be a point, line or area, which is a much more flexible and powerful approach in many applications. This is something that you really have to design into your system from the beginning, as it fundamentally changes many aspects of the system from data modeling through to many aspects of the user interface, so I suspect that it is unlikely that ESRI will ever adopt this model at this point. I talk about some examples of how this model is useful in the technical paper AM/FM Data Modeling for Utilities, as well as discussing some other modeling constructs that were unique to Smallworld at the time, like multiple worlds.
• Integration of raster and vector data in a common database, with features for automated line following to help with "heads up digitizing" for data capture from (scanned) paper maps. Smallworld never did support traditional digitizing tablets, which were the primary means of data conversion back then, and we fought quite a lot of battles in RFPs explaining why we didn't support this often "mandatory" requirement.
• An integrated CASE tool for graphical design of your data model, which was itself version managed (the CASE tool was actually just a specialized Smallworld GIS application). Another feature of Smallworld was that you could modify the data model of a production database just within an individual version, which was incredibly powerful for being able to test changes without impacting users of the system, and then just apply them to all versions once you had tested everything. There is also a technical paper on the CASE tool.

So if Smallworld was so great, you may be asking, why it isn't the leading GIS in the marketplace today? I think that's also a story for another post (or several)! But as I said, Smallworld did become the leader in the utility and telecoms market and still holds a very strong position there today, based on the same core technologies of Magik and VMDS that were initially developed 20 years ago, which is a testimony to their strength. I would say that the early to mid nineties when Smallworld burst onto the scene in the US were the most enjoyable time of my career - it's rare that you have the opportunity to work with a technology that is so far ahead of the competition, in ways which matter to the customer, combined having great sales and marketing and all the other aspects you need to be successful - and to be in this position in a market where lots of companies are buying new systems. These days the utility GIS market is a very tough one - the three major high end systems (Smallworld, Intergraph and ESRI) all have pros and cons but none of them has a clear technical lead, and there are very few new system sales as most utilities have invested a lot in one of these and there really isn't a compelling business case to switch.

So anyway, I'd like to finish by recognizing the 10 founders of Smallworld, and congratulate them for building such a ground-breaking product and great company, and thank them for all the good times. The founders were Dick Newell, Tim Cadman, Richard Green, David Theriault, Mike Williamson, Hugh Fingland, Arthur Chance, Mark Easterfield, Cees Van Der Leeden, and Klas Lindgren. (Slightly belated) Happy Birthday!!

Improving software quality

I had missed this previously but Glenn reported from the ESRI conference that the "#1 time saver" in ArcGIS 9.3 is the ability to report product bugs more efficiently :O. Don't get me wrong, all products have bugs, I'm all for being able to report them easily, and this is a useful feature. And I'm sure customers are pleased to hear about a focus on software quality. But to me it sends a pretty negative message to the marketplace about how much time your users spend entering bugs for any software company to present this as the top productivity improvement in a major new release - it seemed like a very odd way to present it to me!

However, ESRI has not gone as far as Microsoft in its dedication to improving software quality. A friend of mine recently drew my attention to the program highlighted in the video below, which I highly recommend watching. I am thinking of instituting a similar program at Spatial Networking :) !!

Off to San Diego next week - and more GeoWeb coverage

Well, after my first ever visit to the ESRI User Conference last year, I'm going to be back there again this year, with my Enspiria hat on this time. Hope to catch up with some of you there ... I'll be around at the Enspiria booth a good bit of the time (#1907). Also, Enspiria has organized a cruise on Wednesday evening which is filling up but we have a few spots left so if you'd be interested in joining us let me know and I'll see if I can get you a ticket (no promises though!).

In other news, my GeoWeb presentation has been continuing to get some good coverage - after Ogle Earth gave it a good review, it also got mentions at Google Earth blog, Google Maps Mania (twice), and by Paul Bissett at Fiducial Marks. I have a lot of notes from GeoWeb and will try to do a post writing up my thoughts on the conference soon. But I'm off to the Santa Fe Opera tomorrow morning for a long weekend, where I look forward to catching up with my old friend Warren Ferguson, so it may have to wait until next week :).

GeoWeb report - part 1

Well, I made it back from the big road trip up to Vancouver for the GeoWeb conference. We got back home on Sunday evening, after a total of 3439 miles, going up through Wyoming and Montana and then across the Canadian Rockies through Canmore, Banff, Jasper and Whistler, and coming back in more of a straight line, through Washington, Oregon, Idaho and Utah. The Prius did very well, performing better at high speeds than I had expected.

But anyway, the GeoWeb conference was very good. The venue was excellent, especially the room where the plenary sessions were held, which was "in the round", with microphones, power and network connections for all attendees (it felt a bit like being at the United Nations). This was very good for encouraging audience interaction, even with a fairly large group. See the picture below of the closing panel featuring Michael Jones of Google, Vincent Tao of Microsoft, Ron Lake of Galdos, and me (of no fixed abode).

I will do a couple more posts as I work through my notes, but here are a few of the highlights. In his introductory comments, Ron Lake said that in past years the focus of the conference had primarily been on what the web could bring to "geo", but that now we were also seeing increasing discussion on what "geo" can bring to the web - I thought that this was a good and succinct observation.

Perhaps one of the best examples of the latter was given by Michael Jones in his keynote, where he showed a very interesting example from Google book search, which I hadn't come across before. If you do a book search for Around the World in 80 Days, and scroll down to the bottom of the screen, you will see a map with markers showing all the places mentioned in the book. When you click on a marker, you get a list of of the pages where this place is mentioned and in some cases can click through to that page.

This adds a powerful spatial dimension to a traditional text-based document. It is not much of a jump to think about incorporating this spatial dimension into the book search capability, and if you can do this on books, why not all documents indexed by Google? Michael said that he expected to see the "modality" of spatial browsing grow significantly in the next year, and he was originally going to show us a different non-public example in regard to this topic, but he couldn't as he had a problem connecting to the Google VPN. My interpretation of all this is that I think we will see some announcements from Google in the not too distant future that combine their traditional search with geospatial capabilities (of course people like MetaCarta have been doing similar things for a while, but as we have seen with Earth and Maps, if Google does it then things take on a whole new dimension).

Another item of interest that Michael mentioned is that Google is close to reaching an arrangement with the BC (British Columbia) government to publish a variety of their geospatial data via Google Earth and Maps. This was covered in an article in the Vancouver Sun, which has been referenced by various other blogs in the past couple of days (including AnyGeo, The Map Room, and All Points Blog). This could be a very significant development if other government agencies follow suit, which would make a lot of sense - it's a great way for government entities to serve their citizens, by making their data easily available through Google (or Microsoft, or whoever - this is not an exclusive arrangement with Google). There are a few other interesting things Michael mentioned which I'll save for another post.

One other theme which came up quite a lot during the conference was "traditional" geospatial data creation and update versus "user generated" data ("the crowd", "Web 2.0", etc). Several times people commented that we had attendees from two different worlds at the conference, the traditional GIS world and the "neogeography" world, and although events like this are helping to bring the two somewhat closer together, people from the two worlds tend to have differing views on topics like data update. Google's move with BC is one interesting step in bringing these together. Ron Lake also gave a good presentation with some interesting ideas on data update processes which could accommodate elements of both worlds. Important concepts here included the notions of features and observations, and of custodians, observers and subscribers. I may return to this topic in a future post.

As anticipated given the speakers, there were some good keynotes. Vint Cerf, vice president and chief Internet evangelist for Google, and widely known as a "Father of the Internet", kicked things off with an interesting presentation which talked about key architectural principles which he felt had contributed to the success of the Internet, and some thoughts on how some of these might apply to the "GeoWeb" - though as he said, he hadn't had a chance to spend too much time looking specifically at the geospatial area. I will do a separate post on that.

He was followed by Jack Dangermond, who talked on his current theme of "The Geographic Approach" - his presentation was basically a subset of the one he did at the recent ESRI user conference. He was passionate and articulate as always about all that geospatial technology can do for the world. A difference in emphasis between him and speakers from "the other world" is in the focus on the role of "GIS" and "GIS professionals". I agree that there will continue to be a lot of specialized tasks that will need to be done by "GIS professionals" - but what many of the "old guard" still don't realize, or don't want to accept, is that the great majority of useful work that is done with geospatial data will be done by people who are not geospatial professionals and do not have access to "traditional GIS" software. To extend an analogy I've used before, most useful work with numerical data is not done by mathematicians. This is not scary or bad or a knock on mathematicians (I happen to be one by the way), but it does mean that as a society we can leverage the power of numerical information by orders of magnitude more than we could if only a small elite clique of "certified mathematical professionals" were allowed to work with numbers. Substitute "geographical" or "geospatial" as appropriate in this statement to translate this to the current situation in our industry.

For example, one slide in Jack's presentation has the title "GIS servers manage geographic data". This is a true statement, but much more important is that fact that we are now in a world where ANY server can manage geographic data - formats like geoRSS and KML enable this, together with the fact that all the major database management systems are providing support for spatial data. There is a widely stated "fact" that many people in the geospatial industry have quoted over the years, that something like 85% of data has a geospatial component (I have never seen a source for this claim though - has anyone else?). Whatever the actual number, it certainly seems reasonable to claim that "most" data has a spatial component. So does that mean that 85% of data needs to be stored in special "GIS servers"? Of course not, that is why it is so significant that we really are crossing the threshold to where geospatial data is just another data type, which can be handled by a wide range of information systems, so we can just add that spatial component into existing data where it currently is. Jack also continues to label Google and Microsoft as "consumer" systems when, as I've said before, they are clearly much more than that already, and their role in non-consumer applications will continue to increase rapidly.

But anyway, as Ron said in his introduction, it would be hard to get two better qualified people than Jack and Vint to talk about some of the key concepts of "geo" and "web", so it was an excellent opening session. I think that this post is more than long enough by this point, so I'll wrap it up here and save further ramblings for part 2!

The "dark side" poll - neck and neck!

Well, after my previous post where I reported that the rebel forces had pulled ahead, the Empire did indeed regroup and pulled ahead slightly today, but at the time of writing the two sides of the Force are locked in a deadly embrace - 119 votes for the dark side and 119 votes against, with 28 don't knows.
Thanks also to the various people who have emailed me with advice! I should point out by the way, to allay the concerns of some correspondents, that I am exploring a lot of possibilities right now. I am definitely interested in the "neogeography" space and what Google, Microsoft, Yahoo et al are doing there, and there could be opportunities either with the big guys themselves, existing startups leveraging their technology, or in doing my own startup. There are also interesting opportunities in the open source arena. And last and perhaps least (but not necessarily!), there are opportunities with the more established geospatial companies.

So the topic of this poll is really more along the lines of should "going to the dark side" be one of the options that gets into my shortlist for serious consideration? But thanks for all the votes, the poll is on my main blog page if you haven't voted yet and would like to!

Thoughts on GE's next generation system based on Oracle - part 2

This is a continuation of my previous post about GE (General Electric)'s next generation system based on Oracle - please read that before reading this if you haven't already done so. At the end of part 1, I said that in part 2 I would talk about the real reason why this product could be a significant jump forward for the utility industry, which really hasn't been highlighted in the GE announcements or in any of the commentary I've seen - and also why this same factor could be the reason that the product fails. And last but not least, I'll talk about some of the challenges which GE faces in positioning the new product with regard to the existing Smallworld products. So here we go ...

I think that the factor that could make this product a significant jump forward is that, as I understand it from contacts at GE, they are really trying to produce something that is an off the shelf product which can be configured rather than customized (the distinction being that configuration just involves setting certain parameters about how the application behaves, as opposed to customization which involves writing code). Now GE really didn't talk about this in their announcement, and this information comes from informal conversations, so it's possible this emphasis may not be as strong as I had inferred - but either way it is interesting to talk about the pros and cons of this approach.

Historically in the utility industry, the implementation of geospatial systems has involved buying a software product as a starting point (with GE, Intergraph and ESRI being the three primary vendors in the space), and then doing some customization (data modeling, further software development, etc) on top of that core product to meet each individual utility's specific requirements. This approach enables each utility to get a system which closely meets its specific requirements, but has drawbacks: apart from the additional cost for the initial customization, having many different custom systems makes support and upgrade procedures harder for vendor and user alike. All the major vendors have generally had a "starter system" or "template" for the customization, which reduces the cost of the initial implementation, but in general (in my experience) has not really helped in terms of simplifying ongoing support and upgrade issues, and thereby reducing ongoing cost of ownership. In the early 2000s, when Intergraph launched its G/Technology product, initially the intent was that it would be a completely off the shelf system, allowing some configuration but not customization. While many utilities liked the principle, nearly all wanted additional functionality in their systems (and of course typically different utilities wanted different additional functionality). So Intergraph ended up having to rearchitect their initial approach to allow the system to be customized in a more flexible way, which was not a trivial undertaking and probably cost them a few years in terms of getting a competitive version of G/Technology to market.

So it will be interesting to see if GE really does pursue the angle that this will be an "off the shelf solution". It is hard to be wishy-washy about this - if you say that you think it will meet many customer needs off the shelf but you can still customize it if you want, then you really don't address the ongoing cost of ownership issues associated with the complexity of supporting and upgrading all these different systems in the field (unless you put very strict constraints on the customization, and really constrain yourself to not modify APIs from one release to the next). If GE does take a truly off the shelf approach then this would differentiate the product in the market IF it was functionally rich enough - but the risk if the product is not functionally rich enough is that you won't be able to win business and it will set back your entry to the market until either you allow customization (which makes you less differentiated) or you add sufficient functionality that you can be competitive - either of which may take a significant amount of time.

This leads on to the challenges that GE faces in terms of positioning the new product versus the current Smallworld product. Now GE is specifically trying not to position this as a replacement for Smallworld, and is saying that it will continue to develop new functionality on both platforms. It's fine to say that, but obviously the challenge with this approach is that if you develop all functionality on two different platforms which don't support common code, you can only develop half as much new functionality with the same number of developers (well maybe a bit more than half as much, since some design work could be common across both platforms, but certainly you can do a lot less than you could do if you just focused on a single platform). So that really doesn't seem like a sustainable approach for a long period of time, unless GE is prepared to substantially increase the size of its development team, which I imagine would be hard for it to justify. GE is initially focusing the new product specifically on North American mid-size electric utilities, so migrating to the new product will not be an option yet for customers who do not fall into that category, which is a fairly large majority.

For those customers who are in a position to consider migration (the North American mid-size electric utilities - and presumably the segments addressed will expand over time), GE will face the classic challenges of any company going through a major technology upgrade (and which Intergraph went through - and is still going through - with the migration from its older FRAMME product to G/Technology, and ESRI from Arc/Info 7 to ArcGIS). One challenge is that with a product like Smallworld that is 16 years old and exceptionally customizable, most customers have very rich functionality which it is hard to replace with a product that has only been under development for a year or two. There will either be custom development necessary to replace custom functionality in the old system, or the customer will need to be persuaded to give up some existing functionality to get other benefits that can be obtained from the new architecture. This generally means that the migration from the old system to the new system is a large enough project that most organizations will take the opportunity to evaluate other systems on the market and decide whether to stay with the same vendor or switch to a new one. There has been little turnover in the utility market in recent years, so on the rare occasions that utilities do choose a new system, all the vendors are very hungry for those opportunities and pricing tends to be very competitive, especially since the three major systems are not highly differentiated these days. As I said, none of this part is unique to GE, it is the same situation that ESRI and Intergraph have gone through as they have been migrating their customers to newer technologies.

One other challenge GE may have is with the customers who are not yet addressed by the new product. It will presumably take several years before the new product is an option for all of them (I believe that GE is saying that they will have a beta out sometime this year, then a first release first half of next year, and unless it's different from all other software products it will probably need a second release before it's really ready for serious use, so suppose that arrives late 2008 or early 2009, then probably the product is not going to be expanding to substantial additional market segments until 2009 or 2010). Now by and large I think that the Smallworld customer base is still pretty happy, so maybe customers will be willing to continue, assuming that GE continues to invest in Smallworld as it says it will. But there is also the risk that some customers will decide that this all means that the writing is on the wall for the Smallworld products, even if they keep going for a few more years, so maybe they should just go out and look at Intergraph and ESRI, who are running on more mainstream architectures which are in production today.

One other challenge for GE as they try to address moving their larger customers to the new platform at some point in the future (assuming they do) will be how to provide the same level of scalability that Smallworld VMDS does - perhaps that's a topic for a future discussion.

So anyway, it will be interesting to see how all this pans out over the next few years. I wish the GE guys good luck with it - as I said before, the industry can use additional innovation and competition!

Quick update on my "going to the dark side" poll

Wow, since I posted last night that the voting in my pollmappr poll was significantly in favor of me going to the dark side (42 for, 25 against, 11 don't know), the rebel forces have mobilized and have now moved into the lead - the current count is 70 for, 77 against, 17 don't know. So a resounding 45 votes against versus 28 for the dark side since last night. Will the Empire strike back? We'll see :) !!

Thoughts on GE's next generation system based on Oracle - part 1

A number of people have asked about my thoughts on GE's announcement back in March about a next generation system based on top of Oracle technology (note that in this post, GE stands for General Electric, not Google Earth!). This was covered by Joe Francica at All Points Blog under the title "Cutting out the GIS Middleman", by Susan Smith at GISCafe in an interview with Robert Laudati, and in an article by GE. Here are my thoughts ... (warning - this is a rather long post, in fact it was getting so long I have decided to split it into two parts, and the first part is still very long!).

I'll give a quick bit of history on Smallworld for all you "neo" people and those who haven't had any involvement with GIS in the utility industry. Smallworld is where I worked from 1992 to 2002, and during that time we grew from being a small startup in the UK to the global market leader in GIS for utilities and communications (according to Daratech), with revenues of around $100m. Smallworld was bought by General Electric in 2000. Smallworld introduced some radical new ideas in the early 1990s, many of which have now become common practice across the industry - I will talk more about some of these in the future, but Charlie Savage gives a good summary of his perspective here. In the 1990s, Smallworld had a clear technical lead in the utility industry, but in the early 2000s both ESRI and Intergraph introduced new systems (ArcInfo 8.0, now ArcGIS, and G/Technology), and the playing field is now much more even, with no clear leader, in my opinion. The Smallworld product remains very robust and scalable and has very rich functionality, but increasingly GE has been suffering in new sales situations because of the fact that Smallworld is based on its own proprietary language (Magik) and database (VMDS, Version Managed Datastore), while its two primary competitors have more modern and mainstream software architectures. Actually both Magik and VMDS still have some great technical strengths, especially the latter, but as geospatial technology has moved more into the mainstream, it has become increasingly hard to convince the market of the benefits of buying a "proprietary" solution, and in my opinion this has been the primary driver for GE to develop this new product set on top of Oracle.

The most obvious interesting thing about this announcement, which others have commented on also, is that it really reinforces the notion that geospatial technology is becoming absorbed into mainstream IT - many people have talked about this for a while, myself included. The way it was explained to me by someone from GE is that when they first sat down with Oracle to discuss collaboration on this project, they thought that there would be three layers of software: Oracle technology at the back end, a new "GIS" layer in the middle, and specific utility applications built on top of that. But when they looked at what Oracle now offers, including a map viewer, network model, version management (workspace management in Oracle terminology), etc, they concluded that there were really just two layers: Oracle, and the specific utility applications. I don't think this point is especially interesting to customers, actually: whether you regard the solution as 2 tiers or 3 tiers is a bit of an arbitrary distinction - in either case you need software from two vendors, Oracle and GE, and whether specific bits of functionality come from one or the other is not really significant. If GE reduces the cost of their software because they don't have to develop as much functionality themselves, then customers will be interested, but I haven't heard any discussion about that!

The reason this is interesting is from a general industry perspective, because it suggests that it will be harder to have a successful business which focuses purely on the middle "GIS" layer, without delivering applications on top of that which solve specific business problems. There are now many free or cheap options for drawing a map on a screen; you no longer need a specialized and expensive piece of software to do that. Companies like GE and Intergraph sell both geospatial platform software and vertical industry applications, and both are now putting more emphasis on the vertical applications where (in the right areas), they can still show high business value and therefore justify relatively high prices, and less emphasis on the basic geospatial capabilities, many of which are becoming commoditized. ESRI has of course focused heavily on delivering a horizontal GIS product, and has a large partner ecosystem which provides vertical applications on top of this. They have a very dominant position in this horizontal GIS space, and the fact that many of their traditional competitors are focusing more on vertical applications may mean that they can increase their hold there - but nevertheless they are seeing significant pressure on various parts of this space from Oracle, Google, Microsoft, Yahoo and open source solutions. So it will be interesting to see whether ESRI keeps its horizontal focus or also starts to move into more vertical solutions over time. There would be some challenges in this strategy, in particular the issue of potentially competing with its partners - though this is a common problem for platform software companies and is in many ways a natural evolution that many companies go through. We went through this at Smallworld, starting as a platform company with partners and then moving more into vertical applications, and Oracle is experiencing this also, especially with several of its recent acquisitions (more on that in a future post).

OK, so does this new architecture give GE a competitive advantage? Not really, in my opinion. The first major argument in favor of it is that all your data is stored in a standard relational database, which helps with integration, administration, security, etc (see my 1990 article which outlines these benefits - these concepts are not new!). The second major argument is that you can use standard development environments (Java-based, in this case). So GE addresses the concerns that the market today has about its existing "proprietary" solution - but both ESRI and Intergraph have provided solutions based on mainstream databases and development environments for a number of years, so these things will not be differentiators for GE - they are playing catch up in this regard. Oracle likes the fact that the solution is based purely on the Oracle stack - and so customers who have committed to the whole Oracle stack will also see that as an advantage. But on the other hand, customers focused on a Microsoft architecture either on the client or middle tier will see this approach as a disadvantage - and my general feeling is that more utilities probably fall into this category. Those who are not too religious about their IT strategy (probably the majority) will focus on the functionality provided by the main three vendors more than the system architecture.

So, in summary on part 1: I am pleased to see GE make this announcement, as I had personally pretty much written off the possibility of them investing in a "next generation" system. I am happy for the friends I still have at GE that there is investment in the future, and I think it will be good for the industry if GE can make this new product into a strong competitor to ESRI and Intergraph in the utility market - and if they can bring forward the strengths of the current Smallworld system then it will be. The announcement is interesting because it shows that you can now develop complex geospatial applications directly on top of Oracle without needing a traditional "GIS". While I think this is of somewhat academic interest to most customers, it is more significant in terms of what vendors in the geospatial industry will look like in future. I personally don't think that the market will see the new system architecture as a competitive advantage, except in situations where organizations have very strong Oracle religion - I think it will be seen as more of a catch up exercise by most people, which negates the perceived weakness of Smallworld's current "proprietary" architecture.

In part 2 I will talk about the real reason why this product could be a significant jump forward for the utility industry, which really hasn't been highlighted in the GE announcements or in any of the commentary I've seen. And I'll talk about why this same factor could be the reason that the product fails. And last but not least, I'll talk about some of the challenges which GE faces in positioning the new product with regard to the existing Smallworld products.

My pollmappr poll so far

Just for a little fun, and to try out pollmappr from FortiusOne, I asked the question "Should Peter Batty consider 'going to the dark side' and working somewhere in the ESRI universe?". You can see the latest results (and/or vote) in the box below (or click here if you can't see a box).

At the time of writing, the scores are:

• 42 votes (54%) for "Yes, it is your destiny"
  
• 25 votes (32%) for "No, don't do it!"
  
• 11 votes (14%) for "Don't know, don't care, who is Peter Batty?"
  

Where are all you Intergraph and Smallworld voters, I ask?!

Anyway, when I went to look at the spatial distribution of the votes, I was a little disappointed to find that only a small proportion appeared on the map, and all of these were in the US, so I surmised that pollmappr is currently only mapping votes from the US. I checked in with Sean Gorman at FortiusOne and he confirmed that this was the case currently, but he said that the next release will show the results as point locations with a heat map and will not need Google Earth for display, and that they will also become data sets in GeoCommons and will be mashable. So actually on re-reading his email he didn't quite answer my question :), but I will put my vote in for support for global data in future!

Anyway, here's a screen shot of the "yes" votes at the time of writing:

Not too surprisingly I guess, California leads the yes votes, with seven, followed by Colorado with three and seven other states with one each. But the more interesting statistic is that only 17 out of 42 "yes" votes (about 40%) were from the US, and only 4 out of 25 "no" votes (16%) are from the US, and 4 out of 11 "don't know" votes (36%). So if you look at US votes only, the score is 17-4-4, or 68% yes, 16% no, 16% don't know. Outside the US the score is 25-21-7, or 47% yes, 40% no, 13% don't know. So albeit on a small sample size, there is quite a difference between the US and non-US scores, but we don't know anything about the non-US distribution at this point.

I look forward to the next release Sean, will keep gathering data until then!

Adapx digital pen for paper input

One company that got a lot of attention at the ESRI User Conference was Adapx, who launched their application which allows you to use a paper map as an intelligent digital input device. John Calkins of ESRI demonstrated this in the plenary session with various other "cool things". Several others have blogged about this, including Joe Francica at All Points Blog, who includes a short video of a demo of the system. The underlying technology comes from a company called Anoto, who have been around for a while - I talked about this technology in a presentation at the GITA conference in 2004. The trick is that the paper is "watermarked" with a pattern of dots which allow the pen to determine its location in a very large coordinate space. One of the clever things about the solution is that the pen and paper is used not just for entering the location of geospatial features, but for other "user interface" actions such as specifying the type of feature to be captured. So you can click on a legend at the bottom of the map to say that you are about to add manholes (or whatever), and subsequent clicks will be treated accordingly. You can also capture attribute data using printed lists, or handwriting recognition. All of the intelligence to interpret these actions is applied when you upload the data, so the pen needs to store the sequence of events. This also means that if the system doesn't register a click on the legend, or registers the wrong item, you could get a significant amount of data captured incorrectly without the user realizing, as they have no real time feedback on what the system currently thinks they are doing. I guess time will tell whether this is a significant issue in practice or not. But certainly many field users will like the concept of being able to use paper and pen for input.

So it will be interesting to see whether this really takes off for widespread use. I think that one of the most promising uses is probably in areas where people currently use large paper maps or drawings in the field, such as on construction sites. It has always been a problem to provide the same capabilities on a laptop or tablet screen that you can get from a large paper construction drawing with a group of people standing round looking at it - you just don't have the real estate on a screen to show all that detail at one time, and share it with a group. The ability to mark up a map digitally in this situation could be very useful. Still for many work scenarios, for example where people visit lots of different locations which are not predefined, a digital display is likely to be more useful than having to carry very large numbers of paper maps.

But it's definitely an interesting alternative input mechanism in a number of scenarios, I look forward to seeing how it does in the market!

Microsoft SQL Server Spatial update

I attended the Microsoft SQL Server SIG at the ESRI User Conference to hear a short update from Ed Katibah, who is leading the development of the SQL Server spatial capabilities. I thought he said a few interesting things of note, which I hadn't heard elsewhere. One was that they are doing nightly performance tests against all their major competitors, including Oracle, Postgres, Informix and DB2, and, while he wasn't allowed to be specific, he was "very pleased" with the results. He showed an example of a polygon with 600,000 vertices and 7,000 holes, and said that it could be intersected with an offset version of itself in 15 seconds on a 1GHz machine. From the way Ed described the spatial indexing approach, I suspect that it will probably do a pretty good job on this problem of Paul Ramsey's, though I'm sure it would take more than that for Paul to consider a Microsoft solution :) ! He mentioned that Microsoft was rejoining OGC - they had left because of legal concerns relating to how OGC handled certain IP issues, but these had now been resolved. He said that the target for release was middle of next year, and that there was a very strong focus within Microsoft on meeting that deadline. Both Ed and various ESRI people mentioned that Microsoft and ESRI had been working together to ensure that ESRI would support the SQL Server Spatial capabilities.

ESRI integration with Google and Microsoft

I thought that one of the more significant announcements in the plenary session at the ESRI User Conference was the functionality in 9.3 relating to integration with Virtual Earth and Google Maps / Google Earth. Up to this point, as I've commented before, ESRI has seemed a little reluctant to integrate with these systems, and third party software like Arc2Earth has filled that hole. The Microsoft Virtual Earth blog talks in more detail about the integration capabilities with Virtual Earth. In the plenary, there was a brief demo which showed a nice looking analysis from ArcGIS Server overlaid in a Virtual Earth environment. Given the results of the ESRI customer poll in this area, I guess this type of integration was inevitable, but I still think it's a significant step. Jack consistently tried to position Google and Microsoft as "consumer" products in his talk, but it is clear that they are already being used in many business-oriented applications. Once these easier integration capabilities are available, it will be interesting to see whether that accelerates the move of these "consumer" systems into the application spaces traditionally occupied by the established geospatial vendors. This is scheduled to be available from ESRI next year, while Intergraph plans to provide similar capabilities this year.

ESRI User Conference - general impressions

Apologies for not blogging more during the ESRI User Conference, I required all my mental energy to avoid being completely consumed by the Dark Side of the Force :) !! I held out (for the moment at least), but this picture of me towards the end of my three days there shows some cause for concern! After my earlier post on "The Force", Ed Parsons emailed me with the following quote: "Don't be too proud of this technological terror you've constructed. The ability to destroy a planet is insignificant next to the power of the Force". And I have to say that the Force was indeed very powerful in San Diego.
So anyway, this was my first ESRI User Conference after twenty years in the industry, as I have always worked for ESRI competitors, and therefore been on the uninvited list. Others who are more into the specifics of the ESRI products than me have shared lots of details, so I thought I would just share some more general impressions here (mainly for others who are uninitiated), and I will talk about a few more specific things that interested me in subsequent posts.

As many people had told me beforehand, the plenary sessions which fill the whole first day really are a very impressive show, with 15,000 (ish) attendees in a huge hall with 3 giant screens. Jack presided, talking a lot on his usual theme of all the good things that GIS can do for the world, and his talks were interspersed with many presentations and demos from other ESRI staff and customers, all of which were very well rehearsed and choreographed. There was an excellent talk by Nobel Peace Prize winner Wangari Maathai. The message that ESRI customers are doing great and important things was repeated a lot. It is easy to see how people get caught up in all this and catch the "ESRI religion" which I have seen throughout my career, from the "other side". The exhibit floor is also huge and very impressive (and a great place for networking if you're trying to work out your next move in the industry!). And there is also a vast "map gallery" exhibition, where customers show off what they have been doing.

So all that was great, but posts like this one from Sebastian Good help remind you that everything that is presented in the choreographed sessions may not correspond to the real world. There was an admission in the general session that there had been significant support issues with ArcGIS 9.2, and a discussion on what they plan to do about it. And based on my conversations with both customers and implementers who have worked with multiple systems, I think that my former companies, Intergraph and Smallworld (GE), continue to have technical advantages over ESRI in some specific areas including scalability (many concurrent users), workflow, network modeling, and robustness. And of course ESRI faces growing competition, as do all the established geospatial vendors, from the rise of Google, Microsoft and open source solutions (not in all aspects of what it does, but in some significant respects).

But having said all this, when you look at the scale and scope of what ESRI is doing, the religious devotion of its customer base, and the huge effort that it is putting into product development, it remains a daunting task for its competitors to make significant inroads into its Microsoft-like dominance of the industry, especially in its core "professional GIS" space. Though in the area of distributing and sharing geospatial data and certain categories of applications, there is an interesting battle shaping up with the "neogeography" systems, and in some specific markets there is greater competition - for example in the utility market, where it remains a close fought battle between the three major contenders (ESRI, Intergraph and GE Smallworld - General Electric not Google Earth!).

So now the big question for me is ... should I consider going over to "the dark side" with my next career move? For a little bit of fun, I have put together a geospatial poll at pollmappr so you can give me your input. Let me know what you think :) !!

ESRI user conference under way

Here's a picture of Jack doing his stuff

Interesting points from ESRI customer survey

ESRI has published a lengthy pre-conference Q&A document on the user conference blog, which several people have commented on. One answer talked about results from their customer survey, and I thought this highlighted some interesting industry trends.

They said that 45% of customers have asked for tight integration with Google Earth and nearly 47% have asked for support for interoperability - so overall, 92% of ESRI customers are looking for integration with Google Earth (assuming that these two response categories were mutually exclusive, which seems to be the case from the context). For Virtual Earth the numbers were a little lower, 26% and 43%, so 69% in total. This is just reconfirmation of the trend we are all aware of that "serious" GIS users are interested in using Google and Microsoft as a means to distribute their data - but it's interesting to see hard numbers, and 92% is a resounding endorsement for Google. It's also interesting that the vote for Google is quite a bit higher than Microsoft. I think that in the consumer world and the blogosphere, Google has pretty clearly had a higher geospatial profile, but among "corporate" GIS users I have talked to, many have had a bit more of a leaning towards Microsoft, if only because their organizations tend to be doing business with Microsoft already. This survey goes against the subjective impression I had formed on that particular point (admittedly from a small sample size).

The other point that was interesting was that 80% of customers want ESRI to support or tightly integrate their technology with the upcoming Microsoft SQL Server spatial extension - this is a very high number, especially given that Oracle probably still has around 50% of the database market share (48.6% in 2005, according to Gartner). These two numbers don't directly correspond in that the ESRI number is based on number of customers, so is likely to more strongly reflect the interests of smaller organizations (assuming that there are a large number of small organizations responding), whereas the Gartner number is based on revenue so probably more influenced by large organizations. But nevertheless, a very strong statement about the level of interest in Microsoft SQL Server Spatial.

There is a separate statement that less than 19% of customers have asked for tight integration with Oracle Spatial - but unfortunately no comment on what percentage want "support" for Oracle Spatial (which is currently provided via what was ArcSDE, now part of ArcGIS Server), so no direct information on relative levels of interest in Oracle versus SQL Server. I have been thinking for a little while that Oracle Spatial is at an interesting juncture in terms of its position in the market, but I'll save my thoughts on that for a future post :) !

I sense a disturbance in the Force

As I have spent the past 20 years working with various rebel organizations who all regarded ESRI as "The Dark Side", it feels a little strange for me to say that I will be attending my first ESRI user conference next week (I will be there Sunday through Tuesday). Now that I am in a "neutral" role (i.e. unemployed!), the folks at ESRI kindly agreed that I could attend - thanks to David Maguire in particular for helping out with this (despite being a Manchester United fan, David is not such a bad bloke really!). I have heard many stories about the user conference over the years, and am looking forward to the experience and to finding out more about what's going on in that part of the universe.

I was looking around for a picture to illustrate this post, and I thought this one was amusing (from Presentation Zen):I won't make any further comments for now, I've probably got myself into enough trouble with all factions already :) !!