New advisory role with Enspiria

As Enspiria announced on Friday, I have taken on an advisory role with them. As Chief Technical Advisor I'll be working with them on their technical strategy. They're involved in a number of interesting areas, especially around the integration of geospatial technologies with the "smart grid". Enspiria has a lot of good people, many of whom I've been friends with for a long time, so I'm looking forward to working with them. And it's a flexible arrangement so I can continue to focus most of my time on Spatial Networking (which is coming along well ... more on that very shortly!).

Follow up discussion on GE next generation system (about customization and open source)

My two part post last week on General Electric's next generation system based on Oracle (part 1 and part 2) generated some interesting follow up comments and questions (mainly after part 2). I got somewhat sidetracked with the whole iPhone extravaganza over the past few days, but wanted to circle back and follow up on a couple of threads.

The first thread related to customization versus configuration, and the fact that there are potential attractions in being able to provide an "off the shelf" system, which is just configured and not customized, for a specific application area - in this case design and asset management for mid-size electric utilities in North America. If you can achieve this, then potentially you can significantly reduce implementation costs and (in particular) ongoing support and upgrade costs. However, the challenge lies in whether you can meet customer needs well enough in this type of complex application area with just configuration. People have tried to do this multiple times in the utility industry, but so far nobody has been successful, and everyone has fallen back to an approach which requires customization. Both Roberto Falco and Jonathan Hartley were skeptical that a pure configuration approach could be successful, and Jonathan makes a good argument in his response that if you implement a very complex configuration system, then you may end up just recreating a customization environment which is less flexible and less standard than if you had just allowed appropriate customization in the first place. I don't disagree with either of them in general - though I would make the comment to Jon that I don't think we're talking about "GIS" in general, it's a specific application of that to electric utilities in a relatively narrow market segment, so that gives you a little more chance of defining the problem specifically enough that a configurable system could make sense. One other observation I would make is that I think that cost is an element of this equation. If an "off the shelf" system meets 90% of an organization's stated requirements and costs 80% of the amount of a fully customized system which meets 100% of their requirements, that is much less compelling than if the off the shelf system cost (say) 20% of the fully customized system, in which case they might be more willing to make a few workarounds or changes to business processes to accommodate such a system. This may be stating the obvious, but I wonder whether organizations trying to develop this "off the shelf " approach go into it thinking that they will have to offer a substantially lower price (or provide other compelling benefits) to persuade customers to adopt it. Finally on this thread, I think it is also worth observing that the "GIS" (if you call it that) in a utility is not a standalone system - it typically requires some sort of integration or interfaces with many other systems, such as customer information, work management, outage management, workforce management, etc etc. This makes it an even bigger challenge to develop something which does not require customization.

Paul Ramsey raised a different question, that of open source, which I think is interesting in this context. But first I will just answer a couple of other questions that Paul raised. He asked if this was essentially a replacement market, and the answer is yes - especially among the medium and large utilities, and in the more established markets (North America, Western Europe, Japan, Australia / New Zealand, etc), pretty well everyone has a system implemented, and GE/Smallworld, Intergraph and ESRI are the dominant vendors. Because of the customized nature of these systems, and the amount of integration with other business systems, switching to a new vendor is typically a multi-million dollar project, even if the new software is provided for free. So this is obviously a big reason why this is a "sticky" market and customers don't change systems very often. The other clarification is that Paul asks about defining the market of customers "who think that 'classic' Smallworld doesn't cut it anymore", and overall I wouldn't particularly categorize things that way. Overall I think that satisfaction levels are probably as high among the Smallworld customer base as with any of the other vendors, the system is functionally very competitive still, the main concern is probably in obtaining and retaining people with the appropriate specialized skills, especially for smaller organizations. But there has been very little movement of utilities from Smallworld to other companies up to this point. Ironically, GE bringing out a next generation system (which they had to do for new business) is something which may cause existing customers to re-evaluate their strategy. Again though this is just a standard challenge of moving to a new generation architecture for any software company - you're damned if you do and damned if you don't.

Anyway, back to open source - Paul raises the question of whether GE doing something with open source may be an option. I have heard a few people raise this before in regard to Smallworld, and it is an interesting question. There are three major elements to the Smallworld core software. The first is the Magik programming language, which was developed by Smallworld back in the late eighties, and it is very similar to Python and Ruby. Smallworld was way ahead of its time in recognizing the productivity benefits of this type of language, and this was a key reason for its success. The core Magik virtual machine has been stable for a long time, the original developers of it left Smallworld a number of years ago and I suspect that nothing much has changed at this level of the system recently. The second key element is VMDS, the Version Managed Datastore. There is some very complex and low level code in both of these components which I suspect would make it hard to open source them effectively, given the amount of time it would take people to get into the details of these components (currently known to only a very few people either inside GE, or who have left GE), and the relatively small size of the market for the products would probably be a disincentive for people to make the effort to learn this. However, both these components are very stable and probably don't really need much maintenance or change. The vast majority of the system is written in Magik, and the bulk of the Magik source code for Smallworld GIS has always been shipped with the product, which has allowed for a hugely flexible customization environment (much more flexible than Smallworld's main competitors). There is a pretty large base of technical resources in customers and Smallworld partners who know how to enhance the system using this environment. If GE could manage enhancements at this level of the system in an open source fashion, to leverage the strength of the existing Smallworld technical community, who are on the whole very loyal to Smallworld and have a strong interest in seeing it continue to be successful, this could be a very powerful thing. As I noted in my previous post, one of the challenges for GE is that to develop new applications on two platforms concurrently (Smallworld and the new Oracle/Java based platform) will significantly decrease the amount of new functionality they can develop with a finite set of resources.

Of course there are a lot of challenges for an existing commercial software company in making such a switch. A critical one for GE would be that they could maintain their existing maintenance revenue from the Smallworld user base (at least to a large degree), otherwise it would be a non-starter from a business perspective I think. But it is quite conceivable that this approach could harness the talents of a much larger group of developers than Smallworld currently has working on the product, and produce more enhancements and fixes to the products than customers currently see, so you can see scenarios in which customers would be happy to continue to pay maintenance for support. As Paul points out, Autodesk has made this work and most of their customers still pay maintenance. There are differences in the Autodesk scenario as they open sourced a new product rather than an old one, and I think that a big driver for them was that they saw that it was going to be increasingly hard to make money for basic web mapping products, as that area becomes increasingly commoditized, due to the efforts of both the Googles and Microsofts of the world as well as the open source community. I think that this factor probably helped them decide to make the big leap to an open source approach, and it seems to have been a successful one for them.

Could GE also consider open sourcing portions of the new product, which I think was really Paul's question? That could also be an interesting possibility, to help gain traction in the market. If they open sourced some of the more generic base level components of the product, they could still build their specific business applications on top of these components and charge money for those, but leverage the open source community to provide additional functionality.

So the motivations are somewhat different for GE than for Autodesk, and there are multiple scenarios where open source could play a role, but I could envision an open source version of Smallworld significantly extending the product's life, by harnessing the very good technical resources which exist in the broader Smallworld community. Internal GE resources could then be freed up to work on the new product line (with just a small number focused on managing the open source developments of the established Smallworld product line). There are certainly a number of challenges and a lot of complexity in making something like this fly from a business perspective, and I'm not sure if GE would have the imagination or the boldness to take a risk on it, but it's an interesting thing to speculate about!

Thoughts on GE's next generation system based on Oracle - part 2

This is a continuation of my previous post about GE (General Electric)'s next generation system based on Oracle - please read that before reading this if you haven't already done so. At the end of part 1, I said that in part 2 I would talk about the real reason why this product could be a significant jump forward for the utility industry, which really hasn't been highlighted in the GE announcements or in any of the commentary I've seen - and also why this same factor could be the reason that the product fails. And last but not least, I'll talk about some of the challenges which GE faces in positioning the new product with regard to the existing Smallworld products. So here we go ...

I think that the factor that could make this product a significant jump forward is that, as I understand it from contacts at GE, they are really trying to produce something that is an off the shelf product which can be configured rather than customized (the distinction being that configuration just involves setting certain parameters about how the application behaves, as opposed to customization which involves writing code). Now GE really didn't talk about this in their announcement, and this information comes from informal conversations, so it's possible this emphasis may not be as strong as I had inferred - but either way it is interesting to talk about the pros and cons of this approach.

Historically in the utility industry, the implementation of geospatial systems has involved buying a software product as a starting point (with GE, Intergraph and ESRI being the three primary vendors in the space), and then doing some customization (data modeling, further software development, etc) on top of that core product to meet each individual utility's specific requirements. This approach enables each utility to get a system which closely meets its specific requirements, but has drawbacks: apart from the additional cost for the initial customization, having many different custom systems makes support and upgrade procedures harder for vendor and user alike. All the major vendors have generally had a "starter system" or "template" for the customization, which reduces the cost of the initial implementation, but in general (in my experience) has not really helped in terms of simplifying ongoing support and upgrade issues, and thereby reducing ongoing cost of ownership. In the early 2000s, when Intergraph launched its G/Technology product, initially the intent was that it would be a completely off the shelf system, allowing some configuration but not customization. While many utilities liked the principle, nearly all wanted additional functionality in their systems (and of course typically different utilities wanted different additional functionality). So Intergraph ended up having to rearchitect their initial approach to allow the system to be customized in a more flexible way, which was not a trivial undertaking and probably cost them a few years in terms of getting a competitive version of G/Technology to market.

So it will be interesting to see if GE really does pursue the angle that this will be an "off the shelf solution". It is hard to be wishy-washy about this - if you say that you think it will meet many customer needs off the shelf but you can still customize it if you want, then you really don't address the ongoing cost of ownership issues associated with the complexity of supporting and upgrading all these different systems in the field (unless you put very strict constraints on the customization, and really constrain yourself to not modify APIs from one release to the next). If GE does take a truly off the shelf approach then this would differentiate the product in the market IF it was functionally rich enough - but the risk if the product is not functionally rich enough is that you won't be able to win business and it will set back your entry to the market until either you allow customization (which makes you less differentiated) or you add sufficient functionality that you can be competitive - either of which may take a significant amount of time.

This leads on to the challenges that GE faces in terms of positioning the new product versus the current Smallworld product. Now GE is specifically trying not to position this as a replacement for Smallworld, and is saying that it will continue to develop new functionality on both platforms. It's fine to say that, but obviously the challenge with this approach is that if you develop all functionality on two different platforms which don't support common code, you can only develop half as much new functionality with the same number of developers (well maybe a bit more than half as much, since some design work could be common across both platforms, but certainly you can do a lot less than you could do if you just focused on a single platform). So that really doesn't seem like a sustainable approach for a long period of time, unless GE is prepared to substantially increase the size of its development team, which I imagine would be hard for it to justify. GE is initially focusing the new product specifically on North American mid-size electric utilities, so migrating to the new product will not be an option yet for customers who do not fall into that category, which is a fairly large majority.

For those customers who are in a position to consider migration (the North American mid-size electric utilities - and presumably the segments addressed will expand over time), GE will face the classic challenges of any company going through a major technology upgrade (and which Intergraph went through - and is still going through - with the migration from its older FRAMME product to G/Technology, and ESRI from Arc/Info 7 to ArcGIS). One challenge is that with a product like Smallworld that is 16 years old and exceptionally customizable, most customers have very rich functionality which it is hard to replace with a product that has only been under development for a year or two. There will either be custom development necessary to replace custom functionality in the old system, or the customer will need to be persuaded to give up some existing functionality to get other benefits that can be obtained from the new architecture. This generally means that the migration from the old system to the new system is a large enough project that most organizations will take the opportunity to evaluate other systems on the market and decide whether to stay with the same vendor or switch to a new one. There has been little turnover in the utility market in recent years, so on the rare occasions that utilities do choose a new system, all the vendors are very hungry for those opportunities and pricing tends to be very competitive, especially since the three major systems are not highly differentiated these days. As I said, none of this part is unique to GE, it is the same situation that ESRI and Intergraph have gone through as they have been migrating their customers to newer technologies.

One other challenge GE may have is with the customers who are not yet addressed by the new product. It will presumably take several years before the new product is an option for all of them (I believe that GE is saying that they will have a beta out sometime this year, then a first release first half of next year, and unless it's different from all other software products it will probably need a second release before it's really ready for serious use, so suppose that arrives late 2008 or early 2009, then probably the product is not going to be expanding to substantial additional market segments until 2009 or 2010). Now by and large I think that the Smallworld customer base is still pretty happy, so maybe customers will be willing to continue, assuming that GE continues to invest in Smallworld as it says it will. But there is also the risk that some customers will decide that this all means that the writing is on the wall for the Smallworld products, even if they keep going for a few more years, so maybe they should just go out and look at Intergraph and ESRI, who are running on more mainstream architectures which are in production today.

One other challenge for GE as they try to address moving their larger customers to the new platform at some point in the future (assuming they do) will be how to provide the same level of scalability that Smallworld VMDS does - perhaps that's a topic for a future discussion.

So anyway, it will be interesting to see how all this pans out over the next few years. I wish the GE guys good luck with it - as I said before, the industry can use additional innovation and competition!

Thoughts on GE's next generation system based on Oracle - part 1

A number of people have asked about my thoughts on GE's announcement back in March about a next generation system based on top of Oracle technology (note that in this post, GE stands for General Electric, not Google Earth!). This was covered by Joe Francica at All Points Blog under the title "Cutting out the GIS Middleman", by Susan Smith at GISCafe in an interview with Robert Laudati, and in an article by GE. Here are my thoughts ... (warning - this is a rather long post, in fact it was getting so long I have decided to split it into two parts, and the first part is still very long!).

I'll give a quick bit of history on Smallworld for all you "neo" people and those who haven't had any involvement with GIS in the utility industry. Smallworld is where I worked from 1992 to 2002, and during that time we grew from being a small startup in the UK to the global market leader in GIS for utilities and communications (according to Daratech), with revenues of around $100m. Smallworld was bought by General Electric in 2000. Smallworld introduced some radical new ideas in the early 1990s, many of which have now become common practice across the industry - I will talk more about some of these in the future, but Charlie Savage gives a good summary of his perspective here. In the 1990s, Smallworld had a clear technical lead in the utility industry, but in the early 2000s both ESRI and Intergraph introduced new systems (ArcInfo 8.0, now ArcGIS, and G/Technology), and the playing field is now much more even, with no clear leader, in my opinion. The Smallworld product remains very robust and scalable and has very rich functionality, but increasingly GE has been suffering in new sales situations because of the fact that Smallworld is based on its own proprietary language (Magik) and database (VMDS, Version Managed Datastore), while its two primary competitors have more modern and mainstream software architectures. Actually both Magik and VMDS still have some great technical strengths, especially the latter, but as geospatial technology has moved more into the mainstream, it has become increasingly hard to convince the market of the benefits of buying a "proprietary" solution, and in my opinion this has been the primary driver for GE to develop this new product set on top of Oracle.

The most obvious interesting thing about this announcement, which others have commented on also, is that it really reinforces the notion that geospatial technology is becoming absorbed into mainstream IT - many people have talked about this for a while, myself included. The way it was explained to me by someone from GE is that when they first sat down with Oracle to discuss collaboration on this project, they thought that there would be three layers of software: Oracle technology at the back end, a new "GIS" layer in the middle, and specific utility applications built on top of that. But when they looked at what Oracle now offers, including a map viewer, network model, version management (workspace management in Oracle terminology), etc, they concluded that there were really just two layers: Oracle, and the specific utility applications. I don't think this point is especially interesting to customers, actually: whether you regard the solution as 2 tiers or 3 tiers is a bit of an arbitrary distinction - in either case you need software from two vendors, Oracle and GE, and whether specific bits of functionality come from one or the other is not really significant. If GE reduces the cost of their software because they don't have to develop as much functionality themselves, then customers will be interested, but I haven't heard any discussion about that!

The reason this is interesting is from a general industry perspective, because it suggests that it will be harder to have a successful business which focuses purely on the middle "GIS" layer, without delivering applications on top of that which solve specific business problems. There are now many free or cheap options for drawing a map on a screen; you no longer need a specialized and expensive piece of software to do that. Companies like GE and Intergraph sell both geospatial platform software and vertical industry applications, and both are now putting more emphasis on the vertical applications where (in the right areas), they can still show high business value and therefore justify relatively high prices, and less emphasis on the basic geospatial capabilities, many of which are becoming commoditized. ESRI has of course focused heavily on delivering a horizontal GIS product, and has a large partner ecosystem which provides vertical applications on top of this. They have a very dominant position in this horizontal GIS space, and the fact that many of their traditional competitors are focusing more on vertical applications may mean that they can increase their hold there - but nevertheless they are seeing significant pressure on various parts of this space from Oracle, Google, Microsoft, Yahoo and open source solutions. So it will be interesting to see whether ESRI keeps its horizontal focus or also starts to move into more vertical solutions over time. There would be some challenges in this strategy, in particular the issue of potentially competing with its partners - though this is a common problem for platform software companies and is in many ways a natural evolution that many companies go through. We went through this at Smallworld, starting as a platform company with partners and then moving more into vertical applications, and Oracle is experiencing this also, especially with several of its recent acquisitions (more on that in a future post).

OK, so does this new architecture give GE a competitive advantage? Not really, in my opinion. The first major argument in favor of it is that all your data is stored in a standard relational database, which helps with integration, administration, security, etc (see my 1990 article which outlines these benefits - these concepts are not new!). The second major argument is that you can use standard development environments (Java-based, in this case). So GE addresses the concerns that the market today has about its existing "proprietary" solution - but both ESRI and Intergraph have provided solutions based on mainstream databases and development environments for a number of years, so these things will not be differentiators for GE - they are playing catch up in this regard. Oracle likes the fact that the solution is based purely on the Oracle stack - and so customers who have committed to the whole Oracle stack will also see that as an advantage. But on the other hand, customers focused on a Microsoft architecture either on the client or middle tier will see this approach as a disadvantage - and my general feeling is that more utilities probably fall into this category. Those who are not too religious about their IT strategy (probably the majority) will focus on the functionality provided by the main three vendors more than the system architecture.

So, in summary on part 1: I am pleased to see GE make this announcement, as I had personally pretty much written off the possibility of them investing in a "next generation" system. I am happy for the friends I still have at GE that there is investment in the future, and I think it will be good for the industry if GE can make this new product into a strong competitor to ESRI and Intergraph in the utility market - and if they can bring forward the strengths of the current Smallworld system then it will be. The announcement is interesting because it shows that you can now develop complex geospatial applications directly on top of Oracle without needing a traditional "GIS". While I think this is of somewhat academic interest to most customers, it is more significant in terms of what vendors in the geospatial industry will look like in future. I personally don't think that the market will see the new system architecture as a competitive advantage, except in situations where organizations have very strong Oracle religion - I think it will be seen as more of a catch up exercise by most people, which negates the perceived weakness of Smallworld's current "proprietary" architecture.

In part 2 I will talk about the real reason why this product could be a significant jump forward for the utility industry, which really hasn't been highlighted in the GE announcements or in any of the commentary I've seen. And I'll talk about why this same factor could be the reason that the product fails. And last but not least, I'll talk about some of the challenges which GE faces in positioning the new product with regard to the existing Smallworld products.