Member Article

An introduction to enterprise search

Iain Fletcher, VP marketing, Search Technologies, talks ’enterprise search’. This is the first of a series of articles which together aim to provide a solid understanding of enterprise search engines, what to expect of them and to how maximise their effectiveness.

Initially, we will cover the basic principles of search:

Physical plumbing

The term ‘enterprise search’ is used to describe search engines which operate on specific websites, company intranets or embedded in applications such as Content Management Systems. In this wider sense, the phrase differentiates this class of search engine from the ubiquitous Google, Bing and Yahoo.

Enterprise search in its original sense of providing search over a range of enterprise data sources can become complex. This is because of the need to interface with multiple, often diverse systems to acquire data for indexing, at the same time ensuring that document level security is fully respected. This physical plumbing aspect of enterprise search has confounded many projects over the years. However, the necessary technology is available today and, as with most potentially complex tasks, a little experience and expertise is all that is needed for success.

The other main physical components of a search system are the search indexes themselves and a user interface for submitting queries and receiving results.

This series of articles will focus primarily on aspects of Enterprise Search which are generally less well understood than the physical plumbing and which are likely to cause issues or disappointment most often if they are not understood and addressed.

The process of search

First, let’s consider the fundamental process of using a search engine and then, within that framework, plot the evolution of a modern enterprise search system. This process has three stages:

• Create and submit a search clue

• The engine matches that against available documents, and returns results

• Browse the results to find the desired information

Although a massive amount of R&D investment has been made in search technology over the years by some of the world’s largest IT companies, this basic process has stayed the same. Further, it is a useful framework upon which to position all of the important innovations in search.

Data growth is a key issue

Double the size of your haystack and on average it will take you twice as long to find a specific needle. This first principle of search applies equally to enterprise search as is illustrated by this long-standing enterprise search industry graph. Technological innovation won’t change this, although some vendors occasionally pretend otherwise.

Most of the important innovations in search technology have been aimed at mitigating the effect of data growth, which continues to accelerate rapidly. Many disappointments amongst search users, most of whom - especially in large organisations - do not feel well served by enterprise search, are a consequence of data growth. We will explore what can be done to change this.

Diverse user needs & varied data sets

Some search systems must serve a wide range of user needs, from simple fact-checking by administrative staff to complex research enquiries made by specialists. Others serve a narrower audience with common and easily-defined interests.

Some data sets can be highly normalised and consistent, for example in eCommerce applications. Others can be extremely varied in length, quality, format, language and structure.

The search engine has the task of matching user needs to the data being searched. As the variances above suggest, how easy or challenging this will be is very application specific.

Key innovations

The next article will describe the major innovations in search engine technology over the past 10-15 years and how these help to address these fundamental challenges.

This was posted in Bdaily's Members' News section by Iain Fletcher .