Lee Romero

Archive for October, 2008

I have previously described what I termed the various generations of solutions to the common challenge of workers finding connecting with or finding co-workers within an enterprise.  My most recent post described the fourth generation solution – which enables users to search and connect using much more than simple administrative terms (name, email, address, etc.) for the search.

Over my next couple of posts, I will provide a write-up of a proof of concept implementation I’ve assembled that meets a lot of the need for this with what I believe to be relatively minimal investment.

The follow represent the goals I’ve set for myself in this proof of concept:

1. Demonstrate the usefulness of a people search based on attributes of workers other than purely administrative data – things like their skills, competencies, work, interests, etc.
2. Demonstrate the feasibility of discerning the skills, competencies, work and/or interests through a means that does not depend on maintenance of data by the worker (which, from my experience, is not long-term maintainable).
   1. More specifically, provide a test bed to explore different algorithms for discovering keywords important keywords for people.
3. Demonstrate the feasibility of discovering keywords using only data that is generally “publicly visible” within an enterprise.
4. Provide a path for integrating manually-maintained skills data (if that were to be collected), or any other data (possibly including tags applied by co-workers as seen in IBM’s Dog Ear project).
5. Provide a compelling user experience that draws people in and gives people a visual presentation of what another person is “about” (what describes them).
6. Provide a solution that provides, at minimum, the equivalent of a 3rd generation solution (in other words you can find that worker based on their administrative data).

Also, I wanted to say that part of the inspiration for this proof of concept came from a session I attended at Enterprise Search Summit 2007 as presented by Trent Parkhill.  In his session, he described a mechanism where submissions to a company’s repository would be tagged with the names of participants in the project that produced the document as a deliverable.  Then, when users were searching for content, there was a secondary search that produced a list of people associated with the terms and / or documents found by the user’s search.  I’ve kind of turned that around and treated the people as being tagged by the keywords of the items they produce.

In my next post, I will describe the overall design of my proof of concept.

I just read through Kas Thomas’ post In search of a standard search syntax, and have to agree this would be useful for users of search engines.

However, I would go even further and suggest that the search industry (enterprise search as well as internet search engines) would also benefit if it were to define and adopt a standard response syntax for results (at least a response syntax that could be provided as an option).  Obviously, for most users a straightforward HTML presentation is desirable as when they interact with an engine through their browser, they want to be able to view the results in their browser.

However, an ability to request results from an arbitrary engine in a standard format would be a great step forward – it would vastly simplify aggregation of results for federated search and more generally it could present the ability to programmatically interact with multiple engines for a variety of other purposes.

I know of one attempt that seems to drive to this – OpenSearch (which is associated with A9 – Amazon’s search engine) – a set of elements that can be used as extensions to an RSS format.  Are there others?  How widely known (and adopted?) is OpenSearch as a format?

Or, in other words, “How do you apply the application standards to improve findability to applications built by third-party providers who do not follow your standards?”

I’ve previously written about the standards I’ve put together for (web-based) applications that help ensure good findability for content / data within that application. These standards are generally relatively easy to apply to custom applications (though it can still be challenging to get involved with the design and development of those applications at the right time to keep the time investment minimal, as I’ve also previously written about).

However, it can be particularly challenging to apply these standards to third-party applications – For example, your CRM application, your learning management system, or your HR system, etc. Applying the existing standards could take a couple of different forms:

1. Ideally, when your organization goes through the selection process for such an application, your application standards are explicitly included in the selection criteria and used to ensure you select a solution that will conform to your standards
2. More commonly, you will identify compliance to the standards (perhaps during selection but perhaps later during implementation) and you might need to implement some type of customization within the application to provide compliance.
3. Hopefully, you identify compliance to the standards during selection or later, but you find you can not customize the application and you need a different solution.

The rest of this post will discuss a solution for option #3 above – how you can implement a different solution. Note that some search engines will provide pre-built functionality to enable search within many of the more common third party solutions – those are great and useful, but what I will present here is a solution that can be implemented independent of the search engine (as long as the search engine has a crawler-based indexing function) and which is relatively minimal in investment.

Solving the third-party application conundrum for Enterprise Search

So, you have a third party application and, for whatever reason, it does not adhere to your application standards for findability. Perhaps it fails the coverage principle and it’s not possible to adequate find the useful content without getting many, many useless items; or perhaps it’s the identity principle and, while you can find all of the desirable targets, they have redundant titles; or it might even be that the application fails the relevance principle and you can index the high value targets and they show up with good names in results but they do not show up as relevant for keywords which you would expect. Likely, it’s a combination of all three of these issues.

The core idea in this solution is that you will need a helper application that creates what I call “shadow pages” of the high value targets you want to include in your enterprise search.

Note: I adopted the use of the term “shadow page” based on some informal discussions with co-workers on this topic – I am aware that others use this term in similar ways (though I don’t think it means the exact same thing) and also am aware that some search engines address what they call shadow domains and discourage their inclusion in their search results. If there is a preferred term for the idea described here – please let me know!

What is a shadow page? For my purposes here, I define a shadow page as:

• A page which uniquely corresponds to a single desirable search target;
• A page that has a distinct, unique URL;
• A page that has a <title> and description that reflects the search target of which it is a shadow, and that title is distinct and provides a searcher who sees it in a search results page with insight about what the item is;
• A page that has good metadata (keywords or other fields) that describe the target using terminology a searcher would use;
• A page which contains text (likely hidden) that also reflects all of the above as well to enhance relevance for the words in the title, keywords, etc.;
• A page which, when accessed, will automatically redirect a user to the page of which the page is a shadow.

To make this solution work, there are a couple of minimal assumptions of the application. A caveat: I recognize that, while I consider these as relatively simple assumptions, it is very likely that some applications will still not be able to meet these and so not be able to be exposed via your enterprise search with this type of solution.

1. Each desirable search target must be addressable by a unique URL;
2. It should be possible to define a query which will give you a list of the desirable targets in the application; this query could be an SQL query run against a database or possible a web services method call that returns a result in XML (or probably other formats but these are the most common in my experience);
3. Given the identity (say, a primary key if you’re using a SQL database of some type) of a desirable search target, you must be able to also query the application for additional information about the search target.

Building a Shadow Page

Given the description of a shadow page and the assumptions about what is necessary to support it, it is probably obvious how they are used and how they are constructed, but here’s a description:

First – you would use the query that gives you a list of targets (item #2 from the assumptions) from your source application to generate an index page which you can give your indexer as a starting point.  This index page would have one link on it for each desirable target’s shadow page.  This index page would also have “robots” <meta> tags of “noindex,follow” to ensure that the index page itself is not included as a potential target.

Second – The shadow page for each target (which the crawler reaches thanks to the index page) is dynamically built from the query of the application given the identity of the desirable search target (item #3 from the assumptions).  The business rules defining how the desirable target should behave in search help define the necessary query, but the query would need to contain at minimum some of the following data: the name of the target, a description or summary of the target, some keywords that describe the target, a value which will help define the true URL of the actual target (per assumption #1, there must be a way to directly address each target).

The shadow page would be built something like the following:

• The <title> tag would be the name of the target from the query (perhaps plus an application name to provide context)
• The “description” <meta> tag would be the description or summary of the target from the query, perhaps plus a few static keywords that help ensure the presence of additional insight about the target.   For example, if the target represents a learning activity, the additional static text might indicate that.
• The “keywords” <meta> tag would include the keywords from the query, plus some static keywords to ensure good coverage.  To follow the previous example, it might be appropriate to include words like “learning”, “training”, “class”, etc. in a target that is a learning activity to ensure that, if the keywords for the specific target do not include those words, searchers can still find the shadow page target in search.
• The <body> of the page can be built to include all of the above text – from my experience, wrapping the body in a CSS style that visually hides the text keeps the text from actually appearing in a browser.
• Lastly, the shadow page has a bit of JavaScript in it that redirects a browser to the actual target – this is why you need to have the target addressable via a URL and also that the query needs to provide the information necessary to create that URL.  Most engines (I know of none) will not be able to execute the JavaScript, so will not know that the page is really a redirect to the desired target.

The overall effect of this is that the search engine will index the shadow page, which has been constructed to ensure good adherence to the principles of enterprise search, and to a searcher, it will behave like a good search target but when the user clicks on it from a search result, the user ends up looking at the actual desired target.  The only clue the user might have is that the URL of the target in the search results is not what they end up looking at in their browser’s address bar.

The following provides a simple example of the source (in HTML – sorry for those who might not be able to read it) for a shadow page (the parts that change from page to page are in bold):

<html>
<head>
<TITLE>title of target</TITLE>
<meta name="robots" content="index, nofollow">
<meta name="keywords" content="keywords for target">
<meta name="description" content="description of target">
<script type="text/javascript">
document.location.href="URL of actual target";
</script>
</head>

<body>
<div style="display:none;">
<h1>title of target</h1>
description of target and keywords of target
</div>
</body>
</html>

Advantages of this Solution

A few things that are immediately obvious advantages of this approach:

1. First and foremost, with this approach, you can provide searchers with the ability to find content which otherwise would be locked away and not available via your enterprise search!
2. You can easily control the targets that are available via your enterprise search within the application (potentially much easier than trying to figure out the right combination of robots tags or inclusion / exclusion settings for your indexer).
3. You can very tightly control how a target looks to the search engine (including integration with your taxonomy to provide elaborated keywords, synonyms, etc)

Problems with this Solution

There are also a number of issues that I need to highlight with this approach – unfortunately, it’s not perfect!

1. The most obvious issue is that this depends on the ability to query for a set of targets against a database or web service of some sort.
   1. Most applications will be technically able to support this, but in many organizations, this could present too great a risk from a data security perspective (the judicious use of database views and proper management of read rights on the database should solve this, however!)
   2. This potentially creates too high a level of dependence between your search solution and the inner workings of the application – an upgrade of the application could change the data schema enough to break this approach.  Again, I think that the use of database views can solve this (by abstracting away the details of the implementation into a single view which can be changed as necessary through any upgrade).
2. Some applications may simply not offer a “deep linking” ability into high value content – there is no way to uniquely address a content item without the context of the application.  This solution can not be applied to such applications.  (Though my opinion is that such applications are poorly designed, but that’s another matter entirely!)
3. This solution depends on JavaScript to forward the user from the shadow page to the actual target.  If your user population has a large percentage of people who do not use JavaScript, this solution fails them utterly.
4. This solution depends on your search engine not following the JavaScript or somehow otherwise determining that the shadow page is a very low quality target (perhaps by examining the styles on the text and determining the text is not visible).  If you have a search engine that is this smart, hopefully you have a way to configure it to ignore this for at least some areas or page types.
5. Another major issue is that this solution largely circumvents a search engine’s built in ability to do item-by-item security as the target to the search engine is the shadow page.  I think the key here is to not use this solution for content that requires this level of security.

Conclusion

There you have it – a solution to the exposure of your high value targets from your enterprise applications that is independent of your search engine and can provide you (the search administrator) with a good level of control over how content appears to your search engine, while ensuring that what is included highly adheres to my principles of enterprise search.

I’ve previously written about the three principles of enterprise search and also about the specific business process challenges I’ve run into again and again with web applications in terms of findability.

Here, I will provide some insights on the specific standards I’ve established to improve findability, primarily within web applications.

As you might expect, these standards map closely to the three principles of enterprise search and so that’s how I will discuss them.

Coverage

When an application is being specified, the application team must ensure that they discuss the following question with business users – What are the business objects within this application and which of those should be visible through enterprise search?

The first question is pretty standard and likely forms the basis for any kind of UML or entity relationship diagram that would be part of a design process for the application. The second part is often not asked but it forms the basis for what will eventually be the specific targets that will show in search results through the enterprise search.

Given the identification of which objects should be visible in search results, you can then easily start to plan out how they might show up, how the search engine will encounter them, whether the application might best provide a dynamic index page of links to the entities or support a standard crawl or perhaps even a direct index of the database(s) behind the application.

Basically, the standard here is that the application must provide a means to ensure that a search engine can find all of the objects that need to be visible and also to ensure that the search engine does not include things that it should not.

Some specific things that are included here:

• The entities that need to show up in search results should be visible as an individual target, addressable via a unique and stable URL. This ensures that when an item shows up in a set of search results, a searcher will see an entity that looks and behaves like what they want – if they’re looking for a document, they see that document and not a page that links to that document.
• The application should have a strategy for the implementation of “robots” meta tags – pages that should not be indexed should have a “noindex”. Pages that are navigational (and not destinations themselves for search) should be marked “noindex”. Pages that provide navigation to the items through various options (filters, sorting, etc) may need to have “nofollow” as well as so that a crawler does not get hung up in looking at multitudes of various pages all of which are marked “noindex” anyway.
• The application should not be frame-based. This is a more general standard for web applications, but frame-based applications consistently cause problems for crawlers as a crawler will index the individual frames but those individual frames are not ,themselves, useful targets.
• To simplify things for a search engine, an application can simply provide an index page that directly links to all desired objects that should show up in search; I’ve found this to be very useful and can be much simpler than working through the logic of a strategy for robots tags to ensure good coverage. This index page would be marked “noindex, follow” for its robot tags so that it, itself, is not indexed (otherwise it might show up as a potential result for a lot of searches if, say, the title of the items are included in this index page).
• Note that it is possible that for some applications, the answer to the leading question for this may be that nothing within the application is intended to be found via an enterprise search solution. That might be the case if the application provides its own local search function and there is no value in higher visibility (or possibly if the cost of that higher visibility is too high – say in the case that the application provides sophisticated access control which might be hard to translate to an enterprise solution).

Identity

With the standard for Coverage defined, we can be comfortable with knowing that the right things are going to show in search and the wrong things will not show up. How useful will they be as search results, though? If a searcher sees an item in a results list, will they be able to know that it’s what they’re looking for? So we need to ensure that the application addresses the identity principle.

The standard here is that the pages (ASP pages, JSP files, etc) that comprise the desirable targets for search must be designed to address the identity principle – specifically:

• Each page that shows a search target must dynamically generate a <title> tag that clearly describes what it shows.
• An application should also adopt a standard for how it identifies where the content / data is (the application name perhaps) as well as the content-specific name.
• Within our infrastructure, a standard like, “<application name>: <item name>” has worked well.
• In addition, each page that shows a search target must dynamically generate a “description” <meta> tag. This description can (and for our search does) be used as part of the results snippet displayed in a search results page, so it can provide a searcher important clues before the searcher even clicks on a target.
• The application team should develop a strategy for what to include in the “description”:
  • In many applications, each item of interest will typically have some kind of user-entered text that can be interpreted as a description or which could be combined with some static text to make it so.
  • For example, an entity might have a name (used in the <title> tag) and something referred to as a the “summary” or “subject” or maybe “description” – simply use that text.
  • Alternately, the “description” might be generated as something like, “The help desk ticket <ticket ID> named <ticket name>”, for a page that might be part of a help desk ticket application.

Relevance

Now we know that the search includes what it should and we also know that when those items show in search, they will be identifiable for what they are. How do we ensure that the items show up in search for searches for which they are relevant, though?

The standards to address the relevance issue are:

• Follow the standard above for titles (the words in the <title> tag will normally significantly boost relevancy for searches on those words regardless of your search engine)
• Each page that shows a search target must dynamically generate a “keywords” <meta> tag.
• The application team should devise a strategy for what would be included in the keywords, though some common concepts emerge:
  • Any field that a user can assign to the entity would be a candidate – for example, if a user can select a Product with which an item is associated or a geography, an industry, etc. All of those terms are good candidates for inclusion in keywords
  • While redundant, simply using the title of the item in the keywords can be useful (and reinforce the relevance of those words)
  • If an application integrates with a taxonomy system (specifically, a thesaurus) any taxonomic tags assigned to an entity should be included.
  • In addition, for a thesaurus, if the content will be indexed by internet search engines, directly including synonyms for taxonomic terms in the keywords can sometimes help – you might also include those synonyms directly in your own search engine’s configuration but you can’t do that with a search engine you don’t control. (Many internet search engines no longer consider the contents of these tags due to spamming in them but these can’t hurt even then.)
• The application may also generate additional <meta> tags that are specific to its needs. When integrated with a taxonomy that has defined facets, including a <meta> tag with the name for each facet and the assigned values can improve results.
  • For example, if the application allows assignment of a product, it can generate a tag like: <meta name=”product” contents=”<selected values>”/>
  • Some search engines will allow searching within named fields like this – providing you a combination of a full text search and fielded search ability.

Additional resources

For a good review of the <meta> tags in HTML pages, you can look at:

• Metadata fundamentals for intranets and websites
• Preparing a Site for Searching

So we get to the exciting conclusion of my essays on the inclusion of employees in enterprise search. If you’ve read this far, you know how I have characters the first and second generation solutions and also provided a description of a third generation solution (which included some details on how we implemented it).

Here I will describe what I think of as a fourth generation solution to people finding within the enterprise. As I mentioned in the description of the third generation solution, one major omission still at this point is that the only types of searches with which you can find people is through administrative information – things like their name, address, phone number, user ID, email, etc.

This is useful when you have an idea of the person you’re looking for or at least the organization in which they might work. What do you do when you don’t know the person and may not even know the organization in which they work? You might know the particular skills or competencies they have but that may be it. This problem is particularly problematic in larger organizations or organizations that are physically very distributed.

The core idea with this type of solution is to provide the ability to find and work with people based on aspects beyond the administrative – the skills of the people, their interests, perhaps the network of people with which they interact, and more. While this might be a simplification, I think of this as expertise location, though that, perhaps, most cleanly fits into the first use case described below.

Some common use cases for this type of capability include:

• Peer-to-peer connections – an employee is trying to solve a particular problem and they suspect someone in the company may have some skills that would enable them to solve the problem more quickly. Searching using those skills as keywords would enable them to directly contact relevant employees.
• Resource planning – a consulting organization needs to staff a particular project and needs to find specific people with a particular skill set.
• Skill assessment – an organization needs to be able to ascertain the overall competency of their employees in particular skill sets to identify potential training programs to make available.

This capability is something that has often been discussed and requested at my current employer, but which no one has really been willing to sponsor. That being said, I know there are several vendors with solutions in this space, including (at least – please share if you know of others):

• Connectbeam – A company I first found out about at KM World 2007. They had some interesting technology on display that combines expertise location with the ability to visualize and explore social networks based on that expertise. Their product could digest content from a number of systems to automatically discern expertise.
• ActiveNet – A product from Tacit Software, which (at a high level) is similar to Connectbeam. An interesting twist to this product is that it leaves the individuals whose expertise are managed in the system in control of how visible they are to others. In the discussions I’ve had with this company about the product, I’ve always had the impression that, in part, this provides a kind of virtual mailing list functionality where you can contact others (those with the necessary expertise) by sending an email without knowing who it’s going to. Those who receive it can either act on it or not and, as the sender, you only know who replies.
• Another product about which I only know a bit is from a company named Trampoline Systems. I heard about them as I was doing some research on how to tune a prototype system of my own and understand that their Sonar platform provides similar functionality.
• [Edit: Added this on 03 November, 2008] I have also found that Recommind provides expertise location functionality – you can read more about it here.
• [Edit: Added this on 03 November, 2008] I also understand that the Inquira search product provides expertise location, though it’s not entirely clear to me from what I can find about this tool how it does this.

A common aspect of these is that they attempt to (and perhaps succeed) in automating the process of expertise discovery. I’ve seen systems where an employee has to maintain their own skill set and the problem with these is that the business process to maintain the data does not seem to really embed itself into a company – inevitably, the data gets out of date and is ill-maintained and so the system does not work.

I can not vouch for the accuracy of these systems but I firmly believe that if people search in the enterprise is going to meet the promise of enabling people to find each other and connect based on of-the-moment needs (skills, interests, areas of work, etc), it will be based on this type of capability – automatically discovering those aspects of a worker based on their work products, their project teams, their work assignments, etc.

I imagine within the not too distant future, as we see more merger of the “web 2.0” functionality into the enterprise this type of capability will become expected and welcome – it will be exciting to see how people will work together then.

This brings to a close my discussion of the various types of people search within the enterprise. I hope you’ve found this of interest. Please feel free to let me know if you think I have any omissions or misstatements in here – I’m happy to correct and/or fill in.

I plan another few posts that discuss a proof of concept I have put together based around the ideas of this fourth generation solution – look for those soon!

In my last post, I wrote about what I termed the first generation and second generation solution to people search in enterprise. This time, I will describe what I call a “third generation” solution to the problem that will integration people search with your enterprise search solution.

This is the stage of people search in use within my current employer’s enterprise.

What is the third generation?

What I refer to as a third generation solution for people search is one where an employee’s profile (their directory entry, i.e., the set of information about a particular employee) becomes a viable and useful target within your enterprise search solution. That is, when a user performs a search using the pervasive “search box” (you do have one, right?), they should be able to expect to find their fellow workers in the results (obviously, depending on the particular terms used to do the search) along with any content that matches that.

You remove the need for a searcher to know they need to look in another place (another application, i.e., the company’s yellow pages) and, instead, reinforce the primacy of that single search experience that brings everything together that a worker needs to do their job.

You also offer the full power of your enterprise search engine:

• Full text search – no need to specifically search within a field, though most engines will offer a way to support that as well if you want to ffer that as an option;
• The power of the search engine to work on multi-word searches to boost relevancy – so a search on just a last name might include a worker’s profile in the search results but one that includes both a first and last name (or user ID or location or other keywords that might appear in the worker’s profile) likely ensures that the person shows in the first page of results amidst other content that match;
• The power of synonyms – so you can define synonyms for names in your engine and get matches for “Rob Smith” when a user searches on “Robert Smith” or “Bob Smith”;
• Spelling corrections – Your engine likely has this functionality, so it can automatically offer up corrections if someone misspells a name, even.

Below, you will find a discussion of the implementation process we used and the problems we encountered. It might be of use to you if you attempt this type of thing.

Before getting to that, though, I would like to discuss what I believe to be remaining issue with a third generation solution in order to set up my follow-up post on this topic, which will describe additional ideas to solving the “people finder” problem within an enterprise.

The primary issue with the current solution we have (or any similar solution based strictly on information from a corporate directory) is that the profile of a worker consists only of administrative information. That is, you can find someone based on their name, title, department, address, email, etc., etc., etc., but you can not do anything useful to find someone based on much more useful attributes – what they actually do, what their skills or competencies are or what their interests might be. More on this topic in my next post!

The implementation of our third generation solution (read on for the gory details)

Read on from here for some insights on the challenges we faced in our implementation of this solution. It gets pretty detailed from here on out, so you’ve been warned!

(more…)

This post is the first of a brief series of posts I plan to write about the integration of “people search” (employee directory) with your enterprise search solution. In a sense, this treats “people” as just another piece of content within your search, though they represent a very valuable type of content.

This post will be an introduction and describe both a first and second generation solution to this problem. In subsequent posts, I plan to describe a solution that takes this solution forward one step (simplifying things for your users among other things) and then into some research that I believe shows a lot of promise and which you might be able to take advantage of within your own enterprise search solution.

Why People Search?

Finding contact information for your co-workers is such a common need that people have, forever, maintained phone lists – commonly just as word processing documents or spreadsheets – and also org charts, probably in a presentation file format of some type. I think of this approach as a first generation solution to the people search problem.

Its challenges are numerous, including:

1. The maintenance of the document is fraught with the typical issues of maintaining any document (versioning, availability, etc.)
2. In even a moderately large organization, the phone list may need to be updated by several people throughout the organization to keep it current.
3. Search within this kind of phone list is limited – you can ensure you always have the latest version and then open it up and use your word processor’s search function or (I remember this well, myself) always keep a printout of the latest version of the phone list next to your workspace so you can look through it when you need to contact someone.

As computer technology has evolved and companies implemented corporate directories for authentication purposes (Active Directory, LDAP, eDirectory, etc.), it has become common to maintain your phone book as a purely online system based on your corporate directory. What does such a solution look like and what are its challenges?

A “Second Generation” Solution

I think it’s quite common now that companies will have an online (available via their intranet) employee directory that you can search using some (local, specific to the directory) search tools. Obvious things like doing fielded searches on name, title, phone number, etc. My current employer has sold a product named eGuide for quite some time that provides exactly this type of capability.

eGuide is basically a web interface for exposing parts of your corporate Directory for search and also for viewing the org chart of a company (as reflected in the Directory).

We have had this implemented on our intranet for many years now. It has been (and continues to be) one of the more commonly used applications on our intranet.

The problems with this second generation solution, though, triggered me to try to provide a better solution a few years ago using our enterprise search. What are the problems with this approach? Here are the issues that triggered a different (better?) solution:

1. First and foremost, with nothing more than the employee finder as a separate place to search, you immediately force a searcher to have to make a decision before they do their search as to where they want to search. Many users might expect that the “enterprise” search actually does include anything that they can navigate to as potential targets so when they search on a person’s name and don’t see it in the result set they immediately think either A) why does the search not include individual people’s information, or B) this search engine is so bad that, even though it must include people information, it can’t even show the result at a high enough relevance to get it on the first page!
   1. Despite my statement to the contrary above, I am aware that Jakob Nielsen does actually advocate the presence of both a “people search” box and a more general search box because people are aware of the distinction between searching for content and search for people. We do still have both search boxes on our intranet, though, in a sense, the people search box is redundant.
2. Secondly, the corporate directory commonly is a purely fielded search – you have to select which field(s) you want to search in and then you are restricted to searching just those fields.
   1. In other words, you as a searcher, need to know in which field a particular string (or partial string) might appear. For many fields, this might not be an issue – generally, first and last name are clear (though not always), email, phone number, etc., but the challenge is that a user has to decide in which field they want to look.
3. Third, related to the previous point, directory searches are generally simplistic searches based on string matching or partial string matching. With a full search engine, you introduce the possibility of taking advantage of synonyms (especially useful on first names), doing spelling corrections, etc.

So there’s a brief description of what I would characterize as a first generation solution and a second generation solution along with highlights of some issues with each.

Up next, I’ll describe the next step forward in the solution to this issue – integrating people into your enterprise search solution.

The title of this post – “People know where to find that, though!” is a very common phrase I hear as the search analyst and the primary search advocate at my company. Another version would be, “Why would someone expect to find that in our enterprise search?”

Why do I hear this so often? I assume that many organizations, like my own, have many custom web applications available on their intranet and even their public site. It is because of that prevalence, combined with a lack of communication between the Business and the Application team, that I hear these phrases so often.

I have (unfortunately!) lost count of the number of times a new web-based application goes into production without anyone even considering the findability of the application and its content (data) within the context of our enterprise search.

Typically, the conversation seems to go something like this:

• Business: “We need an application that does X, Y and Z and is available on our web site.”
• Application team: “OK – let’s get the requirements laid out and build the site. You need it to do X, Y and Z. So we will build a web application that has page archetypes A, B and C.”
• Application team then builds the application, probably building in some kind of local search function – so that someone can find data once they are within the application.
• The Business accepts the usability of the application and it goes into production

What did we completely miss in this discussion? Well, no one in the above process (unfortunately) has explicitly asked the question, “Does the content (data) in this site need to be exposed via our enterprise search?” Nor has anyone even asked the more basic question, “Should someone be able to find this application [the “home page” of the application in the context of a web application] via the enterprise search?”

• Typically, the Business makes the (reasonable) assumption that goes something like, “Hey – I can find this application and navigate through its content via a web browser, so it will naturally work well with our enterprise search and I will easily be able to find it, right?!”
• On the other hand, the Application Team has likely made 2 assumptions: 1) the Business did not explicitly ask for any kind of visibility in the enterprise search solution, so they don’t expect that, and 2) they’ve (likely) provided a local search function, so that would be completely sufficient as a search.

I’ve seen this scenario play out many, many times in just the last few years here. What often happens next depends on the application but includes many of the following symptoms:

• The page archetypes designed by the Application Team will have the same (static) <title> tag in every instance of the page, regardless of the data displayed (generally, the data would be different based on query string parameters).
  • The effect? A web-crawler-based search engine (which we use) likely uses the <title> tag as an identifier for content and every instance of each page type has the same title, resulting in a whole lot of pretty useless (undifferentiated) search results. Yuck.
• The page archetypes have either no or maybe redundant other metadata – keywords, description, content-date, author, etc.
  • The effect? The crawler has no differentiation based on <titles> and no additional hints from metadata. That is, lousy relevance.
• The application has a variety of navigation or data manipulation capabilities (say, sorting data) based on standard HTML links.
  • The effect? The crawler happily follows all of the links – possibly (redundant) indexing the same data many, many times simply sorted on different columns.
  • Another effect? The dreaded calendar affect – the crawler will basically never stop finding new links because there’s always another page.
  • In either case, we see poor coverage of the content.

The overall effect is likely that the application does not work well with the enterprise search, or possibly that the application is that the application does not hold up to the pressure of the crawler hitting its pages much faster than anticipated (so I end up having to configure the crawler to avoid the application) and ending with yet another set of content that’s basically invisible in search.

Bringing this back around to the title – the response I often get when inquiring about a newly released application is something like, “People will know how to find that content – it’s in this application! Why would this need to be in the enterprise search?”

When I then ask, “Well, how do people know that they even need to navigate to or look in this application?” I’ll get a (virtual) shuffling of feet and shoulder shrugs.

All because of a perpetual lack of asking a few basic questions during a requirements gather stage of a project or (another way to look at it) lack of standards or policies which have “teeth” about the design and development of web application. The unfortunate thing is that, in my experience, if you ask the questions early, it’s typically on the scale of a few hours of a developer’s time to make the application work at least reasonably well with any crawler-based search engine. Unfortunately, because I often don’t find out about an application until after it’s in production, it then becomes a significant obstacle to get any changes made like this.

I’ll write more in a future post about the standards I have worked to establish (which are making some headway into adoption, finally!) to avoid this.

Edit: I’ve now posted the standards as mentioned above – you can find them in my post Standards to Improve Findability in Enterprise Applications.

Having written previously about my own principles of enterprise search and then some ideas on how to select a search engine, I thought it might be time to back up a bit and write about what I think of as “enterprise search”. Perhaps a bit basic or unnecessary but it gives some context to future posts.

The Enterprise in Enterprise Search

For me, the factors of a search solution that make it an enterprise solution include the following:

The user interface to access the solution is available to all employees of the company.

This has the following implications:

• Given today’s technologies, this probably means that it’s a web-based interface to access the search.
  • More generally, the interface needs to be easily made available across the enterprise. In any somewhat-large organization, that means something either available online or easily installed or accessed from a user’s workspace.
• I would also suggest that the search interface should be easily accessible from an employee’s standard workspace or a common starting point for employees.
  • One easy way to achieve this is to make access to an enterprise search solution part of the general intranet experience – especially on an intranet that shares a standard look-and-feel (and so, hopefully, a standard template). This is the ubiquitous “search box”.
  • Alternately, if users commonly use a specific application (say a CRM application or a collaboration tool), integrating the enterprise search into that is a better solution.
  • Lastly, it might be necessary to make access to the search solution “many-headed”. Meaning, it might be best to make it available through a number of means, including through a standard intranet search, a specialized client-based application and embedded in other, user-specific tools.
• Given the likely broad range of users who will use it, the search interface should be subject to very thorough usability design and testing.
• Adopting some of the standard conventions of a search experience are a good idea.

The content available through the solution covers all (relevant) content available to employees

This has the following implications:

• If your enterprise has a significant volume of web content, your enterprise search should index all of those web pages – either via a web crawling approach or via indexing the file system containing the files (if it’s all static).
• If your enterprise has a significant volume of content (data) in enterprise applications (CRM solution, HR system, etc.), you should have a strategy to determine which (if any) of the content from those systems would be included, how it will be included and how it will be presented in search results (potentially combined with content from many other systems in the same results page)
• If your enterprise has custom web applications (and what organization does not), you should expect to provide a set of standards for design and development of web applications to ensure good findability from them and also expect to have to monitor compliance with those.
• If your enterprise has significant content in collaboration tools (and who doesn’t – at least email!), you should have a strategy for including or not including that content. This could be very broad-ranging – email, SharePoint (and similar applications from companies like Interwoven, Open Text, Vignette, Novell, etc.), shared file systems, IM logs, and so on. At the very least, you need to consider the cost and value of including these types of content.
• If you have content repositories available to employees (a document management system (or systems!) or a records management system), again, you should consider the cost and value of including content from these in your enterprise search.
• While it is very useful to have a separate search for finding employees in a corporate directory, I believe that an enterprise search solution should include employees as a distinct “content type” and include them in standard search results page as well when relevant (e.g., searching on employee names, etc)
• Another major question regarding the content of your enterprise search is security. If you include all of that content in your search, how will you manage the security of the items? The two major options are early binding (building ACLs into the search) or late binding (checking security at search time). If you are not familiar with these, I would recommend you do a bit of internet searching on the topics as it’s very important to your solution. I’ve found some interesting articles on this topic.
  • In my mind, it’s also feasible to “punt” on security in a sense and work to ensure that your enterprise search solution includes everything that is generally accessible to your employee population but does not include anything with specific access control on it.
  • If you can achieve the effect of getting a user “close to” the content (ensuring some level of “information scent” shows up) but leaving it to the user to make the final step (through any application-specific access control) seems to work well.

The Search in Enterprise Search

The other half of your enterprise search solution will be the search engine itself. There are plenty (many!) options available with a variety of strengths and weaknesses. I think if you plan to implement a truly enterprise search, the above list of content-based considerations should get you thinking of all of the places where you may have content “hiding” in your organization.

From that list, you should have a good sense of the volume of content and the complexity of sources your search will need to deal with.

Combining that with a careful requirements definition process and evaluation of alternatives should lead to a successful selection of a tool.

Once you have a tool, you “just” need to apply the proper amount of elbow grease to get it to index all of the content you wish and present it in a sensible way to your users! No big deal, right?

I was recently asked by a former co-worker (Ray Sims) for some suggestions around requirements that he might use as the basis for an evaluation of search engines. Having just gone through such an evaluation myself, and also having posted here about the general methodology I used for the evaluation, I thought I’d follow that up with some thoughts on requirements.

If you find yourself needing to evaluate a search engine, these might be of value – at least in giving you some areas to further detail.

I normally think of requirements for search in two very broad categories – those that are more basically about helping the user doing the search (End User Search Requirements) and those that are more directed at the people (person) who is responsible for administering / maintaining the search experience (Administrator Requirements).

End User Search Requirements

• Search result page customization – Is it straightforward to provide your own UI on top of the search results (for integration into your web site)?
• Search result integration with other UIs (outside of a web experience) – Specifically, it’s possible you might want to use search results in a non-web-based application – can the engine do that? (If you can provide result pages in different formats, a simple way to do this is to provide an XML result format that an application can pull in via a URL.)
• Search result summaries for items – Specifically, these should be dynamic. The snippet shown in the results should show something relevant to what the searcher searched on – not just a static piece of text (like a metadata description field). This, by itself, can greatly enhance the perceived quality of results because it makes it easier for a user to make a determination on the quality of an item right from the search results – no need to look at the item (even a highlighted version of it).
• Highlighting – it should be possible to see a highlighted version of a result (i.e., search terms are highlighted in the display of the document)
• “Best Bets” (or key match or whatever) – Some don’t like these, but I think it’s important to have some ability to “hand pick” (or nearly hand pick) some results for some terms – also, I think it’s very desirable to be able to say “If a user searches on X, show this item as the top result” regardless of where that item would organically show in the result (or it might not even be really indexable)
• Relevancy calculation “soundness” – This basically means that the engine generates a good measure of relevancy for searches and encompasses most of what differentiates engines. You should understand at a general level what effects the relevancy as computed by the engine. (For many search engines, this is a the “magic dust” they can bring to the table – so they may not be willing to expose too much about how they do this but you should ask.)
• Stemming – The engine should support stemming – if a user searches on “run”, it should automatically match the use of words that share the same stem – “runs”, “running”, “ran”, etc.
• Similar to stemming, the engine should support synonyms – if I search on “shoe”, it might be useful to include content that matches “boot” or “slipper”, etc.
• Concept understanding (entity extraction) – Can the engine determine the entities in a piece of content even when the content is not explicitly defined? A piece of content might be about “Product X”, say, but it may never even explicitly mention “Product X”. Some search engines will claim to do this type of analysis.
• Performance – Obviously, good performance is important and you should understand how it scales. Do you expect a few thousand searches a week? Tens of thousands? Hundreds of thousands? You need to understand your needs and ensure that the engine will meet them.
• Customization of error / not found presentation – Can you define what happens when no results found or some type of system error happens – It can be useful to be able to define a specific behavior when an engine would otherwise return no results (a behavior that might be outside of the engine, specifically).
• Related queries – It might be desirable to have something like, “Users who searched on X also commonly searched on Y”

Administrator Requirements

• Indexing of web content – Most times, it’s important to be able to index web content – commonly through a crawler, especially if it’s dynamic content.
• Indexing of repositories – You should understand your repository architecture and which repositories will need to be indexed and how the engines will do so. Some engines provide special hooks to index different major vendors (Open Text, SharePoint, Documentum, etc.) These types of tools are often not crawlable using a general web spider / crawling approach.
• File System indexes – Many companies still have a significant content accessible on good old file servers – understand what types of file systems can be indexed and the protocol that the search engine supports (Samba, NFS, etc.)
• Security of search results – Often, you might want to provide a single search experience that users can use to search any content to which they can navigate, even if that content is in its own repository which follows its own (proprietary) mechanism to secure documents.
  • This is something we have not tackled, but some engines do so. You typically have two approaches – “early binding”, when the security is basically rolled into the index and “late binding” which does security checking as users do searching.
  • Most vendors do the former because it can be very expensive to do a security check on every document that might show up in search results.
  • The primary advantage of late binding is that if you refresh your index weekly on, say, Saturday and there’s a document to which I did not have access, if someone provides me access on Monday, I still won’t see it in search (until after the next refresh); conversely, people can see items in search results that they no longer have access to as well.
• Index scheduling / control – Regardless of the type of index, you should be able to control the schedule of indexing or how fast the indexer might hit your web sites / repositories / file systems. Also, it can be very useful to have different parts of the site refreshed at different rates. You might want press releases refreshed (or at least checked) hourly, while product documentation might only need to be refreshed weekly or monthly.
• Relevancy control – It should be possible to administratively modify the relevancy for items – up or down. Ideally, this should be based on attributes of the content such as: the server it’s on, the path on the server, the date range of the content, presence of particular meta data, etc.
• Synonyms – It should be possible to define business-specific synonyms. Some insight from Avi Rappoport (via the SearchCoP), is that you should be careful in the use of generic synonyms – they may cause more problems they fix (so if an engine provides synonym support, you might want to know if you get some default synonyms and how you might disable them).
• Automation / integration – It is nice if the search engine can integrate or somehow provide support for automatic maintenance of some aspects of its configuration. For example, synonyms – you might already have a means to manage those (say, in your taxonomy tool!) and having to manually administer them as a separate work process would probably lead to long-term maintainability issues. In that case, some type of import mechanism. Or, another example, have your relevancy adjustments integrated with your web analytics (so that more popular content based on usage goes up in relevancy).
• Performance (again) – How much content do you expect to index? How fast can that content be indexed by the engine? Does the engine do full re-indexing? Incremental? Real-time?
• Reporting – You need to have good reporting.
  • Obvious stuff like most common searches (grouped by different spans like day, hour, week, month, etc., and also for time periods you can define – meaning, “Show me most common searches for the last six months grouped by week”), most common “no result” searches, common “error” searches, etc.
  • It would be especially useful to be able to do time analysis across these types of dimensions – Most engines don’t provide that from my experience; you can get a dump for a time period and a separate one for another period and you have to manually compare them. Being able to say, “How common has this search been for the last six months in each month?” helps you understand longer-term trends.
  • Also, it can be very useful to see reports where the search terms are somehow grouped. So a search for “email” and a search for “e-mail” (to use a very simple example) would show up together – basically some kind of categorization / standardization of the searches. Doing grouping based purely on the individual searches can make it very hard to “see the forest for the trees”.
  • Lastly – reports on what people do with search results can be very useful. OK – fine, “Product X” is a top ten search consistently, but what are people selecting when they search on that? Do they not click on anything? Do they click on the same item 90% of the time? Etc.
  • I’m also planning to post separately on more details around search metrics and analytics.  Keep watch!
• Last but certainly not least – Architectural “fit” – Make sure you understand how well the engine will fit in your data center. OS(es) it runs on? Hardware compatibility, etc.  For some engines where you purchase a closed appliance, this may not be relevant but you should involve your data center people in understanding this area.