I’ve previously written about the standards I’ve put together for (web-based) applications that help ensure good findability for content / data within that application. These standards are generally relatively easy to apply to custom applications (though it can still be challenging to get involved with the design and development of those applications at the right time to keep the time investment minimal, as I’ve also previously written about).

However, it can be particularly challenging to apply these standards to third-party applications – For example, your CRM application, your learning management system, or your HR system, etc. Applying the existing standards could take a couple of different forms:

1. Ideally, when your organization goes through the selection process for such an application, your application standards are explicitly included in the selection criteria and used to ensure you select a solution that will conform to your standards
2. More commonly, you will identify compliance to the standards (perhaps during selection but perhaps later during implementation) and you might need to implement some type of customization within the application to provide compliance.
3. Hopefully, you identify compliance to the standards during selection or later, but you find you can not customize the application and you need a different solution.

The rest of this post will discuss a solution for option #3 above – how you can implement a different solution. Note that some search engines will provide pre-built functionality to enable search within many of the more common third party solutions – those are great and useful, but what I will present here is a solution that can be implemented independent of the search engine (as long as the search engine has a crawler-based indexing function) and which is relatively minimal in investment.

Solving the third-party application conundrum for Enterprise Search

So, you have a third party application and, for whatever reason, it does not adhere to your application standards for findability. Perhaps it fails the coverage principle and it’s not possible to adequate find the useful content without getting many, many useless items; or perhaps it’s the identity principle and, while you can find all of the desirable targets, they have redundant titles; or it might even be that the application fails the relevance principle and you can index the high value targets and they show up with good names in results but they do not show up as relevant for keywords which you would expect. Likely, it’s a combination of all three of these issues.

The core idea in this solution is that you will need a helper application that creates what I call “shadow pages” of the high value targets you want to include in your enterprise search.

Note: I adopted the use of the term “shadow page” based on some informal discussions with co-workers on this topic – I am aware that others use this term in similar ways (though I don’t think it means the exact same thing) and also am aware that some search engines address what they call shadow domains and discourage their inclusion in their search results. If there is a preferred term for the idea described here – please let me know!

What is a shadow page? For my purposes here, I define a shadow page as:

• A page which uniquely corresponds to a single desirable search target;
• A page that has a distinct, unique URL;
• A page that has a <title> and description that reflects the search target of which it is a shadow, and that title is distinct and provides a searcher who sees it in a search results page with insight about what the item is;
• A page that has good metadata (keywords or other fields) that describe the target using terminology a searcher would use;
• A page which contains text (likely hidden) that also reflects all of the above as well to enhance relevance for the words in the title, keywords, etc.;
• A page which, when accessed, will automatically redirect a user to the page of which the page is a shadow.

To make this solution work, there are a couple of minimal assumptions of the application. A caveat: I recognize that, while I consider these as relatively simple assumptions, it is very likely that some applications will still not be able to meet these and so not be able to be exposed via your enterprise search with this type of solution.

1. Each desirable search target must be addressable by a unique URL;
2. It should be possible to define a query which will give you a list of the desirable targets in the application; this query could be an SQL query run against a database or possible a web services method call that returns a result in XML (or probably other formats but these are the most common in my experience);
3. Given the identity (say, a primary key if you’re using a SQL database of some type) of a desirable search target, you must be able to also query the application for additional information about the search target.

Building a Shadow Page

Given the description of a shadow page and the assumptions about what is necessary to support it, it is probably obvious how they are used and how they are constructed, but here’s a description:

First – you would use the query that gives you a list of targets (item #2 from the assumptions) from your source application to generate an index page which you can give your indexer as a starting point.  This index page would have one link on it for each desirable target’s shadow page.  This index page would also have “robots” <meta> tags of “noindex,follow” to ensure that the index page itself is not included as a potential target.

Second – The shadow page for each target (which the crawler reaches thanks to the index page) is dynamically built from the query of the application given the identity of the desirable search target (item #3 from the assumptions).  The business rules defining how the desirable target should behave in search help define the necessary query, but the query would need to contain at minimum some of the following data: the name of the target, a description or summary of the target, some keywords that describe the target, a value which will help define the true URL of the actual target (per assumption #1, there must be a way to directly address each target).

The shadow page would be built something like the following:

• The <title> tag would be the name of the target from the query (perhaps plus an application name to provide context)
• The “description” <meta> tag would be the description or summary of the target from the query, perhaps plus a few static keywords that help ensure the presence of additional insight about the target.   For example, if the target represents a learning activity, the additional static text might indicate that.
• The “keywords” <meta> tag would include the keywords from the query, plus some static keywords to ensure good coverage.  To follow the previous example, it might be appropriate to include words like “learning”, “training”, “class”, etc. in a target that is a learning activity to ensure that, if the keywords for the specific target do not include those words, searchers can still find the shadow page target in search.
• The <body> of the page can be built to include all of the above text – from my experience, wrapping the body in a CSS style that visually hides the text keeps the text from actually appearing in a browser.
• Lastly, the shadow page has a bit of JavaScript in it that redirects a browser to the actual target – this is why you need to have the target addressable via a URL and also that the query needs to provide the information necessary to create that URL.  Most engines (I know of none) will not be able to execute the JavaScript, so will not know that the page is really a redirect to the desired target.

The overall effect of this is that the search engine will index the shadow page, which has been constructed to ensure good adherence to the principles of enterprise search, and to a searcher, it will behave like a good search target but when the user clicks on it from a search result, the user ends up looking at the actual desired target.  The only clue the user might have is that the URL of the target in the search results is not what they end up looking at in their browser’s address bar.

The following provides a simple example of the source (in HTML – sorry for those who might not be able to read it) for a shadow page (the parts that change from page to page are in bold):

<html>
<head>
<TITLE>title of target</TITLE>
<meta name="robots" content="index, nofollow">
<meta name="keywords" content="keywords for target">
<meta name="description" content="description of target">
<script type="text/javascript">
document.location.href="URL of actual target";
</script>
</head>

<body>
<div style="display:none;">
<h1>title of target</h1>
description of target and keywords of target
</div>
</body>
</html>

Advantages of this Solution

A few things that are immediately obvious advantages of this approach:

1. First and foremost, with this approach, you can provide searchers with the ability to find content which otherwise would be locked away and not available via your enterprise search!
2. You can easily control the targets that are available via your enterprise search within the application (potentially much easier than trying to figure out the right combination of robots tags or inclusion / exclusion settings for your indexer).
3. You can very tightly control how a target looks to the search engine (including integration with your taxonomy to provide elaborated keywords, synonyms, etc)

Problems with this Solution

There are also a number of issues that I need to highlight with this approach – unfortunately, it’s not perfect!

1. The most obvious issue is that this depends on the ability to query for a set of targets against a database or web service of some sort.
   1. Most applications will be technically able to support this, but in many organizations, this could present too great a risk from a data security perspective (the judicious use of database views and proper management of read rights on the database should solve this, however!)
   2. This potentially creates too high a level of dependence between your search solution and the inner workings of the application – an upgrade of the application could change the data schema enough to break this approach.  Again, I think that the use of database views can solve this (by abstracting away the details of the implementation into a single view which can be changed as necessary through any upgrade).
2. Some applications may simply not offer a “deep linking” ability into high value content – there is no way to uniquely address a content item without the context of the application.  This solution can not be applied to such applications.  (Though my opinion is that such applications are poorly designed, but that’s another matter entirely!)
3. This solution depends on JavaScript to forward the user from the shadow page to the actual target.  If your user population has a large percentage of people who do not use JavaScript, this solution fails them utterly.
4. This solution depends on your search engine not following the JavaScript or somehow otherwise determining that the shadow page is a very low quality target (perhaps by examining the styles on the text and determining the text is not visible).  If you have a search engine that is this smart, hopefully you have a way to configure it to ignore this for at least some areas or page types.
5. Another major issue is that this solution largely circumvents a search engine’s built in ability to do item-by-item security as the target to the search engine is the shadow page.  I think the key here is to not use this solution for content that requires this level of security.

Conclusion

There you have it – a solution to the exposure of your high value targets from your enterprise applications that is independent of your search engine and can provide you (the search administrator) with a good level of control over how content appears to your search engine, while ensuring that what is included highly adheres to my principles of enterprise search.

Here I will describe what I think of as a fourth generation solution to people finding within the enterprise. As I mentioned in the description of the third generation solution, one major omission still at this point is that the only types of searches with which you can find people is through administrative information – things like their name, address, phone number, user ID, email, etc.

This is useful when you have an idea of the person you’re looking for or at least the organization in which they might work. What do you do when you don’t know the person and may not even know the organization in which they work? You might know the particular skills or competencies they have but that may be it. This problem is particularly problematic in larger organizations or organizations that are physically very distributed.

The core idea with this type of solution is to provide the ability to find and work with people based on aspects beyond the administrative – the skills of the people, their interests, perhaps the network of people with which they interact, and more. While this might be a simplification, I think of this as expertise location, though that, perhaps, most cleanly fits into the first use case described below.

Some common use cases for this type of capability include:

• Peer-to-peer connections – an employee is trying to solve a particular problem and they suspect someone in the company may have some skills that would enable them to solve the problem more quickly. Searching using those skills as keywords would enable them to directly contact relevant employees.
• Resource planning – a consulting organization needs to staff a particular project and needs to find specific people with a particular skill set.
• Skill assessment – an organization needs to be able to ascertain the overall competency of their employees in particular skill sets to identify potential training programs to make available.

This capability is something that has often been discussed and requested at my current employer, but which no one has really been willing to sponsor. That being said, I know there are several vendors with solutions in this space, including (at least – please share if you know of others):

• Connectbeam – A company I first found out about at KM World 2007. They had some interesting technology on display that combines expertise location with the ability to visualize and explore social networks based on that expertise. Their product could digest content from a number of systems to automatically discern expertise.
• ActiveNet – A product from Tacit Software, which (at a high level) is similar to Connectbeam. An interesting twist to this product is that it leaves the individuals whose expertise are managed in the system in control of how visible they are to others. In the discussions I’ve had with this company about the product, I’ve always had the impression that, in part, this provides a kind of virtual mailing list functionality where you can contact others (those with the necessary expertise) by sending an email without knowing who it’s going to. Those who receive it can either act on it or not and, as the sender, you only know who replies.
• Another product about which I only know a bit is from a company named Trampoline Systems. I heard about them as I was doing some research on how to tune a prototype system of my own and understand that their Sonar platform provides similar functionality.
• [Edit: Added this on 03 November, 2008] I have also found that Recommind provides expertise location functionality – you can read more about it here.
• [Edit: Added this on 03 November, 2008] I also understand that the Inquira search product provides expertise location, though it’s not entirely clear to me from what I can find about this tool how it does this.

A common aspect of these is that they attempt to (and perhaps succeed) in automating the process of expertise discovery. I’ve seen systems where an employee has to maintain their own skill set and the problem with these is that the business process to maintain the data does not seem to really embed itself into a company – inevitably, the data gets out of date and is ill-maintained and so the system does not work.

I can not vouch for the accuracy of these systems but I firmly believe that if people search in the enterprise is going to meet the promise of enabling people to find each other and connect based on of-the-moment needs (skills, interests, areas of work, etc), it will be based on this type of capability – automatically discovering those aspects of a worker based on their work products, their project teams, their work assignments, etc.

I imagine within the not too distant future, as we see more merger of the “web 2.0″ functionality into the enterprise this type of capability will become expected and welcome – it will be exciting to see how people will work together then.

This brings to a close my discussion of the various types of people search within the enterprise. I hope you’ve found this of interest. Please feel free to let me know if you think I have any omissions or misstatements in here – I’m happy to correct and/or fill in.

I plan another few posts that discuss a proof of concept I have put together based around the ideas of this fourth generation solution – look for those soon!

This is the stage of people search in use within my current employer’s enterprise.

What is the third generation?

What I refer to as a third generation solution for people search is one where an employee’s profile (their directory entry, i.e., the set of information about a particular employee) becomes a viable and useful target within your enterprise search solution. That is, when a user performs a search using the pervasive “search box” (you do have one, right?), they should be able to expect to find their fellow workers in the results (obviously, depending on the particular terms used to do the search) along with any content that matches that.

You remove the need for a searcher to know they need to look in another place (another application, i.e., the company’s yellow pages) and, instead, reinforce the primacy of that single search experience that brings everything together that a worker needs to do their job.

You also offer the full power of your enterprise search engine:

• Full text search – no need to specifically search within a field, though most engines will offer a way to support that as well if you want to ffer that as an option;
• The power of the search engine to work on multi-word searches to boost relevancy – so a search on just a last name might include a worker’s profile in the search results but one that includes both a first and last name (or user ID or location or other keywords that might appear in the worker’s profile) likely ensures that the person shows in the first page of results amidst other content that match;
• The power of synonyms – so you can define synonyms for names in your engine and get matches for “Rob Smith” when a user searches on “Robert Smith” or “Bob Smith”;
• Spelling corrections – Your engine likely has this functionality, so it can automatically offer up corrections if someone misspells a name, even.

Below, you will find a discussion of the implementation process we used and the problems we encountered. It might be of use to you if you attempt this type of thing.

Before getting to that, though, I would like to discuss what I believe to be remaining issue with a third generation solution in order to set up my follow-up post on this topic, which will describe additional ideas to solving the “people finder” problem within an enterprise.

The primary issue with the current solution we have (or any similar solution based strictly on information from a corporate directory) is that the profile of a worker consists only of administrative information. That is, you can find someone based on their name, title, department, address, email, etc., etc., etc., but you can not do anything useful to find someone based on much more useful attributes – what they actually do, what their skills or competencies are or what their interests might be. More on this topic in my next post!

The implementation of our third generation solution (read on for the gory details)

Read on from here for some insights on the challenges we faced in our implementation of this solution. It gets pretty detailed from here on out, so you’ve been warned!

First up – How to ensure that we get correct coverage of the content set? As I’ve written about before, our search solution is based on Novell’s QuickFinder – which is a good, though not particularly sophisticated, search engine. It offers two types of indexes – a crawled index and a file system index. Given that our targets for this were web pages dynamically generated by eGuide, the only feasible option was a crawled index. Simple enough, right?

My first attempt to solve this was to simply point the indexer at the eGuide application and let it go. However, because eGuide has been built assuming your only (or at least primary) experience is going to be using it to do searches, there’s nothing of any real use that a crawler will find to index (it’s all hidden away behind HTML forms). All a crawler will find is the “home page” of the application and possibly a few additional informational pages that are linked to directly from the home page.

Commonly, I have found that it’s possible to provide a URL based on the HTTP GET to a crawler that has the effect of doing a search using a form – many applications will treat a GET that specifies parameters in a query string the same as a POST that passes form-based input variables in the content of the HTTP post. I tried this approach with eGuide but then ran into two additional issues:

• The results come in limited blocks of results (100 at most) and,
• Within the results pages and profile pages, there are links to many, many, useless pages (sorting the results on different columns, “printable” versions of an employee’s profile, etc.)

Both of these issues could be resolved with some changes in the eGuide application itself, but that was not feasible, due to resource constraints.

The solution we came up with was a simple web application which was directly integrated with the directory and which would do one thing: generate a single HTML file that contained a link to each and every employee’s profile – effectively a simple index page with a bunch of links. Easy enough. Then we defined the index for this to start at that page and go one level deep from it and the problem is resolved – we get exactly the set of profiles and nothing more.

Now onto the second principle of any search solution – identity. So we managed to get everything indexed that we wanted to and could try searches to find people (and they worked)! However… We then found that eGuide (or, more accurately, the templates we were using for eGuide) suffered from one of the many problems you encounter with web application: Every single page had the same text in the <title> tag – “Novell eGuide”. So you can perform searches and get the correct items showing in the results page, but, because QuickFinder uses the <title> as a primary identifier in search results, you end up seeing 10 items all titled, “Novell eGuide”! Not so very useful.

Now onto the third principle of enterprise search – relevance (you’ll see why I don’t jump right to the solution for the identity issue in a moment). So we now have all employees as viable targets in the enterprise search and (assuming we fix the issue with identity mentioned above), we then run into the issue that, by itself, the profile of an employee may actually not be that relevant even when someone searches on that person’s name. Why? Because it is very possible (likely) that that person’s name is on a number of other pages or embedded in the metadata of documents that are also part of the enterprise search. So their profile may show up as a result but is likely to not be high enough up the results to show up.

The solution to both of these issues turned out to be some very simple changes to the eGuide template. First, stick the employee’s name into the <title> tag – now, when someone’s profile shows up in the enterprise search, it shows as “Novell eGuide: <person’s name>”. Very nice. This has the additional benefit (with our search engine) of also boosting relevancy of profiles based on searches on employee names as words in titles significantly boost relevance of the content for searches on those words.

In addition, we made two additional enhancements – we included a “keywords” <meta> tag in the template that includes the values of an employee’s name, title, department, etc. as keywords. Again, this boosts relevancy for searches on those keywords. We also added a “description” <meta> tag in the template which is something general like “eGuide Profile for <person’s name>” – with our engine, the “description” <meta> tag is used as part of the “snippet” for a result in the results page.

With these changes, we finally had excellent coverage of employee profiles, excellent identity of the target items and good relevance of the items. A success all around.

The last issue we had to deal with revolves around our own infrastructure and corporate policies and security. Novell has a fairly sophisticated authentication and provisioning infrastructure. So it’s possible that a worker can have general access to our intranet and also to search but not have access to eGuide (a contractor or similar type of worker). Control for this is provided by Access Manager but that works based on ACL rules defined on paths on the intranet. So if a worker does not have access to eGuide, they can not access the specific path on the intranet through which that application is available.

Search, on the other hand, cuts across all paths. Also, our search engine does not integrate (at least we have not integrated it) with Access Manager to provide either early binding or late binding on security of search results. So the question is, how do we make these valuable search results appear in our enterprise search without facing the possibility of allowing access to information that someone shouldn’t have?

The solution we came up with was a compromise: From a policy perspective, the business was OK with people (well, their profiles) showing up as targets in search results. But, we needed to ensure that only their name showed – no other details should be visible. As mentioned above, QuickFinder uses the “description” <meta> tag as part of the snippet shown for results, but it will also pull text from the page to generate a snippet – possibly showing more directly in search results then we should. We achieved the compromise by ensuring that people will only show in our “Best Bets” section of a results page – and items in “Best Bets” only show the title and an icon indicating the item type. No more details are visible.

Below is a partial screen showing how a search on my own last name displays in the results (I’ve blurred out the names of other people that show up but left in a piece of content that is also a “best bet” on my name – in this case, the home page for a community of practice of which I am a community leader). You’ll note that it shows the name (and only the name) of the person and also displays a specific icon next to the results so that they “stand out” a bit more in the results list as being “people” (as opposed to a Word document or an HTML page, etc.)

Partial Search Results Screen when searching on romero

The Enterprise in Enterprise Search

For me, the factors of a search solution that make it an enterprise solution include the following:

The user interface to access the solution is available to all employees of the company.

This has the following implications:

• Given today’s technologies, this probably means that it’s a web-based interface to access the search.
  • More generally, the interface needs to be easily made available across the enterprise. In any somewhat-large organization, that means something either available online or easily installed or accessed from a user’s workspace.
• I would also suggest that the search interface should be easily accessible from an employee’s standard workspace or a common starting point for employees.
  • One easy way to achieve this is to make access to an enterprise search solution part of the general intranet experience – especially on an intranet that shares a standard look-and-feel (and so, hopefully, a standard template). This is the ubiquitous “search box”.
  • Alternately, if users commonly use a specific application (say a CRM application or a collaboration tool), integrating the enterprise search into that is a better solution.
  • Lastly, it might be necessary to make access to the search solution “many-headed”. Meaning, it might be best to make it available through a number of means, including through a standard intranet search, a specialized client-based application and embedded in other, user-specific tools.
• Given the likely broad range of users who will use it, the search interface should be subject to very thorough usability design and testing.
• Adopting some of the standard conventions of a search experience are a good idea.

The content available through the solution covers all (relevant) content available to employees

This has the following implications:

• If your enterprise has a significant volume of web content, your enterprise search should index all of those web pages – either via a web crawling approach or via indexing the file system containing the files (if it’s all static).
• If your enterprise has a significant volume of content (data) in enterprise applications (CRM solution, HR system, etc.), you should have a strategy to determine which (if any) of the content from those systems would be included, how it will be included and how it will be presented in search results (potentially combined with content from many other systems in the same results page)
• If your enterprise has custom web applications (and what organization does not), you should expect to provide a set of standards for design and development of web applications to ensure good findability from them and also expect to have to monitor compliance with those.
• If your enterprise has significant content in collaboration tools (and who doesn’t – at least email!), you should have a strategy for including or not including that content. This could be very broad-ranging – email, SharePoint (and similar applications from companies like Interwoven, Open Text, Vignette, Novell, etc.), shared file systems, IM logs, and so on. At the very least, you need to consider the cost and value of including these types of content.
• If you have content repositories available to employees (a document management system (or systems!) or a records management system), again, you should consider the cost and value of including content from these in your enterprise search.
• While it is very useful to have a separate search for finding employees in a corporate directory, I believe that an enterprise search solution should include employees as a distinct “content type” and include them in standard search results page as well when relevant (e.g., searching on employee names, etc)
• Another major question regarding the content of your enterprise search is security. If you include all of that content in your search, how will you manage the security of the items? The two major options are early binding (building ACLs into the search) or late binding (checking security at search time). If you are not familiar with these, I would recommend you do a bit of internet searching on the topics as it’s very important to your solution. I’ve found some interesting articles on this topic.
  • In my mind, it’s also feasible to “punt” on security in a sense and work to ensure that your enterprise search solution includes everything that is generally accessible to your employee population but does not include anything with specific access control on it.
  • If you can achieve the effect of getting a user “close to” the content (ensuring some level of “information scent” shows up) but leaving it to the user to make the final step (through any application-specific access control) seems to work well.

The Search in Enterprise Search

The other half of your enterprise search solution will be the search engine itself. There are plenty (many!) options available with a variety of strengths and weaknesses. I think if you plan to implement a truly enterprise search, the above list of content-based considerations should get you thinking of all of the places where you may have content “hiding” in your organization.

From that list, you should have a good sense of the volume of content and the complexity of sources your search will need to deal with.

Combining that with a careful requirements definition process and evaluation of alternatives should lead to a successful selection of a tool.

Once you have a tool, you “just” need to apply the proper amount of elbow grease to get it to index all of the content you wish and present it in a sensible way to your users! No big deal, right?