Lee Romero

In my last post, I described the goals I have tried to achieve with my proof of concept people search function. Here I will describe the design and implementation of this proof of concept.

Designing the Solution

Given the goals above, here’s the general outline of the design for this solution:

• It would be built as a web application that generates a “profile page” for each worker – it is the set of all such profile pages that comprise the targets for a search engine to index.
• Combined with a search engine (probably any search engine capable of indexing web pages would be sufficient – I used QuickFinder), it becomes trivial to integrate the search of these profiles into your enterprise search to provide a fourth generation solution to people search.
• The core tenet of the data used is that I wanted to identify a set of activities for workers. The aggregation of keywords related to those activity is used to generate a profile for a worker.
• An activity could potentially be anything that represents an event, action, writing, task, assignment, etc., that is associated with the worker.
• Some examples of activities might include: edit of a wiki article, assignment of a task in an online workspace, posting of a message in a discussion form, membership in a project team, publishing a document in a corporate repository, posting an email to a mailing list, and so on.

Initially the web application directly queried the various systems used as sources when generating a profile for a worker. That is not scalable and also limits the amount of processing you can do, so I designed a simple SQL database to contain the data for this (implemented in MySQL). This database is essentially a data mart of worker data. The primary tables are:

• worker (one row for each worker); this table contains the basic administrative data for a worker (it’s effectively a mirror of the organization’s corporate directory)
• activity_source (each row describes a single source of activity which a worker might produce)
• activity (one row for each individual “activity” associated with a worker); an activity must have a “description” – typically the title of an item or the subject of an email, etc.
• From these tables, a few additional tables are generated by processing the data from the activity table
  • activity_keyword (contains a row for each keyword associated with an activity); a keyword is either any (individual) word from the description of the activity or a piece of metadata associated with the item (for systems which support such);
  • worker_top_keyword (aggregates the individual keywords associated with a worker [by association from activity_keyword through activity to the worker table]) so it’s easy to identify the top keywords for a worker without doing aggregation queries; each keyword in this table is weighted (see the description below of weights); I think of the set of keywords in this table for a worker to be that worker’s “attributes”
  • worker_connection (aggregates “linkage” between workers based on similarity of their keyword profiles); more on this later.

With the implementation of this database, I also implemented a synchronization tool that updates the data in the tables from the source systems for the various types of activities.

By automatically pulling data from these source systems (which workers use in their regular day-to-day work), you remove the need for the workers to maintain data.

• By simply doing their job and “leaving traces” of that worker, they generate the data necessary for generating this profile. This achieves goal #2.
• By restricting the set of data sources used to ones which anyone could examine for a worker’s activities (for example, I can view the history of a Wiki article and see who has edited it), I achieve goal #3.

Now, how should the profile page for a worker be presented?

Initially, I put together a design that did two things: 1) provided a typical employee directory style layout of my administrative details and 2) provided a list of all of the activities for a worker, grouped by activity source. In other words, you would see a list of all of the Wiki articles edited by the worker, a list of mailing list memberships, a list of community memberships, project team memberships, task assignments, etc. Each activity source’s list would be separately displayed (in a simple bulleted list). (Before this would go into production, I always have assumed I would ask for some design help from our electronic marketing group to give it a more professional look, but I thought the bulleted list worked perfectly well functionally.)

This proved simple and effective and also enabled the profile page to provide direct links to those activities that are addressable via a link (for example, the profile page could link directly to a Wiki article I’ve edited from my profile page, it could link to each discussion post, etc.)

However, this approach suffered from at least two problems: 1) it lacked an immediately obvious visual presentation of a worker’s attributes, and 2) it exposed every detailed activity of a worker to anyone who viewed the profile (I found when I demoed this to people, some had the immediate reaction of, “Wow – anyone can see all of these details? I’m not sure I like that!” – a reaction that surprised me given that any of the details are generally visible to anyone who wants to look, but go figure).

After looking for alternatives, I found that the keywords for a worker (when combined with their weights) provided good input for a tag cloud – which is what I ended up using as the default presentation of a worker’s keywords (visible to everyone). This helps to highlight what someone is “about”, presents a generally attractive visualization of the data, and, if the default view of a worker displays this tag cloud (and the worker’s administrative data) and does not show all of the details, it alleviates the concern mentioned above.

I have found the implementation of the tag cloud to be the trigger that pulls people into this tool – it helps satisfy my goal #5 because, for most people who have looked at this, it provides immediate validation when they see words they expect to see in their own tag cloud.

Here’s a shot of what part of my profile page looks like (partially obscured):

Lee Romero Profile

Additional Design Considerations

I wanted to keep the initial proof of concept simple in order to try to test different ways of using the data from the activity sources. With that in mind, here are some details on how I’ve done this so far:

• When parsing the text associated with an activity into “keywords”, I took the simplest approach I could: the words from an activity are split into separate words when any non-alphanumeric is found. So a string like “content-management infrastructure” would result in 3 keywords: content, management and infrastructure.
• I also removed any words that are stop words in our search engine.
• Each keyword for a worker is assigned a weight. Simplistically, the weight of a keyword is the number of times that keyword shows up in that worker’s stream of activities.
• However, the tool that maintains the keywords allows an administrator to assign a weight to each activity source – so some sources can be given an artificial boost just by assigning a weight for that activity source higher than 1. The only source whose weight I’ve really toyed with so far is the corporate directory itself – I have given that a weight of 20 instead of 1.
• The weights for keywords are used in two ways:
  • The top 50 keywords (by weight) for a worker are used in the tag cloud for that worker. The weight is then used to size the words in the tag cloud.
  • When the “keywords” <meta> tag is being computed for a worker’s profile, the keywords are sorted by weight and the keywords are included until the length of the keywords content attribute is greater than 250 characters. This means that the top keywords are the ones which will give the worker higher relevance for searches on those words.
• Because all workers will have, at absolute minimum, the same details in this profile as they would in the corporate directory, and because the keywords from that activity source are given extra weight, those keywords will almost certainly be in the “keywords” <meta> tag for their profile – this helps satisfy my goal #6 by ensuring good relevance when people search on worker’s administrative data (first name, last name, etc.)

Some additional functions I have layered on top of the basic profile / search mechanism that I believe will make this a valuable solution:

• The keywords in the tag cloud are links to pages that provide details about that keyword. When a user clicks on a keyword in a tag cloud, they are presented with a tag cloud of keywords related to their starting keyword (related by way of people who have the keywords in common). In other words, it provides a set of keywords that have a lot in common with their starting keyword. The “keyword profile” page also provides a list of workers who use the selected keyword (the list is sorted by keyword weight).
• When you view a worker, you are also presented with a list of workers who are “similar to” the worker you are looking at – the similarity measure is the percent of overlap of the current worker’s profile (weighted keywords) maps to the other workers. This provides a way to explore a neighborhood of similar people.
• In addition to the list of similar worker, a link is provided for each worker which, when clicked, displays a page explaining why the two workers are similar.
• Almost all of the data sources have a date threshold applied to the data pulled from the source – most of them take data from the last year. This ensures that the data used to build a profile is effectively self-maintaining.
• Each worker has control over whether others can see all of the details (the individual activities) in their profile. By default, only the tag cloud and administrative data is visible. A worker can opt in to allow others to see their entire profile.

Issues / Future Directions

The proof of concept has been very interesting to work through and has presented me with some (subjective) proof of the value of this approach, as simple as it is. That being said, there are some issues and additional areas I hope are explored in the future:

• This is a proof of concept built as basically a skunkworks project – I am hoping it will officially get some sponsorship and be launched into production.
• I would like to see it integrated with additional data sources – currently, it uses 12 data sources but some high value sources that are not included would be our CRM system and our HR system. With the sources currently in use, it tends to skew the people whose profiles look sufficiently detailed to be ones who use the sources. Integrating these is relatively easy – a single SQL query from the source system that provides a list of activities for workers (where the source system can define whatever it wants to represent activities) is all that’s needed. It is this ease of adding in sources that achieves my goal #4.
• I believe there is still a lot of work to do around tweaking the weights of activity sources to balance out the effects of various sources.
• I would like to see some exploration of workers directly tagging other workers (to add keywords) or possibly allowing workers to give a thumbs up / thumbs down to individual keywords in a profile for a worker. This would add a powerful way for people to influence their own and others’ profiles.
• This approach also needs to receive more testing from others to validate its effectiveness. I have had a few dozen people look at it and provide feedback but some more quantitative approach to this would be valuable.
• I think this profile for a worker could be presented in a FOAF format as well – I’m not sure if that provides additional value, but it is a path to explore.
• The algorithm for parsing out keywords from the activities could be improved beyond the very simplistic parsing applied now.
• And, finally, I think that the measurement of similarity between workers could be significantly improved and the data from the links between workers embedded in this could be used to do some research to find “invisible communities” within the company. This would be a kind of organizational network analysis through data mining, which

I have previously described what I termed the various generations of solutions to the common challenge of workers finding connecting with or finding co-workers within an enterprise.  My most recent post described the fourth generation solution – which enables users to search and connect using much more than simple administrative terms (name, email, address, etc.) for the search.

Over my next couple of posts, I will provide a write-up of a proof of concept implementation I’ve assembled that meets a lot of the need for this with what I believe to be relatively minimal investment.

The follow represent the goals I’ve set for myself in this proof of concept:

1. Demonstrate the usefulness of a people search based on attributes of workers other than purely administrative data – things like their skills, competencies, work, interests, etc.
2. Demonstrate the feasibility of discerning the skills, competencies, work and/or interests through a means that does not depend on maintenance of data by the worker (which, from my experience, is not long-term maintainable).
   1. More specifically, provide a test bed to explore different algorithms for discovering keywords important keywords for people.
3. Demonstrate the feasibility of discovering keywords using only data that is generally “publicly visible” within an enterprise.
4. Provide a path for integrating manually-maintained skills data (if that were to be collected), or any other data (possibly including tags applied by co-workers as seen in IBM’s Dog Ear project).
5. Provide a compelling user experience that draws people in and gives people a visual presentation of what another person is “about” (what describes them).
6. Provide a solution that provides, at minimum, the equivalent of a 3rd generation solution (in other words you can find that worker based on their administrative data).

Also, I wanted to say that part of the inspiration for this proof of concept came from a session I attended at Enterprise Search Summit 2007 as presented by Trent Parkhill.  In his session, he described a mechanism where submissions to a company’s repository would be tagged with the names of participants in the project that produced the document as a deliverable.  Then, when users were searching for content, there was a secondary search that produced a list of people associated with the terms and / or documents found by the user’s search.  I’ve kind of turned that around and treated the people as being tagged by the keywords of the items they produce.

In my next post, I will describe the overall design of my proof of concept.

So we get to the exciting conclusion of my essays on the inclusion of employees in enterprise search. If you’ve read this far, you know how I have characters the first and second generation solutions and also provided a description of a third generation solution (which included some details on how we implemented it).

Here I will describe what I think of as a fourth generation solution to people finding within the enterprise. As I mentioned in the description of the third generation solution, one major omission still at this point is that the only types of searches with which you can find people is through administrative information – things like their name, address, phone number, user ID, email, etc.

This is useful when you have an idea of the person you’re looking for or at least the organization in which they might work. What do you do when you don’t know the person and may not even know the organization in which they work? You might know the particular skills or competencies they have but that may be it. This problem is particularly problematic in larger organizations or organizations that are physically very distributed.

The core idea with this type of solution is to provide the ability to find and work with people based on aspects beyond the administrative – the skills of the people, their interests, perhaps the network of people with which they interact, and more. While this might be a simplification, I think of this as expertise location, though that, perhaps, most cleanly fits into the first use case described below.

Some common use cases for this type of capability include:

• Peer-to-peer connections – an employee is trying to solve a particular problem and they suspect someone in the company may have some skills that would enable them to solve the problem more quickly. Searching using those skills as keywords would enable them to directly contact relevant employees.
• Resource planning – a consulting organization needs to staff a particular project and needs to find specific people with a particular skill set.
• Skill assessment – an organization needs to be able to ascertain the overall competency of their employees in particular skill sets to identify potential training programs to make available.

This capability is something that has often been discussed and requested at my current employer, but which no one has really been willing to sponsor. That being said, I know there are several vendors with solutions in this space, including (at least – please share if you know of others):

• Connectbeam – A company I first found out about at KM World 2007. They had some interesting technology on display that combines expertise location with the ability to visualize and explore social networks based on that expertise. Their product could digest content from a number of systems to automatically discern expertise.
• ActiveNet – A product from Tacit Software, which (at a high level) is similar to Connectbeam. An interesting twist to this product is that it leaves the individuals whose expertise are managed in the system in control of how visible they are to others. In the discussions I’ve had with this company about the product, I’ve always had the impression that, in part, this provides a kind of virtual mailing list functionality where you can contact others (those with the necessary expertise) by sending an email without knowing who it’s going to. Those who receive it can either act on it or not and, as the sender, you only know who replies.
• Another product about which I only know a bit is from a company named Trampoline Systems. I heard about them as I was doing some research on how to tune a prototype system of my own and understand that their Sonar platform provides similar functionality.
• [Edit: Added this on 03 November, 2008] I have also found that Recommind provides expertise location functionality – you can read more about it here.
• [Edit: Added this on 03 November, 2008] I also understand that the Inquira search product provides expertise location, though it’s not entirely clear to me from what I can find about this tool how it does this.

A common aspect of these is that they attempt to (and perhaps succeed) in automating the process of expertise discovery. I’ve seen systems where an employee has to maintain their own skill set and the problem with these is that the business process to maintain the data does not seem to really embed itself into a company – inevitably, the data gets out of date and is ill-maintained and so the system does not work.

I can not vouch for the accuracy of these systems but I firmly believe that if people search in the enterprise is going to meet the promise of enabling people to find each other and connect based on of-the-moment needs (skills, interests, areas of work, etc), it will be based on this type of capability – automatically discovering those aspects of a worker based on their work products, their project teams, their work assignments, etc.

I imagine within the not too distant future, as we see more merger of the “web 2.0” functionality into the enterprise this type of capability will become expected and welcome – it will be exciting to see how people will work together then.

This brings to a close my discussion of the various types of people search within the enterprise. I hope you’ve found this of interest. Please feel free to let me know if you think I have any omissions or misstatements in here – I’m happy to correct and/or fill in.

I plan another few posts that discuss a proof of concept I have put together based around the ideas of this fourth generation solution – look for those soon!

In my last post, I wrote about what I termed the first generation and second generation solution to people search in enterprise. This time, I will describe what I call a “third generation” solution to the problem that will integration people search with your enterprise search solution.

This is the stage of people search in use within my current employer’s enterprise.

What is the third generation?

What I refer to as a third generation solution for people search is one where an employee’s profile (their directory entry, i.e., the set of information about a particular employee) becomes a viable and useful target within your enterprise search solution. That is, when a user performs a search using the pervasive “search box” (you do have one, right?), they should be able to expect to find their fellow workers in the results (obviously, depending on the particular terms used to do the search) along with any content that matches that.

You remove the need for a searcher to know they need to look in another place (another application, i.e., the company’s yellow pages) and, instead, reinforce the primacy of that single search experience that brings everything together that a worker needs to do their job.

You also offer the full power of your enterprise search engine:

• Full text search – no need to specifically search within a field, though most engines will offer a way to support that as well if you want to ffer that as an option;
• The power of the search engine to work on multi-word searches to boost relevancy – so a search on just a last name might include a worker’s profile in the search results but one that includes both a first and last name (or user ID or location or other keywords that might appear in the worker’s profile) likely ensures that the person shows in the first page of results amidst other content that match;
• The power of synonyms – so you can define synonyms for names in your engine and get matches for “Rob Smith” when a user searches on “Robert Smith” or “Bob Smith”;
• Spelling corrections – Your engine likely has this functionality, so it can automatically offer up corrections if someone misspells a name, even.

Below, you will find a discussion of the implementation process we used and the problems we encountered. It might be of use to you if you attempt this type of thing.

Before getting to that, though, I would like to discuss what I believe to be remaining issue with a third generation solution in order to set up my follow-up post on this topic, which will describe additional ideas to solving the “people finder” problem within an enterprise.

The primary issue with the current solution we have (or any similar solution based strictly on information from a corporate directory) is that the profile of a worker consists only of administrative information. That is, you can find someone based on their name, title, department, address, email, etc., etc., etc., but you can not do anything useful to find someone based on much more useful attributes – what they actually do, what their skills or competencies are or what their interests might be. More on this topic in my next post!

The implementation of our third generation solution (read on for the gory details)

Read on from here for some insights on the challenges we faced in our implementation of this solution. It gets pretty detailed from here on out, so you’ve been warned!

(more…)

This post is the first of a brief series of posts I plan to write about the integration of “people search” (employee directory) with your enterprise search solution. In a sense, this treats “people” as just another piece of content within your search, though they represent a very valuable type of content.

This post will be an introduction and describe both a first and second generation solution to this problem. In subsequent posts, I plan to describe a solution that takes this solution forward one step (simplifying things for your users among other things) and then into some research that I believe shows a lot of promise and which you might be able to take advantage of within your own enterprise search solution.

Why People Search?

Finding contact information for your co-workers is such a common need that people have, forever, maintained phone lists – commonly just as word processing documents or spreadsheets – and also org charts, probably in a presentation file format of some type. I think of this approach as a first generation solution to the people search problem.

Its challenges are numerous, including:

1. The maintenance of the document is fraught with the typical issues of maintaining any document (versioning, availability, etc.)
2. In even a moderately large organization, the phone list may need to be updated by several people throughout the organization to keep it current.
3. Search within this kind of phone list is limited – you can ensure you always have the latest version and then open it up and use your word processor’s search function or (I remember this well, myself) always keep a printout of the latest version of the phone list next to your workspace so you can look through it when you need to contact someone.

As computer technology has evolved and companies implemented corporate directories for authentication purposes (Active Directory, LDAP, eDirectory, etc.), it has become common to maintain your phone book as a purely online system based on your corporate directory. What does such a solution look like and what are its challenges?

A “Second Generation” Solution

I think it’s quite common now that companies will have an online (available via their intranet) employee directory that you can search using some (local, specific to the directory) search tools. Obvious things like doing fielded searches on name, title, phone number, etc. My current employer has sold a product named eGuide for quite some time that provides exactly this type of capability.

eGuide is basically a web interface for exposing parts of your corporate Directory for search and also for viewing the org chart of a company (as reflected in the Directory).

We have had this implemented on our intranet for many years now. It has been (and continues to be) one of the more commonly used applications on our intranet.

The problems with this second generation solution, though, triggered me to try to provide a better solution a few years ago using our enterprise search. What are the problems with this approach? Here are the issues that triggered a different (better?) solution:

1. First and foremost, with nothing more than the employee finder as a separate place to search, you immediately force a searcher to have to make a decision before they do their search as to where they want to search. Many users might expect that the “enterprise” search actually does include anything that they can navigate to as potential targets so when they search on a person’s name and don’t see it in the result set they immediately think either A) why does the search not include individual people’s information, or B) this search engine is so bad that, even though it must include people information, it can’t even show the result at a high enough relevance to get it on the first page!
   1. Despite my statement to the contrary above, I am aware that Jakob Nielsen does actually advocate the presence of both a “people search” box and a more general search box because people are aware of the distinction between searching for content and search for people. We do still have both search boxes on our intranet, though, in a sense, the people search box is redundant.
2. Secondly, the corporate directory commonly is a purely fielded search – you have to select which field(s) you want to search in and then you are restricted to searching just those fields.
   1. In other words, you as a searcher, need to know in which field a particular string (or partial string) might appear. For many fields, this might not be an issue – generally, first and last name are clear (though not always), email, phone number, etc., but the challenge is that a user has to decide in which field they want to look.
3. Third, related to the previous point, directory searches are generally simplistic searches based on string matching or partial string matching. With a full search engine, you introduce the possibility of taking advantage of synonyms (especially useful on first names), doing spelling corrections, etc.

So there’s a brief description of what I would characterize as a first generation solution and a second generation solution along with highlights of some issues with each.

Up next, I’ll describe the next step forward in the solution to this issue – integrating people into your enterprise search solution.