Having introduced some basic, standard, definitions in my previous post, in this one I am going to propose some standard measures derived from those that enable comparisons across solutions. These also are extremely useful for individual solutions where you, as an enterprise search manager, might want to have tools at hand to proactively improve your users’ experience.
A quick recap of what I defined before:
The first derived measure is one I call “lost clicks”. This measures the raw number of search sessions that resulted in no click:
This is a useful measure that tells you how many times, in total, users initiated a session but found nothing of interest to click on.
You can also think of this as an indicator that measures the number of total failed search sessions.
One more point I’ll make on this is that, because it is a raw number (not a ratio or percentage), it is not useful as a key performance indicator (KPI).
Now, finally, to my proposal for a standard measure of the quality of a search solution – a measure that, I think, can be usefully applied to all enterprise search solutions, can be used to drive improvement within a solution, and can be used to compare across such solutions.
That measure is “abandonment rate”, which I define as the percent of sessions that are ‘failed sessions’:
which, after a bit of simplifying, I normally write as:
This measure has some important advantages over a simpler click-rate model (e.g., [success rate] = [click] / [search]). For one thing, it avoids some simple problems that can be caused by a few anomalous users; for a second, it avoids the ‘trap’ of assuming a click is a success.
There are two anomalous patterns I see every once in a while:
By using only the first click and also the search session as the denominator, these scenarios don’t come into play (note that because I am recommending still capturing the simpler ‘search’ and the simpler ‘click’ metrics, you can still do some interesting analyses with these!).
The second advantage I mentioned above is more of a philosophical one – the success rate measure as defined builds in more strongly that you are measuring user success. This is a strong statement.
By focusing on abandonment, I find it a more honest view – your metrics don’t build in an assumption that a click is likely a success but, instead, that a failure to find something of interest to click on is more clearly an indication of likely failure.
What do I mean?
When I consider the ideas of “success” and “failure” in a search solution, I always have to remind myself of the good and bad sides of both – what do I mean by that??
However, there are other possibilities to consider!
Getting back to my description of how measuring and tracking abandonment rate is better then a success rate – my assumption has been that good abandonment and bad success will always exist for your users, however, good abandonment is likely a much smaller percentage of sessions than bad success and, more importantly, it is much easier to “improve” your search by increasing bad success then decreasing good abandonment.
There is my proposal for a measure to be used to assess search solutions for the quality of the user experience – abandonment rate.
It is not perfect and it is still just an indicator but I have found it incredibly useful to actually drive action for improvement. I’ll share more on this in my next post.
In my last few posts, I have commented on the lack of standard measures to use for enterprise search (leading to challenges of comparing various solutions to others among other things) and suggested some criteria for what standard measures to use.
In this post, I am going to propose a few basic measures that I think meet the criteria and that any enterprise search solution should be able to provide. The labels are not critical for these, but the meaning of them is, I think, very important.
First, and most important, is a search. A search is a single action in which a user retrieves a set of results from the search engine. Different user experiences may “count” these events differently.
When a user starts the process (in my experience, typically with a search term typed into a box on a web page somewhere), that is a single search.
If that user navigates to a second page of results, that is another search. Navigating to a third page counts as yet another search, etc.
Applying a filter (if the user interface supports such) counts as yet another search.
Re-sorting results counts as yet another search.
In a browser-based experience, even a user simply doing a page refresh counts as another search (though I will also say that in this case, if the interface uses some kind of caching of results, this might not actually truly retrieve a new set of results from the search engine, so this one could be a bit “squishy”).
In a user experience with an infinite scroll, the act of a user scrolling to the bottom of one ‘chunk’ of results and thus triggering the interface to retrieve the next ‘chunk’ also counts as yet another search (this is effectively equivalent to paging through result except it doesn’t require any action by the user).
The second basic measure is the click. A click is counted any time a user clicks on any results in the experience.
Depending on the implementation, differentiating the type of thing a user clicks on (an organic result or a ‘best bet’, etc.) can be useful – but I don’t consider that differentiation critical at the high level.
One thing to note here that I know is a gap – there are some scenarios where a user does not need to click on anything in the search results. The user might meet their information need simply by seeing the search results.
This could be because they just wanted to know if anything was returned at all. It could be because the information they need is visible right on the results screen (the classic example of this would be a search experience that shows people profiles and the display shows some pertinent piece of information like a phone number). In a sophisticated search experience that offers “answers” to question, the answer might be displayed right on the results screen. I have been puzzled about how to measure this scenario for a while. Other than some mechanism on the interface that allows a user to take some action to acknowledge that they achieved there need (“Was this answer useful?”), I’m not sure what that is. Very interested if others have solved this puzzle.
A third important metric is the search session. This is closely related to the search metric, but I do think that it is important to differentiate.
A search session is a series of actions a user takes that, together, constitute an attempt to satisfy a specific information need.
This definition, though, is really not deterministically measurable. There is no meaningful way (unless you can read the user’s mind) to know when they are “done”.
One possibility is to equate a search session to a visit – I find a good definition for this on Wikipedia in the Web analytics article:
A visit or session is defined as a series of page requests or, in the case of tags, image requests from the same uniquely identified client.
In the current solution I am working with, however, we have defined a search session to be a series of actions taken in sequence where the user does not change their search term. The user might navigate through a series of pages of results, reorder them, apply multiple filters, click on one or more results, etc., but, none of these count as another search session.
The rationale for this is that, based on anecdotal discussions with users, users tend to think of an effort using a single search term as a notional “search”. If the user fails with that term, they try another, but that is a different “search”.
Obviously, this is not truly accurate in all situations – if we could meaningfully detect (at scale, meaning across all of our activity) when changing the search term is really a restatement of the same information need vs. a completely different information need, we could do something more accurate, but we are not there, yet.
The last basic measure I propose is the first click.
A first click is counted the first time a user clicks on a result within a search session. If a user clicks on multiple things within a search session, they are all still counted as clicks, but not as first clicks.
If the user starts a new search session (which, in the current solution I work with, means they have changed their search term), then, if they click on some result, that is another first click.
That is the set of basic measures that I think could be useful to establish as a standard.
Next steps – I hope to engage with others working in this domain to refine these and tighten them up (especially a search session). I hope to make some contacts through the Enterprise Search Engine Professionals group on LinkedIn and perhaps other communities for this. If you are interested, please let me know!
In my next post, I will be sharing definitions of some important metrics derived from the basic measures above that I use and provide some examples of each.
In my last post, I wondered about the lack of meaningful standards for evaluating enterprise search implementations.
I did get some excellent comments on the post and also some very useful commentary from a LinkedIn discussion about this topic – I would recommend you read through that discussion. Udo Kruschwitz and Charlie Hull both provided links to some very good resources.
In this post, I thought I would describe what I think to be some important attributes of any standard measures that could be adopted. Here I will be addressing the specific actions to measure – in a subsequent post I will write about how these can be used to actually evaluate a solution.
Measurable
To state the obvious, we need to have metrics that are measurable and objective. Ideally, metrics that directly reflect user interaction with the search solution.
Measures that depend on subjective evaluation or get feedback from users through means other than their direct use of the tool can be very useful but introduce problems in terms of interpretation differences and sustainability.
For example, a feedback function built into the interface (“Are these results useful?” or even a more specific, “Is this specific result useful for you here?”) can provide excellent insight but are used so little that the data is not useful overall.
Surveys of users inevitably fall into the problem of faulty or biased memory – in my experience, users have such a negative perception of enterprise search that individual negative experiences will overwhelm positive experiences with the search when you ask them to recall and assess their experience a day or week after their usage.
Common / Useful to compare implementations
Another important consideration is that a standard for evaluating enterprise search should include aspects of search that are common across the broad variety of solutions you might see.
In addition, they should lend themselves to comparing different solutions in a useful way.
Some implementations might be web-based (in my experience, this is by far the most common way to make enterprise search available). Some might be based on a desktop application or mobile app. Some implementations might depend only on users enterprise search terms to start a search session; some might only support searching based on search terms (no filtering or refining at all). Some implementations might provide a “search as you type” (showing results immediately based on part of what the user has entered). Many variations to consider here.
I would want to have measures that allow me to compare one solution to another – “Is this one better than that one?” “Are there specific user needs where this solution is better than that one?”
Likely to be insightful
Another obvious aspect is that we want to include measures that are likely to be useful.
Useful in what way, though?
My first thought is that it must measure if the solution is useful for the users – does it meet the users’ needs? (With search, I would simplify this to “does it provide the information the user needs efficiently?” but there are likely a lot of other ways to define “useful” even within a search experience.
Operationalizable
I would want all measures I use to be consistently available (no need to “take a measurement” at a given time) and also to not depend on someone actively having to “take a measurement”.
As mentioned above, measures that directly reflect what happens in the user experience are what I would be looking for. In this case, I would add in that the measures should be taken directly from the user experience – data captured into a search log file somewhere or captured via some other means.
This provides a data set that can be reviewed and used at basically any time and which (other than maintaining the system capturing the measurements) don’t require any effort to capture and maintain – the users use the search solution and their activities are captured.
Usable for overall and when broken down by dimensions
Finally, I would expect that measures would support analysis at broad scales and also should support the ability to drill in to details and use the same measures?
Examples of “broad scale” applicability: How good is this search solution overall? How good is my search solution in comparison to the overall industry average? How good are search solutions supporting the needs of users in the XYZ industry? How good are search solutions at supporting “known item” searching in comparison with “exploratory searching”?
Examples of drilling in: Within my user base, how successful are my users by department? How useful is the search solution in different topic areas of content? How good are results for individual, specific search criteria?
Others?
I’m sure I am missing a lot of potential criteria here – What would you add? Remove? Edit?
Over the past several years of working very closely with the enterprise search solution at Deloitte, I have tried to look “outside” as best as I can in order to understand what others in the industry are doing to evaluate their solutions in order to understand where ours ‘fits’.
I’ve attended a number of conferences and webcasts and read papers (many, I’ll admit, that are highlighted by Martin White on Twitter. I can’t recommend a follow of Martin enough!)
One thing I have never found is any common way to evaluate or talk about enterprise search solutions. I have seen several people (including Martin) comment on the relatively little research on enterprise search (as opposed to internet search, which has a lot of research behind it), and I am sure a significant reason for that is that there is no common way to evaluate the solutions.
If we could compare in a systematic way, we could start to understand how to do things like:
Why do we not have a common set of definitions?
One possibility is certainly that I have still not read up enough on the topic – perhaps there is a common set of definitions – if so, feel free to share.
Another possibility is that this is a result of dependency on the metrics that are implemented within the search solutions enterprises are using. I have found that these are useful but they don’t come with a lot of detail or clarity of definition. And, more specifically, they don’t seem common across products. That said, I have relatively limited exposure to multiple search solutions – Again, I would be interested in insights from those who have (perhaps any consultants working in this space?)
And, one more possible driver behind a lack of commonality is the proprietary nature of most implementations. I try to speak externally as frequently as I can, but I am always hesitant (and have been coached) to not be too detailed on the implementation.
I do plan to put up a small series here, though, with some of the more elemental components of our metrics implementation for comparison with anyone who cares to share.
More soon!
Or, in other words, “How do you apply the application standards to improve findability to applications built by third-party providers who do not follow your standards?”
I’ve previously written about the standards I’ve put together for (web-based) applications that help ensure good findability for content / data within that application. These standards are generally relatively easy to apply to custom applications (though it can still be challenging to get involved with the design and development of those applications at the right time to keep the time investment minimal, as I’ve also previously written about).
However, it can be particularly challenging to apply these standards to third-party applications – For example, your CRM application, your learning management system, or your HR system, etc. Applying the existing standards could take a couple of different forms:
The rest of this post will discuss a solution for option #3 above – how you can implement a different solution. Note that some search engines will provide pre-built functionality to enable search within many of the more common third party solutions – those are great and useful, but what I will present here is a solution that can be implemented independent of the search engine (as long as the search engine has a crawler-based indexing function) and which is relatively minimal in investment.
So, you have a third party application and, for whatever reason, it does not adhere to your application standards for findability. Perhaps it fails the coverage principle and it’s not possible to adequate find the useful content without getting many, many useless items; or perhaps it’s the identity principle and, while you can find all of the desirable targets, they have redundant titles; or it might even be that the application fails the relevance principle and you can index the high value targets and they show up with good names in results but they do not show up as relevant for keywords which you would expect. Likely, it’s a combination of all three of these issues.
The core idea in this solution is that you will need a helper application that creates what I call “shadow pages” of the high value targets you want to include in your enterprise search.
Note: I adopted the use of the term “shadow page” based on some informal discussions with co-workers on this topic – I am aware that others use this term in similar ways (though I don’t think it means the exact same thing) and also am aware that some search engines address what they call shadow domains and discourage their inclusion in their search results. If there is a preferred term for the idea described here – please let me know!
What is a shadow page? For my purposes here, I define a shadow page as:
To make this solution work, there are a couple of minimal assumptions of the application. A caveat: I recognize that, while I consider these as relatively simple assumptions, it is very likely that some applications will still not be able to meet these and so not be able to be exposed via your enterprise search with this type of solution.
Given the description of a shadow page and the assumptions about what is necessary to support it, it is probably obvious how they are used and how they are constructed, but here’s a description:
First – you would use the query that gives you a list of targets (item #2 from the assumptions) from your source application to generate an index page which you can give your indexer as a starting point. This index page would have one link on it for each desirable target’s shadow page. This index page would also have “robots” <meta> tags of “noindex,follow” to ensure that the index page itself is not included as a potential target.
Second – The shadow page for each target (which the crawler reaches thanks to the index page) is dynamically built from the query of the application given the identity of the desirable search target (item #3 from the assumptions). The business rules defining how the desirable target should behave in search help define the necessary query, but the query would need to contain at minimum some of the following data: the name of the target, a description or summary of the target, some keywords that describe the target, a value which will help define the true URL of the actual target (per assumption #1, there must be a way to directly address each target).
The shadow page would be built something like the following:
The overall effect of this is that the search engine will index the shadow page, which has been constructed to ensure good adherence to the principles of enterprise search, and to a searcher, it will behave like a good search target but when the user clicks on it from a search result, the user ends up looking at the actual desired target. The only clue the user might have is that the URL of the target in the search results is not what they end up looking at in their browser’s address bar.
The following provides a simple example of the source (in HTML – sorry for those who might not be able to read it) for a shadow page (the parts that change from page to page are in bold):
<html> <head> <TITLE>title of target</TITLE> <meta name="robots" content="index, nofollow"> <meta name="keywords" content="keywords for target"> <meta name="description" content="description of target"> <script type="text/javascript"> document.location.href="URL of actual target"; </script> </head>
<body> <div style="display:none;"> <h1>title of target</h1> description of target and keywords of target </div> </body> </html>
A few things that are immediately obvious advantages of this approach:
There are also a number of issues that I need to highlight with this approach – unfortunately, it’s not perfect!
There you have it – a solution to the exposure of your high value targets from your enterprise applications that is independent of your search engine and can provide you (the search administrator) with a good level of control over how content appears to your search engine, while ensuring that what is included highly adheres to my principles of enterprise search.
So we get to the exciting conclusion of my essays on the inclusion of employees in enterprise search. If you’ve read this far, you know how I have characters the first and second generation solutions and also provided a description of a third generation solution (which included some details on how we implemented it).
Here I will describe what I think of as a fourth generation solution to people finding within the enterprise. As I mentioned in the description of the third generation solution, one major omission still at this point is that the only types of searches with which you can find people is through administrative information – things like their name, address, phone number, user ID, email, etc.
This is useful when you have an idea of the person you’re looking for or at least the organization in which they might work. What do you do when you don’t know the person and may not even know the organization in which they work? You might know the particular skills or competencies they have but that may be it. This problem is particularly problematic in larger organizations or organizations that are physically very distributed.
The core idea with this type of solution is to provide the ability to find and work with people based on aspects beyond the administrative – the skills of the people, their interests, perhaps the network of people with which they interact, and more. While this might be a simplification, I think of this as expertise location, though that, perhaps, most cleanly fits into the first use case described below.
Some common use cases for this type of capability include:
This capability is something that has often been discussed and requested at my current employer, but which no one has really been willing to sponsor. That being said, I know there are several vendors with solutions in this space, including (at least – please share if you know of others):
A common aspect of these is that they attempt to (and perhaps succeed) in automating the process of expertise discovery. I’ve seen systems where an employee has to maintain their own skill set and the problem with these is that the business process to maintain the data does not seem to really embed itself into a company – inevitably, the data gets out of date and is ill-maintained and so the system does not work.
I can not vouch for the accuracy of these systems but I firmly believe that if people search in the enterprise is going to meet the promise of enabling people to find each other and connect based on of-the-moment needs (skills, interests, areas of work, etc), it will be based on this type of capability – automatically discovering those aspects of a worker based on their work products, their project teams, their work assignments, etc.
I imagine within the not too distant future, as we see more merger of the “web 2.0” functionality into the enterprise this type of capability will become expected and welcome – it will be exciting to see how people will work together then.
This brings to a close my discussion of the various types of people search within the enterprise. I hope you’ve found this of interest. Please feel free to let me know if you think I have any omissions or misstatements in here – I’m happy to correct and/or fill in.
I plan another few posts that discuss a proof of concept I have put together based around the ideas of this fourth generation solution – look for those soon!
In my last post, I wrote about what I termed the first generation and second generation solution to people search in enterprise. This time, I will describe what I call a “third generation” solution to the problem that will integration people search with your enterprise search solution.
This is the stage of people search in use within my current employer’s enterprise.
What I refer to as a third generation solution for people search is one where an employee’s profile (their directory entry, i.e., the set of information about a particular employee) becomes a viable and useful target within your enterprise search solution. That is, when a user performs a search using the pervasive “search box” (you do have one, right?), they should be able to expect to find their fellow workers in the results (obviously, depending on the particular terms used to do the search) along with any content that matches that.
You remove the need for a searcher to know they need to look in another place (another application, i.e., the company’s yellow pages) and, instead, reinforce the primacy of that single search experience that brings everything together that a worker needs to do their job.
You also offer the full power of your enterprise search engine:
Below, you will find a discussion of the implementation process we used and the problems we encountered. It might be of use to you if you attempt this type of thing.
Before getting to that, though, I would like to discuss what I believe to be remaining issue with a third generation solution in order to set up my follow-up post on this topic, which will describe additional ideas to solving the “people finder” problem within an enterprise.
The primary issue with the current solution we have (or any similar solution based strictly on information from a corporate directory) is that the profile of a worker consists only of administrative information. That is, you can find someone based on their name, title, department, address, email, etc., etc., etc., but you can not do anything useful to find someone based on much more useful attributes – what they actually do, what their skills or competencies are or what their interests might be. More on this topic in my next post!
Read on from here for some insights on the challenges we faced in our implementation of this solution. It gets pretty detailed from here on out, so you’ve been warned!
Having written previously about my own principles of enterprise search and then some ideas on how to select a search engine, I thought it might be time to back up a bit and write about what I think of as “enterprise search”. Perhaps a bit basic or unnecessary but it gives some context to future posts.
For me, the factors of a search solution that make it an enterprise solution include the following:
The user interface to access the solution is available to all employees of the company.
This has the following implications:
The content available through the solution covers all (relevant) content available to employees
This has the following implications:
The other half of your enterprise search solution will be the search engine itself. There are plenty (many!) options available with a variety of strengths and weaknesses. I think if you plan to implement a truly enterprise search, the above list of content-based considerations should get you thinking of all of the places where you may have content “hiding” in your organization.
From that list, you should have a good sense of the volume of content and the complexity of sources your search will need to deal with.
Combining that with a careful requirements definition process and evaluation of alternatives should lead to a successful selection of a tool.
Once you have a tool, you “just” need to apply the proper amount of elbow grease to get it to index all of the content you wish and present it in a sensible way to your users! No big deal, right?