Lee Romero

Archive for February, 2021

Having introduced some basic, standard, definitions in my previous post, in this one I am going to propose some standard measures derived from those that enable comparisons across solutions. These also are extremely useful for individual solutions where you, as an enterprise search manager, might want to have tools at hand to proactively improve your users’ experience.

A quick recap of what I defined before:

• Search: A single action a user takes that retrieves a set of results. Initiating a searching effort, applying a sort to result, pagination, applying filters would typically all increment this metric.
• Click: A user clicking on a result presented to them.
• Search Session: A sequence of actions (clicks or searches) that are taken in order without changing the search term (more generally, the criteria of the search).
• First Click: The first click within a search session.

Lost Clicks

The first derived measure is one I call “lost clicks”. This measures the raw number of search sessions that resulted in no click:

This is a useful measure that tells you how many times, in total, users initiated a session but found nothing of interest to click on.

You can also think of this as an indicator that measures the number of total failed search sessions.

One more point I’ll make on this is that, because it is a raw number (not a ratio or percentage), it is not useful as a key performance indicator (KPI).

Abandonment rate

Now, finally, to my proposal for a standard measure of the quality of a search solution – a measure that, I think, can be usefully applied to all enterprise search solutions, can be used to drive improvement within a solution, and can be used to compare across such solutions.

That measure is “abandonment rate”, which I define as the percent of sessions that are ‘failed sessions’:

which, after a bit of simplifying, I normally write as:

This measure has some important advantages over a simpler click-rate model (e.g., [success rate] = [click] / [search]). For one thing, it avoids some simple problems that can be caused by a few anomalous users; for a second, it avoids the ‘trap’ of assuming a click is a success.

Anomalous usage patterns

There are two anomalous patterns I see every once in a while:

1. A single dedicated user (or a small number of such users) might page through dozens or hundreds of pages of results (I actually have seen this before!) – generating a LOT of search actions – and yet click on nothing or just a result or two.
   • If every other user found something interesting to click on and did so on the first page of results, the click rate is still artificially lowered by these “extra” searches.
2. Inversely, users who are in a ‘research mode’ of usage (not a known item search) will click on a lot of results (I have also seen instances where a single user clicks on 100s of results all in the same search session).
   • Even if no other user found anything interesting to click on, the click rate is still artificially raised by these “extra” clicks.

By using only the first click and also the search session as the denominator, these scenarios don’t come into play (note that because I am recommending still capturing the simpler ‘search’ and the simpler ‘click’ metrics, you can still do some interesting analyses with these!).

Bad Success and Good Abandonment

The second advantage I mentioned above is more of a philosophical one – the success rate measure as defined builds in more strongly that you are measuring user success. This is a strong statement.

By focusing on abandonment, I find it a more honest view – your metrics don’t build in an assumption that a click is likely a success but, instead, that a failure to find something of interest to click on is more clearly an indication of likely failure.

What do I mean?

When I consider the ideas of “success” and “failure” in a search solution, I always have to remind myself of the good and bad sides of both – what do I mean by that??

• Good success – Good success is a click on a result that was actually useful and what the user needs to do their job. This, ultimately, is what you want to get to – however, because there is no way for a search solution to (at scale) know if any given result is “good” or “useful”, this is impossible to really measure.
• Bad abandonment – This is the flip side – this is how I think of the experience where a user has a search session where they find nothing useful at all. Again, this is the clear definition of failure.

However, there are other possibilities to consider!

• Bad success – This is when a user finds something that appears to be useful or what they need and they click on it, but it turns out to be something entirely different and not useful at all.
  • A classic example of bad success I have seen is in regard to my firm’s branding library (named ‘Brand Space’). For whatever reason, many intranet managers like to create image libraries in their sites and name them ‘Brand Space’ (I think this is because they think of this image library as their own instance of ‘Brand Space’). They then leave that image library exposed in search (we train them not to do so, but sometimes they don’t listen) and if an end user initiates a search session looking for Brand Space, they find the image library in results, click on it, and are likely disappointed (I imagine such a user thinking, “What is this useless web page?”)
  • A different way to think of this is in regard to the perspective of someone who is responsible for a particular type of content (let’s say benefits information for your company) – they may think they know what users *should* access when they search in particular ways and clicking on anything else is an instance of ‘bad success’. I get this but, as the manager of the search solution, I am not in the position of defining what users *should* click on – I cannot read their minds to understand intent.
• Good abandonment – This is when a user finds the information they need right on the search results screen. Technically, such a session would count as ‘abandoned’ even though the user got what they needed.
  • This is exactly the scenario I mentioned in the definition of a ‘click’ in my last post where I would like to define how to measure this but have never been able to figure out a way to do so.

Getting back to my description of how measuring and tracking abandonment rate is better then a success rate – my assumption has been that good abandonment and bad success will always exist for your users, however, good abandonment is likely a much smaller percentage of sessions than bad success and, more importantly, it is much easier to “improve” your search by increasing bad success then decreasing good abandonment.

Conclusion

There is my proposal for a measure to be used to assess search solutions for the quality of the user experience – abandonment rate.

It is not perfect and it is still just an indicator but I have found it incredibly useful to actually drive action for improvement. I’ll share more on this in my next post.

In my last few posts, I have commented on the lack of standard measures to use for enterprise search (leading to challenges of comparing various solutions to others among other things) and suggested some criteria for what standard measures to use.

In this post, I am going to propose a few basic measures that I think meet the criteria and that any enterprise search solution should be able to provide. The labels are not critical for these, but the meaning of them is, I think, very important.

Search

First, and most important, is a search. A search is a single action in which a user retrieves a set of results from the search engine. Different user experiences may “count” these events differently.

When a user starts the process (in my experience, typically with a search term typed into a box on a web page somewhere), that is a single search.

If that user navigates to a second page of results, that is another search. Navigating to a third page counts as yet another search, etc.

Applying a filter (if the user interface supports such) counts as yet another search.

Re-sorting results counts as yet another search.

In a browser-based experience, even a user simply doing a page refresh counts as another search (though I will also say that in this case, if the interface uses some kind of caching of results, this might not actually truly retrieve a new set of results from the search engine, so this one could be a bit “squishy”).

In a user experience with an infinite scroll, the act of a user scrolling to the bottom of one ‘chunk’ of results and thus triggering the interface to retrieve the next ‘chunk’ also counts as yet another search (this is effectively equivalent to paging through result except it doesn’t require any action by the user).

Click

The second basic measure is the click. A click is counted any time a user clicks on any results in the experience.

Depending on the implementation, differentiating the type of thing a user clicks on (an organic result or a ‘best bet’, etc.) can be useful – but I don’t consider that differentiation critical at the high level.

One thing to note here that I know is a gap – there are some scenarios where a user does not need to click on anything in the search results. The user might meet their information need simply by seeing the search results.

This could be because they just wanted to know if anything was returned at all. It could be because the information they need is visible right on the results screen (the classic example of this would be a search experience that shows people profiles and the display shows some pertinent piece of information like a phone number). In a sophisticated search experience that offers “answers” to question, the answer might be displayed right on the results screen. I have been puzzled about how to measure this scenario for a while. Other than some mechanism on the interface that allows a user to take some action to acknowledge that they achieved there need (“Was this answer useful?”), I’m not sure what that is. Very interested if others have solved this puzzle.

Search Session

A third important metric is the search session. This is closely related to the search metric, but I do think that it is important to differentiate.

A search session is a series of actions a user takes that, together, constitute an attempt to satisfy a specific information need.

This definition, though, is really not deterministically measurable. There is no meaningful way (unless you can read the user’s mind) to know when they are “done”.

One possibility is to equate a search session to a visit – I find a good definition for this on Wikipedia in the Web analytics article:

A visit or session is defined as a series of page requests or, in the case of tags, image requests from the same uniquely identified client.

In the current solution I am working with, however, we have defined a search session to be a series of actions taken in sequence where the user does not change their search term. The user might navigate through a series of pages of results, reorder them, apply multiple filters, click on one or more results, etc., but, none of these count as another search session.

The rationale for this is that, based on anecdotal discussions with users, users tend to think of an effort using a single search term as a notional “search”. If the user fails with that term, they try another, but that is a different “search”.

Obviously, this is not truly accurate in all situations – if we could meaningfully detect (at scale, meaning across all of our activity) when changing the search term is really a restatement of the same information need vs. a completely different information need, we could do something more accurate, but we are not there, yet.

First Click

The last basic measure I propose is the first click.

A first click is counted the first time a user clicks on a result within a search session. If a user clicks on multiple things within a search session, they are all still counted as clicks, but not as first clicks.

If the user starts a new search session (which, in the current solution I work with, means they have changed their search term), then, if they click on some result, that is another first click.

Conclusion and what’s next

That is the set of basic measures that I think could be useful to establish as a standard.

Next steps – I hope to engage with others working in this domain to refine these and tighten them up (especially a search session). I hope to make some contacts through the Enterprise Search Engine Professionals group on LinkedIn and perhaps other communities for this. If you are interested, please let me know!

In my next post, I will be sharing definitions of some important metrics derived from the basic measures above that I use and provide some examples of each.