Lee Romero

On Content, Collaboration and Findability
January 18th, 2021

Evaluating enterprise search – standards?

Over the past several years of working very closely with the enterprise search solution at Deloitte, I have tried to look “outside” as best as I can in order to understand what others in the industry are doing to evaluate their solutions in order to understand where ours ‘fits’.

I’ve attended a number of conferences and webcasts and read papers (many, I’ll admit, that are highlighted by Martin White on Twitter. I can’t recommend a follow of Martin enough!)

One thing I have never found is any common way to evaluate or talk about enterprise search solutions. I have seen several people (including Martin) comment on the relatively little research on enterprise search (as opposed to internet search, which has a lot of research behind it), and I am sure a significant reason for that is that there is no common way to evaluate the solutions.

If we could compare in a systematic way, we could start to understand how to do things like:

  • Identify common use cases that are visible in user behavior (via metrics)
  • Compare how ‘good’ different solutions are at meeting the core need (an employee needs to access some resource to do their job)
  • Compare different industries approaches to information seeking (again, as identified by user behavior via metrics) – for example, do users search differently in industrial companies vs. professional services companies vs. research companies?

Why do we not have a common set of definitions?

One possibility is certainly that I have still not read up enough on the topic – perhaps there is a common set of definitions – if so, feel free to share.

Another possibility is that this is a result of dependency on the metrics that are implemented within the search solutions enterprises are using. I have found that these are useful but they don’t come with a lot of detail or clarity of definition. And, more specifically, they don’t seem common across products. That said, I have relatively limited exposure to multiple search solutions – Again, I would be interested in insights from those who have (perhaps any consultants working in this space?)

And, one more possible driver behind a lack of commonality is the proprietary nature of most implementations. I try to speak externally as frequently as I can, but I am always hesitant (and have been coached) to not be too detailed on the implementation.

I do plan to put up a small series here, though, with some of the more elemental components of our metrics implementation for comparison with anyone who cares to share.

More soon!

2 Responses to “Evaluating enterprise search – standards?”

  1. Very well presented, and it is a problem I have been trying to solve for 20 years. At one stage there was a ‘Metrics Club’ set up by a group of European companies in a particular vertical sector (details are confidential) which foundered because of concerns about both data sharing and because it might look as though the companies concerned were working together against the competition rules of the EU.Conference presentations about ES are always poor, mainly because most search managers know deep down that search is not working (as the IntraTeam survey shows) and do not want to display the problems they have to the general public, in particular to their competitors and shareholders.

    For some years Findwise ran a global survey of enterprise search implementations but had a relatively more uptake. There was also an AIIM report back in 2014 but that has never been repeated. The questions asked were pretty much spot on.

    Some research has been published that looks at professional search, headed up by Tony Russell-Rose and Leif Azzopardi.

    As it happens I am giving a workshop to IntraTeam in Denmark on 28 Jan which is focused on the challenges of collecting and interpreting search analytics. I’d be happy to write this up as a briefing paper to start the discussion going, especially about the challenges of collating consistent meaningful data!

    I could write for ever on this topic but it’s time to stop and calm down!

  2. As a followup to my polemic above I do have a benchmark metrics service called SearchCheck which ran as a very successful workshop at KMWorld in November. http://intranetfocus.com/searchcheck-service/ I’m happy to admit that take up has been weak despite a launch fee which was negligible. Which seems to suggest that no one wants to find out how good or bad their enterprise search application is performing.

Leave a Reply