Lee Romero

What is a Search Analyst?

Having written about what I consider to be the principles of enterprise search, about people search in the enterprise, about search analytics and several other topics related to search in some detail, I thought I would share some insights on a role I have called search analyst – the person(s) who are responsible for the care and feeding of an enterprise search solution. The purpose of this post is to share some thoughts and experiences and help others who might be facing a problem similar to what my team faced several years back – we had a search solution in place that no one was maintaining and we needed to figure out what to do to improve it.

Regarding the name of the role – when this role first came into being in my company, I did not know what to call the role, exactly, but we started using the term search analyst because it related to the domain (search) and reflected the fact that the role was detailed (analytical) but was not a technical job like a developer. Subsequently, I’ve heard the term used by others so it seems to be fairly common terminology now – it’s possible that by now I’ve muddled the timeline enough in my head that I had heard the term prior to using it but just don’t recall that!

What does a Search Analyst do?

What does a search analyst do for you? The short answer is that a search analyst is the point person for improving the quality of results in your search solution. The longer answer is that a search analyst needs to:

• Review data related to your search solution and understand its implications
• Formulate new questions to continually improve upon the insights gained from the data
• Formulate action plans from insights gained from monitoring that data in order to continually improve your search solution – this requires that the search analyst understand your search solution at a deep enough level of understand to be able to translate analytic insights into specific changes or actions
• Follow through on those action plans and work with others as necessary to effect the necessary changes

Measuring Success as a Search Analyst

In order to define success for a search analyst, you need to set some specific objectives for the search analyst(s). Ultimately, given the job description, they translate to measuring how the search analyst has been successful in improving search, but here are some specific suggestions about how you might measure that:

• Execute a regular survey of users of your search (perhaps annually?) – this can be a very direct way of measuring increased quality, though ensuring you get good coverage of the target audience (and reflect appropriate demographics) may be a challenge. We have used this and results do reflect increases in satisfaction.
• Provide ability to rate search results – a more direct way than a survey to measure satisfaction with search, though implementing it and integrating it with the search experience in a way that invites users to provide feedback can be a challenge.
• Measure overall increase in search usage – No need to directly work with users of your search but also begs the question about whether increasing search usage is really a measure of quality.
• Measure increase in search usage relative to visits to your site (assuming your search solution is integrated with your intranet, for example) – I mentioned this in post on advanced metrics as a metric to monitor. I think this can be more useful than just measuring increases in usage, however, it might also reflect changes (good or bad) in navigation as much as changes in search.
• Measure overall coverage of search (total potential targets) – How much content does your search solution make available as potential search results? By itself, increases in this do not equate to an improvement in search but if combined with other metrics that more directly measure quality of results, increases in coverage do translate to a user being more likely to get what they need from search. In other words, if you can assure users that they can gain direct access to more potential results in search while also ensuring that the quality of results returned is at least as good as before, that’s a good thing. On the other hand, if adding in new content pollutes the experience with many less-relevant search results, you are not doing anyone any favors by including them.
• Measure number of specific enhancements / changes made to improve the quality of results – especially for highly sought content. Assuming you track the specific changes made, a measure of effectiveness could be to track how many changes a search analyst has made over a given time period. Did the search analyst effect 5 changes in a month? 50? Again, the number itself doesn’t directly reflect improvements (some of those changes could have been deleterious to search quality) but it can be an indicator of value.

Time Commitment for a Search Analyst

Another common question I’ve received is what percentage of time should a search analyst expect to spend on this type of work? Some organizations may have large enough search needs to warrant multiple full-time people on this task but we are not such an organization and I suspect many other organizations will be in the same situation. So you might have someone who splits their time among several roles and this is just one of them.

I don’t have a full answer to the question because, ultimately, it will depend on the value your organization does place on search. My experience has been that in an organization of approximately 5-6,000 users (employees) covering a total corpus of about a million items spread across several dozen sites / applications / repositories, spending about .25 FTE on search analyst tasks seems to provide for steady improvements and progress.

Spending less than that (down to about .1 FTE), I’ve found, results in a “steady state” – no real improvements but at least the solution does not seem to degrade. Obviously, spending more than that could result in better improvements but I find that dependence on others (content owners, application owners, etc.) can be a limiting factor in effectiveness – full organizational support for the efforts of the search analyst (giving the search analyst a voice in prioritization of work) can help alleviate that. (A search analyst with a software development background may find this less of an issue as, depending on your organization, you may find yourself less tied to development resources than you would otherwise be, though this also likely raises your own FTE commitment.)

The above description is worded as if your organization has a single person focused on search analyst responsibilities. It might also be useful to spread the responsibility among multiple people. One reason would be if your enterprise’s search solution is large enough to warrant a team of people instead of a single person. A second would be that it can be useful to have different search analysts focused (perhaps part time still for each of them) on different content areas. In this second situation, you will want to be careful about how “territorial” search analysts are, especially in the face of significant new content sources (you want to ensure that someone takes on whatever responsibility there might be for that content in regards to ensuring good findability).

What Skills does a Search Analyst Need

So far I’ve provided a description of the role of a search analyst, suggestions for objectives you can assign to a search analyst and at least an idea of the time commitment you might expect to have an effective search analyst. But, if you were looking to staff such a position, what kinds of skills should you look for? Here are my thoughts:

• First, I would expect that a search analyst is a capable business analyst. I would expect that anyone who I would consider a capable search analyst would be able to also work with business users to elicit, structure and document requirements in general. I would also expect a search analyst to be able to understand and document business processes in general. Some other insights on a business analyst’s skills can be found here and here.
• I would also expect that a search analyst should be naturally curious and knows how to ask the right questions. Especially with regard to the exploratory nature of dealing with a lot of analytical data (as seen in my recent posts about search analytics).
• A search analyst must be very capable of analyzing data sets. Specifically, I would expect a search analyst to be very proficient in using spreadsheets to view large data collections – filtering, sorting, formulae, pivot tables, etc. – in order to understand the data they’re looking at. Depending on your search solution, I would also expect a search analyst to be proficient with building SQL queries; ideally they would use reports built in a reporting system (and so not have to directly manipulate data using SQL) but I find that the ad hoc / exploratory nature of looking at data makes that hard.
• I would expect a search analyst to have an understanding of taxonomy in general and, specifically, understands your organization’s taxonomy and its management processes. This is important because the taxonomy needs to be an input into their analysis of search data and also (as highlighted in the potential actions taken from insights from search analytics), many insights can be gained from a search analyst that can influence your taxonomy.
• I would also look for a search analyst to understand information architecture and how it influences navigation on your organization’s web sites. As with the taxonomy, I find that the search analyst will often discover insights that can influence your navigation.
• I would expect a search analyst to have some understanding in basic web technologies. Most especially HTML and the use of meta tags within it. Also, XML is important (perhaps moreso, depending on your search engine). Some understanding of JavaScript (at least in so far as how / if your engine deals with it) can be useful.
• I would expect that a search analyst should be able to quickly learn details of computer systems – specifically, how to manage and administer your search solution. I would not be hung up on whether your search analyst already knows the specific engine you might be using but that can obviously be useful.
• This is not a skill, but another important piece of knowledge your search analyst should have is a good understanding of your major content sources and content types. In general, what kinds of things should be expected to be found in what places? What formats? What kinds of processes are behind their maintenance?
• This is also not a skill per se, but it is important for your search analyst to be connected to content managers and application teams. The connection might be relatively tight (working in a group with them) or loose (association via a community of practitioners in your organization). The reasons for this suggestion include:
  • The ability to easily have two way communication with content managers enables your search analyst to provide continuous education to content managers about the importance of their impact on findability (education about good content tagging, how content will show in search, etc.) and also enables content managers to reach out to a search analyst when they are trying to proactively improve search of their content (something which does not seem to be as likely as I’d like to see within an enterprise setting!).
  • The ability to communicate with development teams can help in similar ways: The search analyst can use that as a way to continually reinforce the need for developers to consider findability when applications are deployed. Also, connectivity with development teams can provide insights to the search analyst so that they can proactively inject themselves in the testing of the applications (or hopefully even in the requirements definition process!) to ensure findability is actually considered.
• Given that last recommendation, it is also important that a search analyst be able to communicate effectively and also be comfortable in teaching others (formally or informally). I find that education of others about findability is a constant need for a search analyst.

If your search needs warrant more than one person focused on improving your enterprise search solution, as much overlap in the above as feasible is good, though you may have team members specializing in some skills while others focus on other areas.

Organizational location of search analyst

Another important issue to address is where in your overall organization should the search analyst responsibility rest? I don’t have a good answer for this question and am interested in others’ opinions. My own experiences:

• Originally, we have this responsibility falling on the heads of our search engine engineers. Despite their best efforts, this was destined to not be effective because their focus was primarily on the engine and they didn’t have enough background in things like the content sources, applications or repositories to include, connectivity to content managers or application developers. They primarily just ensured that the engine was running and would make changes reactively when someone contacted them about an issue.
• We moved this responsibility into our knowledge management group – I was a trigger for this move as I could see that no one else in the organization was going to “step up”.
• Due to subsequent organizational changes, this responsibility then fell into the IT group.
• At this point, I would suggest that the best fit in our organization was within the KM group.
  • A search analyst is not a technical resource (developer or system admin, for example) though the job is very similar to business analysts that your IT group might have on staff.
  • The real issue I have found with having this responsibility fall into the IT organization is that within many organizations, IT is an organization that is responsive to the business and not an organization that drives business processes or decisions. Much of what the search analyst needs to accomplish will result in IT driving its own priorities, which can present challenges – the voice of the search analyst is not listened to within IT because it’s not coming “from the business”.
  • Also, it can be a challenge for an IT group to position a search analyst within it in order to support success. The internal organization of IT groups will vary so widely I can’t make any specific suggestions here, but I do believe that if your search analyst is located within your IT group, a search analyst could be closely aligned to a group focused on either architecture or business intelligence and be successful.
• If your organization is structured to have a specific group with primary responsibility for your web properties (internal or external), that group would also be a potential candidate for positioning this responsibility. If that group primarily focuses externally, you would likely find that a search analyst really plays more of an SEO role than being able to focus on your enterprise search solution.

Enough about my own insights – What does anyone else have to share about how you perceive this role?   Where does it fit in your organization?  What are your objectives for this role?