Lee Romero

My first post back after too-long a period of time off.  I wanted to jump back in and share some concrete thoughts on best bet governance.

I’ve previously written about best bets and how I thought, while not perfect, they were an important part of a search solution.  In that post, I also described the process we had adopted for managing best bets, which was a relatively indirect means supported by the search engine we used for the search solution.

Since moving employers, I now have responsibility for a local search solution as well as input on an enterprise search solution where neither of the search engines supports a similar model.  Instead, both support the (more typical?) model where you identify particular search terms that you feel need to have a best bet and you then need to identify a specific target (perhaps multiple targets) for those search terms.

This model offers some advantages such as specificity in the results and the ability to actively determine what search terms have a best bet that will show.

This model also offers some disadvantages, the primary one (in my mind) being that they must be managed – you must have a means to identify which terms should have best bets and which targets those terms should show as a best bet.  This implies some kind of manual management, which, in resource-constrained environments, can be a challenge.  As noted in my previous article, others have provided insight about how they have implemented and how they manage best bets.

Now having responsibility for a search solution requiring manual management of best bets, we’ve faced the same questions of governance and management and I thought I would share the governance model we’ve adopted.  I did review many of the previous writings on this to help shape these, so thanks to those who have written before on the topic!

Our governance model is largely based on trying to provide a framework for consistency and usability of our best bets.  We need some way to ensure we do not spend inordinate time on managing requests while also ensuring that we can identify new, valuable search terms and targets for best bets.

Without further ado, here is an overview of the governance we are using:

• We will accept best bet requests from all users, though most requests come from site publishers on our portal.  Most of our best bets have web sites as targets, though about 30% have individual pieces of published content (documents) as targets.  As managers of the search solution, my team will also identify best bets when appropriate.
• When we receive a request for a new best bet, we review the request against the following the following criteria:
  • No more than five targets can be identified for any one search term, though we prefer to keep it to one or two targets.
    • Any request for a best bet that would result in more than 2 targets for the search term forces a review of usage of the targets (usage is measured by our web analytics solution for both sites and published content).
    • The overall usage of the targets will identify if one or more targets should be dropped.
  • For a given target, no more than 20 individual search terms can be identified.  Typically, we try to keep this to fewer than 5 when possible.
  • If a target is identified as a best bet target that has not had a best bet search term associated with it previously, we confirm that it is either a highly used piece of content or that it is a significant new piece that is highly known or publicized (or may soon be by way of some type of marketing).
  • We also review the search terms identified for the best bet.  We will not use search terms with little to no usage during the previous 3 months.
  • We will not set up a best bet search term that matches the title of the target.  The relevancy algorithm for our search engine heavily weights titles, so this is not necessary.
  • We do prefer that the best bet search terms do have a logical connection to the title or summary of the target.  This ensures that a user will understand the connection between their search terms and a resulting best bet.  This is not a hard requirement, but a preference.  We do allow for spelling variants, synonyms, pluralized forms, etc.
  • We prefer terms that use words from our global taxonomy.
• Our governance (management process, really) for managing best bets includes:.
  • Our search analyst reviews the usage of each best bet term.
    • If usage over an extended time is too low to warrant the best bet term, it is removed.
  • We also plan to use path analysis (pending some enhancements needed as this is written) to determine if, for specific terms, the best bet selections are used preferentially.  If that is found to not be the case, our intent is that the best bet target is removed.
  • We have integrated the best bet management into both our site life cycle process and our content life cycle
    • With the first, when we are retiring a site or changing the URL of a site we know to remove or update the best bet target
    • With the second, as content is retired, the best bets are removed
    • In each of these cases, we also evaluate the terms to see if there could be other good targets to use.

The one interesting experience we’ve had so far with this governance model is that we get a lot of push back from site publishers who want to provide a lengthy laundry list of terms for their site, even when 75% of that list is never used (or at least in a twelve month period we’ll sometimes check).  They seem convinced that there is value in setting up best bets for terms even when you can show that there is none.  We are currently making changes in the way we manage best bets and also in how we can use these desirable terms to enhance the organic results directly.  More on that later.

There you have our current governance model.  Not too fancy or complicated and still not ideal, but it’s working for us and we recognize that it’s a work in progress.

Now that I have the “monkey off my back” in terms of getting a new post published, I plan to re-start regular writing.  Check back soon for more on search, content management and taxonomy!

The title of this post – “People know where to find that, though!” is a very common phrase I hear as the search analyst and the primary search advocate at my company. Another version would be, “Why would someone expect to find that in our enterprise search?”

Why do I hear this so often? I assume that many organizations, like my own, have many custom web applications available on their intranet and even their public site. It is because of that prevalence, combined with a lack of communication between the Business and the Application team, that I hear these phrases so often.

I have (unfortunately!) lost count of the number of times a new web-based application goes into production without anyone even considering the findability of the application and its content (data) within the context of our enterprise search.

Typically, the conversation seems to go something like this:

• Business: “We need an application that does X, Y and Z and is available on our web site.”
• Application team: “OK – let’s get the requirements laid out and build the site. You need it to do X, Y and Z. So we will build a web application that has page archetypes A, B and C.”
• Application team then builds the application, probably building in some kind of local search function – so that someone can find data once they are within the application.
• The Business accepts the usability of the application and it goes into production

What did we completely miss in this discussion? Well, no one in the above process (unfortunately) has explicitly asked the question, “Does the content (data) in this site need to be exposed via our enterprise search?” Nor has anyone even asked the more basic question, “Should someone be able to find this application [the “home page” of the application in the context of a web application] via the enterprise search?”

• Typically, the Business makes the (reasonable) assumption that goes something like, “Hey – I can find this application and navigate through its content via a web browser, so it will naturally work well with our enterprise search and I will easily be able to find it, right?!”
• On the other hand, the Application Team has likely made 2 assumptions: 1) the Business did not explicitly ask for any kind of visibility in the enterprise search solution, so they don’t expect that, and 2) they’ve (likely) provided a local search function, so that would be completely sufficient as a search.

I’ve seen this scenario play out many, many times in just the last few years here. What often happens next depends on the application but includes many of the following symptoms:

• The page archetypes designed by the Application Team will have the same (static) <title> tag in every instance of the page, regardless of the data displayed (generally, the data would be different based on query string parameters).
  • The effect? A web-crawler-based search engine (which we use) likely uses the <title> tag as an identifier for content and every instance of each page type has the same title, resulting in a whole lot of pretty useless (undifferentiated) search results. Yuck.
• The page archetypes have either no or maybe redundant other metadata – keywords, description, content-date, author, etc.
  • The effect? The crawler has no differentiation based on <titles> and no additional hints from metadata. That is, lousy relevance.
• The application has a variety of navigation or data manipulation capabilities (say, sorting data) based on standard HTML links.
  • The effect? The crawler happily follows all of the links – possibly (redundant) indexing the same data many, many times simply sorted on different columns.
  • Another effect? The dreaded calendar affect – the crawler will basically never stop finding new links because there’s always another page.
  • In either case, we see poor coverage of the content.

The overall effect is likely that the application does not work well with the enterprise search, or possibly that the application is that the application does not hold up to the pressure of the crawler hitting its pages much faster than anticipated (so I end up having to configure the crawler to avoid the application) and ending with yet another set of content that’s basically invisible in search.

Bringing this back around to the title – the response I often get when inquiring about a newly released application is something like, “People will know how to find that content – it’s in this application! Why would this need to be in the enterprise search?”

When I then ask, “Well, how do people know that they even need to navigate to or look in this application?” I’ll get a (virtual) shuffling of feet and shoulder shrugs.

All because of a perpetual lack of asking a few basic questions during a requirements gather stage of a project or (another way to look at it) lack of standards or policies which have “teeth” about the design and development of web application. The unfortunate thing is that, in my experience, if you ask the questions early, it’s typically on the scale of a few hours of a developer’s time to make the application work at least reasonably well with any crawler-based search engine. Unfortunately, because I often don’t find out about an application until after it’s in production, it then becomes a significant obstacle to get any changes made like this.

I’ll write more in a future post about the standards I have worked to establish (which are making some headway into adoption, finally!) to avoid this.

Edit: I’ve now posted the standards as mentioned above – you can find them in my post Standards to Improve Findability in Enterprise Applications.