In my last post, I wrote about a particular process for capturing “knowledge nuggets” from a community’s on-going discussions and toward the end of that write up, I described some ideas for the motivation for members to be involved in this knowledge capture process and how it might translate to an enterprise. All of the ideas I wrote about were pretty general and as I considered it, it occurred to me that another topic is – what are the kinds of specific objectives an employee could be given that would (hopefully) increase knowledge sharing in an enterprise? What can a manager (or, more generally, a company) do to give employees an incentive to share knowledge?
Instead of approaching this from the perspective of what motivates participants, I am going to write about some concrete ideas that can be used to measure how much knowledge sharing is going on in your organization. Ultimately, a company needs to build into its culture and values an expectation of knowledge sharing and management in order to have a long-lasting impact. I would think of the more tactical and concrete ideas here as a way to bootstrap an organization into the mindset of knowledge sharing.
A few caveats: First – Given that these are concrete and measurable, they can be “gamed” like anything else that can be measured. I’ve always thought measures like this need to be part of an overall discussion between a manager and an employee about what the employee is doing to share knowledge and not (necessarily) used as absolute truth.
Second – A knowledge sharing culture is much more than numbers – it’s a set of expectations that employees hold of themselves and others; it’s a set of norms that people follow. That being said, I do believe that it is possible to use some aspects of concrete numbers to understand impacts of knowledge management initiatives and to understand how much the expectations and norms are “taking hold” in the culture of your organization. Said another way – measurement is not the goal but if you can not measure something, how do you know its value?
Third – I, again, need to reference the excellent guide, “How to use KPIs in Knowledge Management” by Patrick Lambe. He provides a very exhaustive list of things to measure, but his guide is primarily written as ways to measure the KM program. Here I am trying to personalize it down to an individual employee and setting that employee’s objectives related to knowledge sharing.
In the rest of this post, I’ll make the assumption that your organization has a performance management program and that that program includes the definition for employees of objectives they need to complete during a specific time period. The ideas below are applicable in that context.
Not all of these will apply to all employees and some employees may not have any specific, measurable knowledge sharing objectives (though that seems hard to imagine regardless of the job). An organization should look at what they want to accomplish, what their tool set will support (or what they’re willing to enhance to get their tool set to support what they want) and then be specific with each employee. This is meant only as a set of ideas or suggestions to consider in making knowledge sharing an explicit, concrete and measurable activity for your employees.
Given some concrete objectives to measure employees with, it seems relatively simply to roll those objectives up to management to measure (and set expectations for up front) knowledge sharing by a team of employees, not just individual employees. On the other hand, a forward-thinking organization will define group-level objectives which can be cascaded down to individual employees.
Given either of these approaches, a manager (or director, VP, etc.) may then have both an organizational level objective and their own individual objectives related to knowledge sharing.
Lastly – while I’ve never explored this, several years ago, a vice president at my company asked for a single index of knowledge sharing. I would make the analogy of some like a stock index – a mathematical combination of measuring different aspects of knowledge sharing within the company. A single number that somehow denotes how much knowledge sharing is going on.
I don’t seriously think this could be meaningful but it’s an interesting idea to explore. Here are some definitions I’ll use to do so:
Given the above, you could imagine the “knowledge sharing index” at any moment in time could be computed as (for – I don’t know how to make this look like a “real” formula!):
Knowledge index at time t = Sum (i=1…N) of Wi * ( Mt,i / Bi )
A specific example:
- Let’s say you have three sources of “knowledge sharing” – a corporate wiki, a mailing list server and a corporate knowledge base
- For the wiki, you’ll measure total edits every week, for the list server, you’ll measure total posts to all mailing lists on it and for the knowledge base, you’ll measure contributions and downloads (as two measures).
- In terms of weights, you want to give the mailing lists the least weight, the wiki an intermediate weight and the combined knowledge base the most weight. Let’s say the weights are 15 for the mailing lists, 25 for the wiki, 25 for the downloads from the knowledge base and 35 for contributions to the knowledge base. (So the weights total to 100!)
- Your baseline for future measurement is 200 edits in the wiki, 150 posts to the list server, 25 contributions to the knowledge base and downloads of 2,000 from the knowledge base
- At some week after the start, you take a measurement and find 180 wiki edits, 160 posts to the list server, 22 knowledge base contributions and 2200 downloads from the knowledge base.
- The knowledge sharing index for that week would be 95.8. This is “down” even though most measures are up (which simply reflects the relative importance of one factor, which is down).
If I were to actually try something like this, I would pick the values of Wi so that the baseline measurement (when t= 0) comes to a nice round value – 100 or something. You can then imagine reporting something like, “Well, knowledge sharing for this month is at 110!” Or, “Knowledge sharing for this month has fallen from 108 to 92″. If nothing else, I find it amusing to think so concretely in terms of “how much” knowledge sharing is going on in an organization.
There are some obvious complexities in this idea that I don’t have good answers for:
In any event – I think this is an interesting, if academic, discussion and would be interested in others’ thoughts on either individual performance management or the idea of a knowledge sharing index.
Recently, there was a thread of discussion on the com-Prac list about the “death of a community” and a follow-up discussion about what or how CoPs should capture discussion-produced knowledge.
I found these to be very interesting and thought-provoking discussions. In this post, I will write about two aspects of these discussions – the retiring of a community and also a case study in how a community centered around a mailing list meets the challenge of knowledge capture.
Before getting into the details – I wanted to (re-)state that I recognize that a community is (much) more than a mailing list – community members interact in many ways, some online, some in “real space”. That being said, I also know that for many communities the tool of choice for group communication is a mailing list, so in this post, I will write about issues related to the use of mailing lists, though the ideas can be transferred to other means of electronic exchange. As John D. Smith notes in the second thread:
“All of the discussion about summarization so far assumes that a community almost exclusively lives on one platform. As Nancy alluded to, I think the reality is quite a bit more messy. Note the private emails between Eric and Miguel that were mentioned in this thread. We ourselves interact in LOTS of different locations.”
In other words, even if you could solve the knowledge capture challenge for one mode of discussion (mailing lists) you are still likely missing out on a lot of the learning and knowledge sharing going on in the community. Keep that in mind!
As I’ve written about before, within the context of my current employer’s community program, mailing lists, and their related archives are an important part of our community of practice initiative (and, by extension our KM program). We have not developed a formal means to retire (or “execute” in the terms used in the first thread mentioned above) a community, but we do have a formal process for retiring mailing lists. While the following is about mailing lists, I think the concepts can scale up to any community – though it might require aggregating similar insights about other channels used by the community.
Within our infrastructure, many of the existing mailing lists are associated with one (or more) communities and we provide a simple means for anyone to request a new mailing list. There is a very light review process, primarily focused on ensuring that the requested list is different enough from existing lists and also doesn’t have such a small topic space that it will likely be very under-utilized), which means that over time we can end up with a lot of mailing lists. Without some regular house-cleaning, this situation can have a very negative impact on how a user’s discovery process – hundreds and hundreds of mailing lists means a lot of confusion.
One way we grapple with this is to use the communities as a categorization of mailing lists. Instead of leaving a user with hundreds of mailing lists to wade through, we encourage them to look for a community in which they’re interested and, through that community, find associated mailing lists. This normally reduces the number of mailing lists to consider down to a small handful.
However, we still have needed a house-cleaning process, so several years ago, this is what we set up:
In our environment, doing this once a year typically reduces the count of lists by about 10% – though the count of lists has remained remarkably stable over time, which would say that we then have that same kind of growth over the next year. On the other hand, if we did not proactively review and retire lists like this, we would be seeing an ever-growing list of mailing lists, making it harder for everyone to find the lists that are engendering valuable discussions.
Or… How to lift knowledge out of the on-going discussion of a community into a better form of reusability.
If a community uses a tool like a mailing list to engender discussion and knowledge sharing – how does a community capture “nuggets” of knowledge from the discussion into a more easily digestible form? Does the community need to (perhaps not given a sophisticated enough means to find information in the archives)?
I have no magic solution to this problem but I did find another comment to be very illustrative of one aspect of the original discussion – who “owns” the archives of a community’s discussion and what is the value of those archives? Even in their raw form, why do those archives have value? As Nancy White notes:
“I suspect that only a small percentage of the members (over time) would actually use the archives. But because they hold the words of members, there may be both individual and collective sense of ownership that have little to do with “utility.”"
The rest of this post will be a brief description of a knowledge capture process I’m very familiar with – though I’m not sure if it will transfer well into other domains. For this description, I’m going completely outside of the enterprise and to a community of which I’m a member that revolves around a table-top fantasy war game named Warhammer.
A bit of background: Warhammer is a rather complex game, with a rulebook that weighs in at several hundred pages and about a dozen additional books that provides details on the various types of armies players can use. All told, probably something like 1,000 pages describing the rules and background of the game. Given the complexity of the game, it is very common that during any given game, the players will run into situations not covered well by the rules – these are usually areas involving interactions of special rules for the armies playing. In the many online forums / mailing lists that exist, one of the most frequent types of discussions revolves around these situations and how to interpret the rules. Many of the same questions come up repeatedly – obvious fodder for an FAQ.
(As an aside, given that Warhammer is published and sold by a company – Games Workshop – one could that they should publish all of the relevant FAQs. They do publish FAQs and errata but they do so at a sporadic pace at best and do not address many of the frequently asked questions.)
One particular Warhammer-related community of which I’m a member – the Direwolf (DW) community – has established a pretty well defined means to gather these FAQs and publish them back to the Warhammer community at large. A brief overview of the process:
Netting it out: A community-selected subset of the community monitors the community for questions in their area of expertise, vettes an answer with the rest of the FAQ council, and then the FAQ documentation is updated as appropriate.
This is pretty straightforward, but the value of this effort is reflected in the fact that the game publisher now very commonly uses input from the Direwolf FAQ council in considering their own responses to FAQs and also in the fact that many players from around the world use the Direwolf FAQ to ensure a consistent interpretation of those “fuzzy” areas of the game. A true value add for the Warhammer community at large.
That being said, this process does take quite a bit of energy and commitment, especially on the part of the “keeper” of the documentation, to keep things up to date. In this case, I believe that the value-add for members of the council is knowing that they are contributing to the Warhammer community at large and also knowing that they are helping themselves in their own engagement of playing the game.
Last summer, I read the article by Kas Thomas from CMS Watch titled “Best Bets – a Worst Practice” with some interest. I found his thesis to be provocative and posted a note to the SearchCoP community asking for other’s insights on the use of Best Bets. I received a number of responses taking some issue with Kas’ concept of what best bets is and some also some responses describing different means to manage best bets (hopefully without requiring the “serious amounts of human intervention” described by Kas.
In this post, I’ll provide a summary of sorts and also describe some of the ways described for managing best bets and also the way we have managed best bets.
Kas’ thesis is that best bets are not a good practice because they are largely a hack layered on top of a search engine and require significant manual intervention. Further, if your search engine isn’t already providing access to appropriate “best bets” for queries, you should get yourself a new search engine.
Some of the most interesting comments from the thread of discussion on the SearchCoP include (I’ll try to provide as cohesive picture of sentiment as I can but will only provide parts of the discussion – if I have portrayed intent incorrectly – that’s my fault and not the original author):
From Tim W:
“Search analytics are not used to determine BB … BB are links commonly used, enterprise resources that the search engine may not always rank highly because for a number of reasons. For example, lack of metadata, lack of links to the resource and content that does not reflect how people might look for the document. Perhaps it is an application and not a document at all.”
From Walter U:
“…manual Best Bets are expensive and error-prone. I consider them a last resort.”
From Jon T:
“Best Bets are not just about pushing certain results to the top. It is also about providing confidence in the results to users.
If you separate out Best Bets from the automatic results, it will show a user that these have been manually singled out as great content – a sign that some quality review has been applied.”
From Avi R:
“Best Bets can be hard to manage, because they require resources.
If no one keeps checking on them, they become stale, full of old content and bad links.
Best Bets are also incredibly useful.
They’re good for linking to content that can’t be indexed, and may even be on another site entirely. They’re good for dealing with … all the sorts of things that are obvious to humans but don’t fit the search paradigm.”
So, lots of differing opinions on best bets and their utility, I guess.
A few more pieces of background for you to consider: Walter U has posted on his blog (Most Casual Observer) a great piece titled “Good to Great Search” that discusses best bets (among other things); and, Dennis Deacon posted an article titled, “Enterprise Search Engine Best Bets – Pros & Cons” (which was also referenced in Kas Thomas’ post). Good reading on both – go take a look at them!
My own opinion – I believe that best bets are an important piece of search and agree with Jon T’s comment above that their presence (and, hopefully, quality!) give users some confidence that there is some human intelligence going into the presentation of the search results as a whole. I also have to agree with Kas’s argument that search engines should be able to consistently place the “right” item at the top of results, but I do not believe any search engine is really able to today – there are still many issues to deal with (see details in my posts on coverage, identity, and relevance for my own insights on some of the major issues).
That being said, I also agree that you need to manage best bets in a way that does not cost your organization more than their value – or to manage them in a way that the value is realized in multiple ways.
Contrary to what Tim W says, and as I have written about in my posts on search analytics (especially in the use of search results usage), I do believe you can use search analytics to inform your best bets but they do not provide a complete solution by any means.
From here on out, I’ll describe some of the ways best bets can be managed – the first few will be summary of what people shared on the SearchCoP community and then I’ll provide some more detail on how we have managed them. The emphasis (bolding) is my own to highlight some of what I think are important points of differentiation.
From Tim W:
“We have a company Intranet index; kind of a phone book for web sites (A B C D…Z). It’s been around for a long time. If you want your web site listed in the company index, it must be registered in our “Content Tracker” application. Basically, the Content Tracker allows content owners to register their web site name, URL, add a description, metadata and an expiration date. This simple database table drives the Intranet index. The content owner must update their record once per year or it expires out of the index.
This database was never intended for Enterprise Search but it has proven to be a great source for Best Bets. We point our ODBC Database Fetch (Autonomy crawler) at the SQL database for the Content Tracker and we got instant, user-driven, high quality Best Bets.
Instead of managing 150+ Best Bets myself, we now have around 800 user-managed Best Bets. They expire out of the search engine if the content owner doesn’t update their record once per year. It has proven very effective for web content. In effect, we’ve turned over management of Best Bets to the collective wisdom of the employees.”
From Jim S:
“We have added an enterprise/business group best bet key word/phrase meta data.
All documents that are best bet are hosted through our WCM and have a keyword meta tag added to indicate they are a best bet. This list is limited and managed through a steering team and search administrator. We primarily only do best bets for popular searches. Employee can suggest a best bet – both the term and the associated link(s). It is collaborative/wiki like but still moderated and in the end approved or rejected by a team. There is probably less than 1 best bet suggestion a month.
If a document is removed or deleted the meta data tag also is removed and the best bet disappears automatically.
Our WCM also has a required review date for all content. The date is adjustable so that content will be deactivated at a specific date if the date is not extended. This is great for posting information that has a short life as well as requiring content owners to interact with the content at least every 30 Months (maximum) to verify that the content is still relevant to the audience. The Content is not removed from the system, rather it’s deactivated (unpublished) so it no longer accessible and the dynamic links and search index automatically remove the invalid references. The content owner can reactivate it by setting the review date into the future.
If an external link (not one in our WCM) is classified as a best bet then a WCM redirect page is created that stores the best bet meta tag. Of course it has a review/expiration so the link doesn’t go on forever and our link testing can flag if the link is no longer responding. If the document is in the DMS it would rarely be deleted. In normal cases it would be archived and a archive note would be placed to indicate the change. Thus no broken links.
Good content engineering on the front end will help automate the maintenance on the back end to keep the quality in search high.“
The first process is external to the content and doesn’t require modifying the content (assuming I’m understanding Tim’s description correctly). There are obvious pros and cons to this approach.
By contrast, the second process embeds the “best bet” attribution in the content (perhaps more accurately in the content management system around the content) and also embeds the content in a larger management process – again, some obvious pros and cons to the approach.
Now for a description of our process -The process and tools in place in our solution are similar to the description provided by Tim W. I spoke about this topic at the Enterprise Search Summit West in November 2007, so you might be able to find the presentation for it there (though I could not just now in a few minutes of searching).
With the search engine we use, the results displayed in best bets are actually just a secondary search performed when a user performs any search – the engine searches the standard corpus (whatever context the user has chosen, which would normally default to “everything”) and separately searches a specific index that include all content that is a potential best bet.
The top 5 (a number that’s configurable) results that match the user’s search from the best bets index are displayed above the regular results and are designated “best bets”.
How do items get into the best bets index, then? Similar to what Tim W describes, on our intranet, we have an “A-Z index” – in our case, it’s a web page that provides a list of all of the resources that have been identified as “important” at some point in the past by a user. (The A-Z index does provide category pages that provide subsets of links, but the main A-Z index includes all items so the sub-pages are not really relevant here.)
So the simple answer to, “How do items get into the best bets index?” is, “They are added to the A-Z index!” The longer answer is that users (any user) can request an item be added to the A-Z index and there is then a simple review process to get it into the A-Z index. We have defined some specific criteria for entries added to the A-Z, several of which are related to ensuring quality search results for the new item, so when a request is submitted, it is reviewed against these criteria and only added if it meets all of the criteria. Typically, findability is not something considered by the submitter, so there will be a cycle with the submitter to improve the findability of the item being added (normally, this would include improving the title of the item, adding keywords and a good description).
Once an item is added to the A-Z index, it is a potential best bet. The search engine indexes the items in the A-Z through a web crawler that is configured to start with the A-Z index page and goes just one link away from that (i.e., it only indexes items directly linked to from the A-Z index).
In this process, there is no way to directly map specific searches (keywords) to specific results showing up in best bets. The best bets will show up in the results for a given search based on normally calculated relevance for the search. However, the best bet population numbers only about 800 items instead of the roughly half million items that might show up in the regular results – as long as the targets in the A-Z index have good titles and are tagged with the proper keywords and description, they will normally show up in best bets results for those words.
Some advantages of this approach:
Some disadvantages of this approach