Lee Romero

Yesterday, I delivered my presentation at Taxonomy Boot Camp 2010 on “Enterprise Taxonomy: Six Components of a vision”.  You can find the presentation on my site here and also on the Taxonomy Boot Camp site here (the latter requires a login you will need to get from the conference).

Some of the most interesting topics for me this week have been about semantic (web) technologies and also some details on the implementation of taxonomy in SharePoint 2010.  Good stuff.

In addition, I’ve had the opportunity to meet and re-meet many people who work in the taxonomy space and also in search, so it’s been a very revitalizing experience.

I also (finally) picked up a copy of the Accidental Taxonomist by Heather Hedden.  I am really looking forward to reading it.

Recently, I have been trying to better understand the language in use by our users in the search solution we use, and in order to do that, I have been trying to determine what tools and techniques one might use to do that. This is the first post in a planned series about this effort.

I have many goals in pursuing this.  The primary goal has been to be able to identify trends from the whole set of language in use by users (and not just the short head).  This goals supports the underlying business desire of identifying content gaps or (more generally) where the variety of content available in certain categories does not match with the variety expected by users (i.e., how do we know when we need to target the creation and publication of specific content?)

Many approaches to this do focus on the short head – typically the top N terms, where N might be 50 or 100 or even 500 (some number that’s manageable).  I am interested in identifying ways to understand the language through the whole long tail as well.

As I have dug into this, I realized an important aspect of this problem is to understand how much commonality there is to the language in use by users and also how much the language in use by users changes over time – and this question leads directly to the topic at hand here.

Chart 1

There is an anecdote I have heard many times about the short head of your search log that “80 percent of your searches are accounted for by the top 20% most commonly-used terms“.  I now question this and wonder what others have seen.

I have worked closely with several different search solutions in my career and the three I have worked most closely with (and have most detailed insight on) do not come even close to the above assertion.  Chart 1 shows the usage curve for one of these.  The X axis is the percent of distinct terms (ordered by use) and the Y axis shows the percent of all searches accounted for by all terms up to X.

From this chart, you can see that it takes approximately 55% of distinct terms to account for 80% of all searches – that is a lot of terms!

This curve shows the usage for one month – I wondered about how similar this would be for other months and found (for this particular search solution) that the curves for every month were basically the exact same!

Wondering if this was an anomaly, I looked at a second search solution I have close access to to wonder if it might show signs of the “80/20” rule.  Chart 2 adds the curve for this second solution (it’s the blue curve – the higher of the two).

Chart 2

In this case, you will find that the curve is “higher” – it reaches 80% of searches at about 37% of distinct terms.  However, it is still pretty far from the “80/20” rule!

After looking at this data in more detail, I have realized why I have always been troubled at the idea of paying close attention to only the so-called “short head” – doing so leaves out an incredible amount of data!

In trying to understand the details of why, even though neither is close to adhering to the “80/20” rule, the usage curves are so different, I realize that there are some important distinctions between the two search solutions:

1. The first solution is from a knowledge repository – a place where users primarily go in order to do research; the second is for a firm intranet – much more focused on news and HR type of information.
2. The first solution provides “search as you type” functionality (showing a drop-down of actual search results as the user types), while the second provides auto-complete (showing a drop-down of possible terms to use).  The auto-complete may be encouraging users to adopt more commonality.

I’m not sure how (or really if) these factor into the shape of these curves.

In understanding this a bit better, I hypothesize two things:  1) the shape of this curve is stable over time for any given search solution, and 2) the shape of this curve tells you something important about how you can manage your search solution.  I am planning to dig more to answer hypothesis #1.

Questions for you:

• Have you looked at term usage in your search solution?
• Can you share your own usage charts like the above for your search solution and describe some important aspects of your solution?  Insight on more solutions might help answer my hypothesis #2.
• Any ideas on what the shape of the curve might tell you?

I will be writing more on these search term usage curves in my next post as I dig more into the time-stability of these curves.

Next week, I will be speaking at Taxonomy Boot Camp 2010 – on the topic of Enterprise Taxonomy: A Vision.  Much of what I will have to share is from a post I wrote here some time ago.

Hopefully I have enough new issue to make it a good session for everyone!

Now that I’ve posted quite a bit on the technical side of an enterprise taxonomy, I thought I’d share a bit on the business process side of how we have managed our taxonomy.

I spoke about this topic at the  2007 Taxonomy Boot Camp. (As an aside, I tried to find if the presentation I used is available on the site but I couldn’t find it – if someone knows of an online archive, please let me know and I can provide a link from here.) The session I delivered was titled, “The Process and Politics of Implementing a Corporate Taxonomy” and focused on the overall process we have implemented.

What follows is an overview of the larger process we used to establish the taxonomy and a description of the smaller process used to maintain it and I’ll close with some of my own thoughts on what it is that triggers changes in a taxonomy.

Getting Started

When we first started trying to formalize a taxonomy, one of the first steps we took was to do an organizational mapping to identify participants in the process. We focused on the following:

• Groups that had significant investments in web content publication
• Groups that had significant interest and investment in knowledge capture and sharing
• Groups that have influence on the corporate culture

We felt that this organizational mapping was important because it would help increase buy-in to the taxonomy from those who have most vested interest in it and also (with help from that last group) would help increase larger scale adoption of the language. Once we felt that we had identified the groups that met these criteria, we engaged with the executives for the groups to help us identify one or more people who could be included in our Taxonomy Review Board.

The rest of the “getting started” process included content audits and analyses to identify terminology used to describe the content, definition of the structure of the taxonomy we wanted to use, organization of the terminology into this structure and then working with the Taxonomy Review Board to confirm the end result as a first version of the (evolving) taxonomy.

We also layed out the objectives we had for the overall process – which you can find in my post on the vision we have developed for our taxonomy. The really pertinent items we wanted to ensure were: We wanted to ensure that the taxonomy was actively managed and we wanted to ensure that the management process was transparent.

The People

Now that the taxonomy had been established, we needed to identify the people and process we would use for maintaining and enhancing the taxonomy.

The people who are involved include:

• The taxonomy manager – a single person responsible for responding to requests for changes, proactively identifying proposed changes within the taxonomy and handling the “administrative” side of the process. If it’s helpful, I’ve found that this responsibility probably takes about 10% of a single person’s time (though that obviously reflects the size of our organization and volume of content, etc., and can vary at different times) This is my role within the process.
• A core team – a group of about 3 people (one of which is the taxonomy manager) who do a first-level check of change requests to make sure that requests that are obviously (at least in the minds of the core team) not worth moving farther in a review process are not further considered. Time commitment for this group is probably on the scale of about a few hours a month.
• The above-mentioned Taxonomy Review Board (TRB)- A cross-organizational group that reviews proposed changes and aligns with them or propose counter-proposals. This group currently has about 15-20 members. Time commitment for this group is minimal – normally, the proposals for change have been considered and detailed enough by the time this group sees them that their involvement is to receive emails with change proposals and either align (so no reply necessary) or write a counter proposal.

This organization has helped to keep the taxonomy managed, while also keeping overall enterprise expense to manage it fairly small.

The Process

Now, I am, at heart, a software engineer. Why is this pertinent? Early on in my career, I came to appreciate the need and value for change control (or, as I prefer to think of it change management or change visibility – I’ve always thought “control” seemed a bit stronger than you could really achieve) and that has seeped into our process.

At its heart, our process is similar to a software development team’s change control board (CCB) process:

1. All changes, upon identification, are captured in the same bug-tracking system used for our engineering and IT systems (an implementation of Bugzilla). Just like with software, all changes are treated as either enhancements (extending beyond what we have now) or defects (a problem or mistake that was not anticipated) and so they follow the same lifecycle (I generically use the word “bug” below to mean the specific documented request in our tracking system for a change, regardless of whether the change is an enhancement or defect).
2. Once a change is documented as a bug (I’ll write a bit more below about the sources of changes to the taxonomy), it is assigned to the taxonomy manager for resolution.
3. The taxonomy manager then needs to do a few things:
   1. Ensure that the bug contains all of the necessary details and any obvious questions are answered. An example of this would be the specific guidelines we have for one of our classifications – I shared these with the TaxoCoP and Patrick Lambe blogged about them as well. In this case, the taxonomy manager is on the hook to ensure a request adheres to these guidelines.
   2. Describing the impact of the change on the rest of the taxonomy (if any).
4. The change is then reviewed by the core team – this review is typically virtual via email exchange but can be a meeting convened by the taxonomy manager.
   1. If the core team aligns with the change (perhaps after some continued evolution of it), it moves forward for a review by the full Taxonomy Review Board.
   2. If the core team rejects the change, it is canceled. The taxonomy manager communicates that back to the requester (if the trigger for the change was a particular person or group).
5. When a change is put to the Taxonomy Review Board, which is a virtual team (and which is geographically very distributed), it is communicated by email to the TRB.
   1. At this point in the process, we want to ensure efficiency in process so we do not use a “request for comment” type of approach.
   2. Instead, the change is detailed for the TRB and the TRB members are given two options: 1) align with the change as stated or 2) provide a counter proposal. This helps keep focused and helps to avoid potentially lengthy discussions on the change at this point.
   3. To further accelerate decision making and reduce time on the part of the TRB members, each request is also positioned as a time-boxed proposal: You have until <this date> to provide a counter proposal or else you are assumed to align to the change. In other words, no reply from a member equates to alignment.
   4. Another implication of this is that by the point a change reaches the TRB it is almost inevitably going to go ahead in some form (perhaps changed by counter-proposals from the TRB). It will not be canceled. That seems possible but so far has not happened in practice.
6. Upon achieving alignment within the TRB on the final proposal, the change is executed in the taxonomy and the request closed. The change is communicated back to the requester to close the loop and (especially for significant changes) the change may also be communicated to our larger content manager community.

Issues with the Process and Framework

While it has worked effectively we still face a number of issues with this process. These include:

• The need to keep on top of organizational changes – specifically, with regard to membership in the Taxonomy Review Board. A member’s role within the enterprise can change to the point where they may not be in the best position to represent a group of interests. In addition, with some organizational changes we’ve seen, it can result in an “unbalanced” TRB.
• Which brings us to the second issue – organizational coverage. Currently, we have a TRB that overly represents our marketing organization and is missing representation from some groups that should be represented.
• Lastly, support of this process from within our IT organization is a concern. I see this in a couple of different ways:
  • Organizationally, the taxonomy manager falls within IT but the responsibility to continue managing the taxonomy is not perceived as a priority (and there’s a question as to whether it should even organizationally be within IT);
  • In terms of adoption, it has been a challenge to educate the IT organization about the value and use of the taxonomy. An example would be integration with a business intelligence solution to ensure consistency in language and, more specifically, to be able to effectively integrate insights about content (which does use the taxonomy) with more transactional-based “data”.

Identifying the Need for Change

What triggers a change in the taxonomy?

As I (re-)gather my thoughts on this topic, one lingering question came back to me about the overall process. The question is external to the process (which takes the approach of “a change comes from somewhere and we’re not going to worry about where it comes from but once it’s been identified, we’ll wedge it into this process”) but I am interested in understanding what other taxonomists might actively do in maintaining a taxonomy. In other words, how much change do you experience that comes from others compared to your own recommendations or insights?

Here’s a list of triggers that have resulted in changes in the taxonomy:

• We provide content publishers with a mechanism to request a change to one facet (“Item Type”) at the point where they are submitting a piece of content. I consider this to be a purely tactical, reactive change and, given the above process, suffers from the problem that a content publisher cannot sit at their computer waiting for the business process to complete before they submit their content. So even if a new value is adopted, they will need to publish their content with a temporary value and remember to come back and change it after the fact.
• I have engaged with content owners several times who were planning to publish a set of content and worked proactively with them to understand their content and ensure that the taxonomy provides good coverage. It’s lucky (though perhaps it shouldn’t be!) when this can happen and I manage to ensure the taxonomy changes are in place before they need to publish content.
• When a new repository is being migrated or merged into a system using the taxonomy, there will likely be a number of changes in the taxonomy, including adoption of whole new classifications and introduction of new values. Also, this almost inevitably require a good mapping from local system values to the taxonomy values where there is (near) overlap.
• Most proactively on my part, I have also used analytics from a number of sources to help refine the taxonomy, including:
  • Reviewing search query logs to understand the language being used by people looking for content
  • Reviewing the “free text” fields (e.g., title, description, etc.) within content management systems to look for terms that are commonly used that might warrant explicit use in a constrained classification.
  • Reviewing the volume of content when split along various dimensions of the existing taxonomy – looking for opportunities to merge (values are under-utilized), split (values are over-utilized) or perhaps retired (values are not utilized)
• Adoption of new terminology by groups responsible for that part of the taxonomy. A common example is the terminology used to describe our various solution offerings – these will, at times, be changed unilaterally by our marketing organization and we then need understand how that translates to the existing taxonomy and to content tagged with that taxonomy.
• Lastly, given that another part of the vision of the taxonomy is to use systems of record where possible, a number of changes are triggered outside of the taxonomy and simply synchronized in from the source system. This approach assumes (true in all cases as far as I am aware) that the source systems provide their own management process on values and these changes do not require any review through the above taxonomy management process.

In my continuing dive into the structure of our taxonomy, which, hopefully might be of use or interest to you to understand and possibly adopt to your own needs, so far, I’ve provided an outline of the application solution and then a high level outline of the data model we’re using.

One of the important features of our solution is that our taxonomy system provides the ability for other systems to consume the taxonomy via an XML document. I’ll explore that a bit here.

Accessing the XML

Access to the XML document for the taxonomy is through a very simple means: a standard HTTP GET. The query string in the request can specify various parameters on the URL – effectively, a very simple web service. The types of parameters supported include:

• Identifying which classification is desired (default is to return all)
• Specifying the statuses of values to include (default will return all)
• Specifying the language to include (default returns English)
• Specifying the level of detail of interest (default returns the briefest format)

With regard to the language – one of the business rules followed in our web sites is that you provide content in the user’s selected language when available and return English when the user’s language is not available (English should always be available). This rule is pushed down into this interface at the level of each value. So a consuming application might request the set of German values for the taxonomy and get all of the classification details in German and, say, 99% of the values in German but if there are values that are not translated, those are returned in English. This approach keeps the taxonomy consistent with our general rules (though if taxonomy values are used directly in a user interface, it does present a possibly confusing same-page mix of non-English and English).

Document structure

The returned XML document looks like the following. I’m not using any formal XML schema syntax – instead showing the elements and how they relate to each other with a brief description of th elements that I don’t think are self-explanatory.

• taxonomy
  • classification – has an attribute id (the ID of the classification)
    • name – has an attribute lang (the language code describing the language of the name element)
    • description – has an attribute lang (the language code describing the language of the description element)
    • status
    • createDate
    • updateDate
    • sourceSystem
    • comments
    • hasValues (a Y/N indicating if a consuming application should expect to find values in the values element)
    • constrained (a Y/N indicating if a consuming application should enforce the rule that values for this classification must come from the list of values provided)
    • multiValued (a Y/N indicating if a consuming application should allow multiple values be assigned for any given content piece)
    • dataType
    • changeHistory – an element with a sequence of elements, one for each auditable event in this item’s life history
      
    • aliases – has attribute count (the number of alias elements included)
      • alias – a structured element providing details on an alias
    • levels – has an attribute count (the number of levels included)
      • level – a structured element providing details on the level (omitted here)
    • values – has an attribute count (the number of values included)
      • value – has an atribute id (the ID of the value in the taxonomy system)
        • name – has an attribute lang (the language code describing the language of the name element)
        • description – has an attribute lang (the language code describing the language of the description element)
        • status
        • createDate
        • updateDate
        • sourceSystemId
        • levelRef – attribute id (identifies the specific level [in the levels element above] with which this value is associated)
        • aliases – attribute count (the number of aliases for this value)
          • alias – a structured element providing details on an alias
        • changeHistory – Same as for classification
        • values – recursive structure reflecting hierarchy within a classification’s set of values
          • value (etc.)

And that’s the schema. Looks complicated, but it’s really pretty simple, I think. The advantage of this has been that consuming applications do not need to directly access the database containing this (which would be pretty simple in principle) and so can be insulated from changes in the underlying structure of the database as we need to make them.

Providing access via an HTTP get keeps the technical cost minimal for consuming applications (they need to be able to read from an HTTP socket and then parse XML, both pretty standard functions in modern languages / libraries).

One last comment – in regard to the level of detail parameter mentioned above – the “brief” level includes the names , descriptions and statuses only of the classifications, levels and values.  The “detailed” includes all details except the changeHistory elements.  The “complete” level includes all of the above.  The “complete” format is probably not very useful for consumers as most will not care about the life history of elements (though that is of interest and value within the taxonomy).

Relationship to other Schemas

Just to connect the dots – I know of other XML schemas that we could conceivably have used to publish this document.  With help from the Taxonomy community of practice, I found the following while researching for a schema to use (I especially want to say thanks to Leonard Will, Mike Taylor, Marcel van Mackelenbergh and Bob Bater for their insights):

• RDF-XML – A (relatively) low-level schema used to publish facts (ontologies when used in whole)
• SKOS – A schema in draft as I write this from W3C for sharing knowledge organization systems among. Based on RDF.
• Zthes – An XML schema for publishing thesauri.
• Topic Maps – An XML schema for representing topics and relationships among topics (Bill French also mentioned this in a comment on my previous post.)
• A good list of references available from the ANSI/NISO Z39.19 – Guidelines for the Construction, Format, and Management of Monolingual Controlled Vocabularies

At the time we were designing (defining) a schema to use, we knew we wanted to keep it as simple as possible and (right or wrong) as close to the underlying model as we could, which made sense within our business environment. It wasn’t clear at the time which of the above might provide the most likely path forward (in terms of standard adoption) so we “rolled our own”. And, another factor was that the schemas seemed far more general than our needs warranted; for example, the broader-than / narrower-than type relations were implicit in our structure and specifying those explicitly seemed confusing. (To be honest, all of which could be interpreted as “we weren’t educated enough to understand the options and took the simpler-at-the-time approach of rolling our own”.)

I am still not as familiar as I would like to be with the above, so I still would not be able to say which would be most appropriate, but the SKOS schema, now in draft from the W3C seems like a potential solution that would fit our needs and could eventually become a broader standard.  Does anyone have any insights as to where this is moving?

In my previous post, I started describing the structure of the taxonomy we are using in some detail; originally, the following was part of my last post but it got a bit too long so I’ve split it. In this post, I’ll explore the structure in yet more detail – getting closer to a data model.

If you are going through a similar process that we’ve been through and you want to organize your taxonomy in a database, this might provide you with enough detail to get moving.

One note on terminology – much of what we have used is not what I would consider “standard” among taxonomist but was derived during a period when we had numerous systems we were trying to pull together, each of which used one of many different terms – categories, attributes, metadata, fields, tags, etc. I was charged at this point (which was before we started digging into the details of defining an enterprise taxonomy) with trying to define some terms that we could all use so that we could at least understand each other. A taxonomy for taxonomies, I guess.

Classification

The primary construct in the taxonomy is called a “Classification”. A better term for this I now know would be “Facet” as that’s what they are. The intent is that a Classification is a specific set of values (perhaps explicitly defined or perhaps defined by a set of guidelines or business rules) with which pieces of content can be associated (they can be tagged with values from the classification).

In our schema, a Classification itself has a number of elements:

• Name – The preferred name for the Classification. Typically used as the label for fields on, for example, data entry forms of various sorts.
• Definition – A concise definition of the Classification. Forcing the explicit definition of this helps reduce fuzzy thinking and gets people to clearly differentiate when a new Classification is needed versus using an existing one. This can be displayed in other systems that allow users to associate classification values with content as a kind of “mini-help”.
• Life History (create date, modification date, audit trail) – We maintain the create date (actually, date added to the taxonomy) and a modification date so we know what happened when to the Classification. More detail is provided below on the audit trail.
• Source System – Each classification might be sourced from another system. An example is a product listing – these are not maintained in the taxonomy but in their own systems and the taxonomy simply uses that list. Another example (where we do not have automation) is language (where we reference ISO standards as the master even though the values are still manually maintained in our taxonomy database).
• Comments – A text field to hold comments for use within the taxonomy. Notes about issues, etc. Not intended for end users as the Definition is.
• Data Type – The type of values expected for this Classification. Most commonly, just Strings, but we do define (for example) Creation Date and Expiration Date as classifications with data type of Date.
• Value Indicators – The taxonomy provides indicators to help other systems know what to do with the Classification – Should assignment be constrained to just the values provided by the taxonomy? Should other systems allow content pieces to be associated with multiple values of a classification?
• Synonyms – We provide for the Classification itself to have synonyms (these are synonyms for the Name of the classification). This can be used when (despite best attempts to the contrary) people want to continue to use different terms for the same classificatoin. An example might be that one system (and its user group) might want to refer to a “Region” whereas another might use the term “Market” or “Area”.
• Status – We provide a status indicator on pretty much everything within the taxonomy (Classifications, individual values, etc). The usage is consistent and breaks down into:
  • “Active” – the value can be assigned to new/modified content; should be displayed in any type of search UI (say as a pick list) if appropriate; and should be displayed if a user views the taxonomic tagging of an item.
  • “Inactive” – the value should not be able to be assigned to new content or be newly assigned to existing content; it should be displayed in search UIs (if appropriate) and should be displayed if a user views the taxonomic tagging of an item. Basically, it was valid at one point and still has value on content already tagged with it but we do not use it any more.
  • “Deleted” – We don’t delete values physically, but mark them “Deleted”. The value can not be assigned when creating or editing content, it should not be displayed in any search UI and it should not be displayed if a user views the taxonomic tagging of a piece of content. Basically, the value is no longer in the taxonomy (though some systems may still have the value associated with content in some ways).
  • “Proposed” – The first status for most items. The value would only be in the Taxonomy system itself and would not propagate to other systems. Indicates that it’s being considered for adding but has not yet been approved.
• A set of Classification Levels – Some classifications have an internal structure, described below in the “Level” section.
• Localizations of Classification – There may be non-English translations of the name and description of a classification in the taxonomy database (see below for more about multiple languages).
• A set of Classification Values – Most classification have a set of explicit values that can be associated with a piece of content. The values might be a flat list or might be hierarchical. The taxonomy database supports both. Currently, we do not support any type of many-to-many relationship or relationships across Classifications – just a simple one-to-many within a Classification which is a value / sub-value relationship (some Classifications provide more explicit constraints on the intended meaning of the relationship). Also, we do not have a construct that allows for an explicit (in the taxonomy database) meaning for any given relationship (specifically, narrower-than, broader-than, etc.) It’s implicit in the structure of the values.

Given the definition of a Classification as above, the terminology we use is that the taxonomy is, itself, the set of all Classifications we have defined and which can be used to tag content.  As with Classification itself, this is not, I think, consistent with standard using (the hierarchical structure within any one Classification would be considered a taxonomy) but adopting this definition at least got us organizationally out of the confusion of how we have a taxonomy when all of the values are not in a single, strict hierarchy.

Value

A Value is a single (usually textual, though might be dates or numbers) term which can be associated with a piece of content. Values are grouped into Classifications. A value association to a piece of content is what connects that piece of content to the taxonomy.

Like a Classification, a Value has a structure, which is only used when the Classification provides explicit values:

• ID – the unique identifier within the taxonomy that identifies the value. Most systems using the taxonomy will store this ID as the associate (and not the associated value). This allows for the Value to have its textual representation changed without having to revisit any content (say a product name changes or a country’s name changes)
• Structure details – What classification this value is associated with and which value in this Classification (if any) is the parent of this value. Also, some values have a designated “Level” (see below for more on that).
• Value – the textual representation of this value. The string users will see and interpret as the “value”.
• Definition – the definition of this value. As with the classifications, forcing this to be clearly defined provides a good “buffer” against people requesting values to be added that are duplicative or not generally useful. I’m surprised by how often asking a requestor for a clear definition (and how it’s different from another value that seems similar) stops them in their tracks.
• Life History – same as the Classifications
• Source System ID – For Classifications whose values come from another system, we maintain the source system’s ID so we can associate it back to the source system for updates. This can also be used by systems that pull from the taxonomy and also might happen (for other business reasons) to pull data from the same source systems and allows those systems to cross between the two sets of values.
• Status – Same as for Classifications
• Synonyms – Same as for Classifications but applied to the individual values. Synonyms for values are much more common than synonyms for classifications. Systems using the synonyms can potentially do many different things with synonyms (displaying them while a content manager is associating values with content, supporting search on them, etc.)
• Localization of Value and Definition – Non-English translations of the value and definition. See below for more details.

Level

Within a single Classification, we have adopted a mechanism we refer to as a “Level” in order to have a structure within the Classification when it’s meaningful to have different Values grouped into semantically different sets. I think of this as the means by which we support a structure of Classifications.

A good example is Geography. We have a single classification for Geography which contains all necessary values for tagging content for geographic relevance (or irrelevance in some cases). However, each Value within that Classification might represent a different type of Geography. Some values are regions of the world (“North America” or “EMEA”); some values are Countries (“France” or “Japan”); and some might be areas within a country of use (“Midwest United States”).

A Level is a hierarchy of terms within a Classification and any given Value can be assigned to a Level.

The value of this is that systems using the taxonomy can provide user interfaces that group similar values (a nested, tree-style interface, say) while we do not need to have multiple Classifications with relationships across the Classifications to support this.

Multiple Languages

In order to support multiple languages on our web sites, we have provided a means to localize the entire taxonomy. Because localized content is a critical component of our customer-facing site, we provide a structure so that all text that can be used outside of the taxonomy (primarily things like the names and definitions of Classifications, the name and definition for Values, Level names, and even synonyms of each of these) can be localized.

Systems that pull from the taxonomy can then use the available localized terms in their displays (falling back to English if a particular term is not available in a specific language). This could be used in field labels on forms or navigation labels in a browsing interface, menu items, etc.

Audit Events

As I mentioned in my post on a vision for an enterprise taxonomy, the taxonomy should provide transparency and allow interested users to examine the history of changes within the taxonomy. This is accomplished by maintaining a history of audit events which can be associated with any of the entities within the taxonomy (classifications, values, levels, etc). Each event is pretty simple:

• Event type – the type of event that occurred (addition of a new entity, modification of an entity, etc.)
• Event description – a longer (description) field describing the event. For bugs added / modified manually (as opposed to changes via feed from another system) this comment will almost always include a reference to the bug (in our bug database) that describes the change more fully.
• Date / time of event – When the event occurred
• User who triggered the event – Who triggered the event
• Associated entity (the value, classification, level, etc. that changed) – what was changed.

With the above, when a user views the taxonomy, they can see the full lifecycle of any given entity in the taxonomy.

The processes that pull taxonomy values from source systems also populate events, so we are gathering these for automated and manually maintained values.

All together, this helps provide interested users with some confidence in what’s changing and why it’s changing. In addition, provides the ability (not exercised) to measure “turbulence” in the taxonomy – amount of change over time, etc.

Up next, I’ll describe the XML schema we use for publishing from the taxonomy.

(Editor’s note – I started this several weeks ago and managed to get myself busy with a lot of other things in the meantime and am finally getting back to it now. Apologies for the lengthy pause in the discussion.)

In my last post, I described the vision we developed for our taxonomy and provided a little bit of insight on how it’s managed. I thought some might find it interesting to understand the structure within the taxonomy at a deeper level.

When we initiated our taxonomy effort, we started (as I think most do) by collecting a lot of the language used throughout our enterprise in a big spreadsheet. We went through the language and organized it into a variety of facets and for many of those facets, we organized the values into a hierarchy. We managed the taxonomy in a spreadsheet for a while with some success but there were problems (of course):

1. It was not possible to actually do any meaningful integration from a spreadsheet into any systems (to use the taxonomy);
2. It was always a challenge to ensure people had access to the most recent view of the taxonomy;
3. It was hard to really to meaningfully integrate the taxonomy with source systems that provide many of our labels in the taxonomy (to pull in values from those source systems).

Given this challenge and a developer resource and some good insights about what the taxonomy needed to do, we have created a relatively simple application that has enabled the taxonomy to be much more visible and also much more directly integrated with other systems. Note: It’s very likely that a commercial product would provide what we’ve done and a lot more, but when we set out on this it was not feasible to spend “hard” money on this, so we spent “soft” money in the form of a developer’s time. Perhaps not the best strategy but it’s been successful for our needs so far.

Given the above challenges we had with the “spreadsheet approach”, my primary interest was to solve the problems of access, display and integration and I was not interested in a system that provided a UI for maintaining the taxonomy (that was also supported by the fact that I’ve strived to have most of the taxonomy sourced from business systems and that the management of the other values has primarily been a one-person job and that person was familiar with databases and could update directly).

So, the taxonomy system comprises the following components:

1. A SQL database (built in MySQL to be specific);
2. A web application that provides a view of what is in the database – basically a mirror of the database structure which is described below;
3. A set of processes that run on schedules to pull data from source systems into the taxonomy;
4. An XML output following a formal(ish) specification to allow other systems to pull values from the taxonomy.

In my next post (possibly later today, even), I’ll provide more details on the structure – closer to a data model for the bits and pieces that comprise the entire taxonomy.

In my continuing coverage of a variety of content management and knowledge management topics, I thought it time to share some thoughts and experiences on managing an enterprise taxonomy for a corporation. I am planning a few posts on the topic – starting with a vision for the taxonomy that we developed at the start of our efforts that have helped to guide us, then moving to covering the management process, some insight on usage of the taxonomy and also a description of what the taxonomy looks like.

One important note – a lot of the initial leg work for the taxonomy was done by Frank Montoya and Meredith Lavine, so credit to them for getting things moving.

When we started out in developing an enterprise taxonomy, the company had nothing in place as any kind of content taxonomy – there was an implicit navigational taxonomy for web sites and there was ad hoc taxonomy in “keywords” type fields in a number of content management systems throughout the company. We knew that to be successful, we needed to have more formality to the taxonomy.

As we set about trying to define what we wanted in the taxonomy, we also realized we needed to ensure we were on a common ground for what we were trying to accomplish – otherwise, it was easy to imagine the taxonomy pulled all over the place, making it hard to achieve meaningful results in the long run. We needed some type of common vision for the taxonomy.

In working with a core group of stakeholders, we came up with the following statements as our vision for the enterprise taxonomy.

The Enterprise Taxonomy will:

• Be adopted for use in all systems that manage content or documents for those classifications that are defined within the Taxonomy
• Be used to tag content within those systems in order to ensure consistent language to describe our content
• Enhance the search experience for users through that tagging
• Be managed as its own asset, including defining the classifications and the values used within those classifications
• Use appropriate systems of record when possible to define the set of values used for a particular classification
• Enable monitoring of changes to the taxonomy values by content managers

One note on this vision – it uses the term “classification” in a number of locations. Within our nomenclature, you can read “classification” as meaning the same thing as a “facet” in a faceted taxonomy.

Some of these are pretty straightforward statements, but I thought I’d share a few thoughts on some of them.

First – part of the vision is that the taxonomy is managed as its own asset – what does that mean? It means:

• The taxonomy is a piece of content (actually, many pieces of content) subject to the same types of business rules we apply to other content.
• The taxonomy is subject to workflows for review of changes.
• The taxonomy is subject to periodic reviews.
• Changes to the taxonomy can be “staged” in the way other content changes can be staged for review.
• The taxonomy must be treated as an asset with value.

The vision also notes that it will use systems of record. Our taxonomy is broken into many classifications (facets), several of which overlap with other business entities in the company – product lines, solution, geographies, etc. Whenever possible, we literally (in a system, database sense) integrate the taxonomy to pull data from systems of record for those classifications that have a system of record. This provides many advantages:

• Commonality across systems for users.
• Standardized language between content tagging language and language used within business intelligence systems.
• Changes to classifications that have a system of record can be managed using the appropriate business process in that system of record – the taxonomy review process does not need to include these classification (we assume that the system of records will ensure appropriate reviews are performed).
• Ownership of the values for these classifications can be kept closer to the business responsible for them. That is, we have enabled a distributed ownership model.
• This helps minimize which classifications must be reviewed within the taxonomy itself – keeping the taxonomy much more nimble. The classifications that need to have a review within the taxonomy are those that are pretty much purely about content (item type being an example).
• Eventually, this will enable deeper integration between business intelligence systems and content management systems through direct linkage of business objects (say, a product) to content tagged with that. This linkage can be done using standard database mechanisms. (Something we have not yet implemented, though.)

Given that the taxonomy is managed as an asset, we also felt that it was important that content managers must able to monitor changes within the taxonomy. This means:

• Content managers have a means to easily find and review all changes being considered to the taxonomy (for classifications managed within the taxonomy – though many classifications managed in a system of record also provide this).
• Content managers should be able to comment any specific proposed change.
• Content managers should be able to inspect any entity (classification, or value, etc.) within the taxonomy and view a life history of it within the taxonomy – what was it added, changed, deprecated, etc.
• Content managers should be able to view all classifications and values – including ones that are no longer “active” within the taxonomy (they have been deprecated).

So there’s a start to taxonomy. Up next, I’ll provide some insight on the details of what the taxonomy looks like.