Lee Romero

Community of Practice Metrics and Membership, Part 2

In my last post, I described an effort to define “community membership” and how we came upon the definition:

The membership of a community would be defined by the membership in the related mailing list(s).

In this post, I will provide some of the work we did to effect this definition and also some of the basic analysis and insights this supported.

Collecting Membership

First up – how did we take this definition and make it useful? The mailing list server mentioned in my previous post was built using MailMan, the GNU Mailing List Server. This is a very flexible tool, though it is primarily focused on providing mailing list functionality in an internet setting – one where your identity is not managed in a corporate directory and one where mailing lists may have nothing in common beyond presence on a single server (no common membership or even commonality in, say, the email addresses of members on the list and no larger “identity” tied to those memberships).

So the first question was how to collect the list of members into an easily manipulated format and, in doing so, how to then connect the memberships with the larger identity of an employee?

MailMan does not provide a directly queryable data source that can easily be combined with data from other systems (basically, some type of SQL database whose data can be joined with others), so we took a two step approach to collecting membership lists:

1. We added a minor customization to MailMan that provided the ability to get a list of members of a list in an XML format. Normally, this list is presented in HTML only. Each member was identified through an email address and (optionally) a full name.
2. We built a simple sync mechanism that queried this XML interface and populated a SQL database with the list members for each mailing list.
   1. In this sync process, we used our corporate directory to match email addresses and were, through that connection, able to relate community members to their larger identify within the corporation.
   2. In addition, the sync process recorded the subscribing / unsubscribing “events” so that it was possible to understand not just who is a member at any point but how membership has changed.
3. In addition to pulling mailing list membership into a SQL database, we also built a sync mechanism that populated a table with a record for every post to every mailing list for easy querying on those (I’ll explain more about why we did this in a future post). For this data, we connected the posts to the membership records (when possible – not all mailing lists are configured to require membership to post to them) and also stored the date / time of the post and the subject line. (I’ll write more about how this data was useful in a future post.)

Using the Membership Data

With this solution in place, it becomes a simple matter to answer basic questions about community membership. That being said, I’ll try to provide some actionable steps for each type of query that we could take based on the insights gained. Without keeping that focus, a lot of the possible analysis becomes academically interesting (perhaps) but does not have any meaningful business value.

Some examples:

• You can easily query on membership for any given mailing list
  • This is useful for community managers to understand what topics are of primary interest within a community. It is often the case that a community will have more than one related mailing list, but which is of “most interest” (based on membership)?
• Given a list of mailing lists associated with a particular community of practice, the community membership is easily queried as the set of distinct members across all related mailing lists
  • For community managers, this was useful to understand the effects of their efforts to increase membership.
• You can track growth in communities by reviewing the number of people who subscribe / unsubscribe from mailing lists over a desired time period.
  • For community managers and community sponsors, this was useful insight to understand the history of the community.
  • You can see a sample chart that shows two communities and their growth over a series of quarters; one was a community that existed prior to the start of the CoP initiative (so it started out quite large) and shows good growth and then shows a decline (the big jump was likely due to some refactoring of community / mailing list alignments, ,though I don’t have the details) and the second community was one launched during this period and it shows a good, steady (what I would call “organic”) growth during the period covered here.

Community Size Chart

• You can measure things like the percentage of the corporation that are members of any community or the percentage of members of specific groups within the company that are members of any community.
  • Looking at it from the perspective of percentage of the entire enterprise – this was useful insight because it provided the sponsors of the community of practice program with insight about how pervasive communities are throughout the enterprise.
  • This also provides useful insight to then contrast “penetration” with specific communities – it provides a baseline for comparison across time and within various slices of the organization.
  • As a baseline, we found that at the outset of the formal CoP program, about 28% of the corporation was a member of at least one community of practice. As we progressed forward, we could measure that penetration over time and, today, percentage is almost 38%. The chart presented below shows a series of quarters with this data displayed.
  • And, finally, given that baseline, it presents the possibility of understanding penetration into specific groups and allowing us to ask questions like, “X% of the community is a member of community A, but only Y% of group B is a member of community A – is this desirable?” In other words, specific groups could be targeted for recruitment if appropriate.

CoP Penetration Chart

• By turning the data around and looking at it from the perspective of the individual employees, we could answer a question like:
  • How many communities is an average employee a member of?
  • How many communities is an average community member a member of?
  • We were never able to specifically identify actionable steps to take from this insight but it gave us some idea of how widely interest between our communities ranged.

That’s an overview of the basic types of things we have been able to do with this data. Here are the topics I plan to cover in subsequent posts in this area:

• Understanding the demographics of your communities
• Using the “activity data” related to posts
• Using this data in our performance management program
• Measuring (part of) knowledge flow within a community

Check back for more on the above soon!