Point Lookout An email newsletter from Chaco Canyon Consulting
Point Lookout, a free weekly email newsletter from Chaco Canyon Consulting
April 18, 2007 Volume 7, Issue 16
 
Recommend this issue to a friend
Join the Friends of Point Lookout
HTML to link to this article…
Archive: By Topic    By Date
Links to Related Articles
Sign Up for A Tip A Day!
Create a perpetual bookmark to the current issue Bookmark and Share
Tweet this! | Follow @RickBrenner Random Article

Ten Reasons Why You Don't Always Get What You Measure: Part I

by Rick Brenner

One of the "truisms" floating around is that "You get what you measure." Belief in this assertion has led many to a metrics-based style of management, but the results have been uneven at best. Why?

Many believe that "You get what you measure." The belief persists, in part, because of anecdotal evidence; because some experiments do appear to be consistent with the assertion; and because so many of us believe that the rest of us believe it.

The Mars Climate Orbiter, which was lost in 1999
The Mars Climate Orbiter, as it would have appeared if it had not been lost upon entry into Mars orbit. Mars Climate Orbiter was lost because of a mismatch between Metric and English units in a file used by its trajectory models. The measurements were all objective, and they were even precise, but confusion led to a loss of the orbiter. Photo courtesy US National Aeronautics and Space Administration.
Still, there are reasons to question its validity. What follows is a catalog of possible explanations for variances between the promise and the reality of metrics-based management. In this Part I, we examine three assumptions underlying the measurement process. See Part II and Part III for more.
We assume that indirect measurements work
Sometimes we try to measure attributes that aren't directly measurable. For instance, when we try to measure immeasurables like loyalty or initiative, we actually measure something else that we assume is highly correlated with what we're trying to measure. We usually do this using interviews or surveys.
But too often, rigorous proof of the assumed correlation is unavailable. Sometimes, we comfort ourselves, saying, "it's so obvious," but this is risky — the history of management, psychology and science is replete with assumptions that, though obvious, were nonetheless false.
We assume that all attributes are measurable We assume (wrongly)
that all organizational
attributes are measurable
The word "measurement" evokes our experiences determining physical attributes like length, weight, or temperature. This leads us to assume that whatever we want to know can be determined by a suitable measurement, but that assumption can lead to trouble. Consider something as important as progress. Suppose a team has been working for three weeks, when, suddenly, someone realizes that their entire approach will never work. This is certainly progress — they've learned something important. But it probably won't register as progress in the organization's metrics. Most likely, it will be reported as a setback.
Measuring non-physical attributes, such as the advance of knowledge, is often possible when changes are incremental. But at times, our metrics fail, and they tend to fail precisely when we most want to know where we stand.
We assume that objectivity implies precision
Sometimes we use measures that are objective but imprecise. That is, we assume wrongly that multiple identical measurements would yield nearly identical results. For instance, when we measure attitudes using a survey "instrument," we assume that the results we obtain are relatively context-independent. We don't actually know that the results are independent of, say, the time of the month, or the price of the company's shares — we just assume it.
Rarely do we test these assumptions. Indeed, we often assume that these factors do affect the results. We know this because we sometimes observe organizations gaming the measurement for favorable results, or trying to influence the results by releasing favorable news.

Assumptions like these can account for some — not all — of the poor performance of metrics-based management. In a future issue, we'll examine the effects of employee behavior. Go to top  Top  Next issue: When Stress Strikes  Next Issue
Bookmark and Share


52 Tips for Leaders of Project-Oriented OrganizationsAre your projects always (or almost always) late and over budget? Are your project teams plagued by turnover, burnout, and high defect rates? Turn your culture around. Read 52 Tips for Leaders of Project-Oriented Organizations, filled with tips and techniques for organizational leaders. Order Now!
Your comments are welcome
Would you like to see your comments posted here? Send me your comments by email, or by Web form.
About Point Lookout
Thank you for reading this article. I hope you enjoyed it and found it useful, and that you'll consider recommending it to a friend.

Point Lookout is a free weekly email newsletter. Browse the archive of past issues. Subscribe for free.

Support Point Lookout by joining the Friends of Point Lookout, as an individual or as an organization.

Do you face a complex interpersonal situation? Send it in, anonymously if you like, and I'll give you my two cents.

Related articles
More articles on Personal, Team, and Organizational Effectiveness:
WildflowersRenewal
Renewal is a time to step out of your usual routine and re-energize. We find renewal in weekends, vacations, days off, even in a special evening or hour in the midst of our usual pattern. Renewal provides perspective. It's a climb to the mountaintop to see if we're heading in the right direction.

A holey grailHoley Grails
How much of the time and energy you spend in meetings goes to finding the best way? or a better way? It's of questionable value unless you first agree on what you mean by "better" or "best."

A  forest fireOrganizational Firefighting
Sometimes companies or projects get into trouble, and "fires" erupt one after another. When this happens, we say we're in "firefighting" mode. But it's more than a metaphor — we have a lot to learn from wildland firefighters.

A mobile phoneVirtual Communications: Part II
Participating in or managing a virtual team presents special communications challenges. Here's Part II of some guidelines for communicating with members of virtual teams.

Two people using an information kioskKnowing Where You're Going
Groups that can't even agree on what to do can often find themselves debating about how to do it. Here are some simple things to remember to help you focus on defining the goal.

See also Personal, Team, and Organizational Effectiveness and Critical Thinking for more related articles.

Coaching services

I offer email and telephone coaching at both corporate and individual rates. Contact me for details at rbrenner@ChacoCanyon.com or (617) 491-6289, or toll-free in the continental US at (866) 378-5470.

Get the e-book!

Past issues of Point Lookout are available in five e-books:
Reprinting this article
Are you a writer, editor or publisher on deadline? Are you looking for an article that will get people talking and get compliments flying your way? You can have 500 words in your inbox in one hour. License any article from this Web site. More info

Public seminars

Person-to-Person Communication for Project Managers
When Person-to-Person Communicationswe talk, listen, send or read emails, read or write memos, or when we leave or listen to voice mail messages, we're communicating person-to-person. And whenever we communicate person-to-person, we risk being misunderstood, offending others, feeling hurt, and being confused. There are so many ways for things to go wrong that we could never learn how to fix all the problems. A more effective approach avoids problems altogether, or at least minimizes their occurrence. In this very interactive program you'll learn a model of inter-personal communications that can help you stay out of the ditch. In those moments of intense involvement, when we're most likely to slip, you'll have a new tool to use to keep things constructive. Read more about this program. Here are some upcoming dates for this program:

The Race to the South Pole: The Power of Agile Development
On 14The Race to the Pole: An Application of Agile Development December 1911, four men led by Roald Amundsen reached the South Pole. Thirty-five days later, Robert F. Scott and four others followed. Amundsen had won the race to the pole. Amundsen's party returned to base on 26 January 1912. Scott's party perished. As historical drama, why this happened is interesting enough. Lessons abound. Among the more important lessons are those that demonstrate the power of the agile approach to project management and product development. Read more about this program. Here's an upcoming date for this program:

Human-Centered Risk Management
Most Human-Centered Risk Managementof us can assess technological risks, but risks related to human behavior tend to resist our best efforts. This session provides a framework for evaluating risks related to the behavior of individuals, teams, organizations and people generally. Human-centered risk differs from technological or market risk, because objective evaluation requires acknowledging personal and organizational limitations and failures. Since some of those limitations and failures might apply to the people assessing the risks, or to their superiors, there's a tendency to deny them or to explain them away. Our approach examines capability, organization, context, risk mitigation, and workplace politics. It has tools for guiding the assessment and management of human-centered risk, and we show how to extend these tools to suit your situation. You'll learn how to identify sources of risk in human behavior; recognize systemic and individual barriers to acknowledging risk; assess the effects of organizational turbulence; determine the risk associated with inappropriate internal risk transfer; estimate the effects of team dysfunction, toxic conflict and turnover; and measure the impact of workplace politics. Read more about this program. Here's an upcoming date for this program:

The Race to the South Pole: Ten Lessons for Project Managers
On 14The Race to the Pole: Ten Lessons for Project Managers December 1911, four men led by Roald Amundsen reached the South Pole. Thirty-five days later, Robert F. Scott and four others followed. Amundsen had won the race to the pole. Amundsen's party returned to base on 26 January 1912. Scott's party perished. As historical drama, why this happened is interesting enough, but to project managers, the story is fascinating. Lessons abound. Read more about this program. Here's an upcoming date for this program:

Project Management in Fluid Environments
Most Managing in Fluid Environmentspeople now work in environments that can best be characterized as fluid, because they're subject to continual change. We never know what's coming next. In such environments, managing — teams, projects, groups, departments, or the enterprise — often entails moving from surprise to surprise while somehow staying almost on track. It's a nerve-wracking existence. This program provides numerous tools that help project managers who work in fluid environments. Read more about this program. Here's an upcoming date for this program:

The Politics of Meetings for People Who Hate Politics
ThereThe Politics of Meetings for People Who Hate Politics's a lot more to running an effective meeting than having the right room, the right equipment, and the right people. With meetings, the whole really is more than the sum of its parts. How the parts interact with each other and with external elements is as important as the parts themselves. And those interactions are the essence of politics for meetings. This program explores techniques for leading meetings that are based on understanding political interactions, and using that knowledge effectively to meet organizational goals. Read more about this program. Here's an upcoming date for this program:

101 Tips for Targets of Workplace BulliesAre you being targeted by a workplace bully? Do you know what to do to end the bullying? Read 101 Tips for Targets of Workplace Bullies to learn powerful methods for dealing with workplace bullies. Check it out!
Go For It: Sometimes It's Easier If You RunBad boss, long commute, troubling ethical questions, hateful colleague? Learn what we can do when we love the work but not the job.
303 Tips for Virtual and Global TeamsLearn how to make your virtual global team sing.
101 Tips for Managing ChangeAre you managing a change effort that faces rampant cynicism, passive non-cooperation, or maybe even outright revolt?
My free weekly email newsletter gives concrete tips and suggestions for dealing with the challenging but everyday situations we all face.
A Tip A DayA Tip a Day arrives by email, or by Yahoo! Widget, each business day. It's 20 to 30 words at most, and gives you a new perspective on the hassles and rewards of work life. Most tips also contain links to related articles. Free!
101 Tips for Effective MeetingsLearn how to make meetings more productive — and more rare.
Exchange your "personal trade secrets" — the tips, tricks and techniques that make you an ace — with other aces, anonymously. Visit the Library of Personal Trade Secrets.
If your teams don't yet consistently achieve state-of-the-art teamwork, check out this catalog. Help is just a few clicks away!
Ebooks, booklets and tip books on project management, conflict, writing email, effective meetings and more.
SSL