How Do We Measure Success?

One of the smartest government web managers I know, Gwynne Kostin, published an interesting blog piece yesterday, pointing out how numbers may not measure what you think they do. The ever-astute Sarah Bourne,a government web manager at the state level, added a comment to Gwynne’s post, echoing those concerns and adding that measuring what really counts – achieving a task – is tough to do. It got me thinking: how do we measure the success of government websites?

Seems important to start by understanding what government websites are supposed to do.

• Provide services to the public. We’re talking about the services that taxpayers pay for and that are the mission of government agencies.
• Offer opportunities for citizen engagement and collaboration. This is the whole Gov 2.0 frontier that web 2.0/social media has opened.
• Help make government transparent by publishing the most critical information, decisions, and data that the public wants and needs.

So…how do you measure a website’s effectiveness at carrying out those objectives? Here are some ideas.

Objective: Provide services to the public

Measure 1: How long does it take to complete (successfully) the most critical tasks/services?

As a reminder, most people who come to government websites come for a purpose – to do something, to complete a task, to use a service. They have busy lives; so they want to get in, get it done, and move on as fast as possible. Government’s job is to make those services easy to find and easy to use. So measure how long it takes an average person to use those services and then see if you can make it easier, faster.

Some of you will recognize this as usability testing. Pick your 3-5 top services/tasks, and run routine scenario tests of a half dozen (or more) typical users.

“You want to buy a home and don’t know how. We’re going to watch you find the answer on HUD’s website. We aren’t testing you – we’re testing our website. Tell us what you’re thinking as you go through the process.”

How many wrong turns did they take? What words didn’t they understand? Can you cut steps? Can you organize it better? Can you use other words to help them find the right path, get to the right result, faster? Measure how long it took them to complete the task. Then make improvements and see if you can trim that time.

Objective: Citizen engagement/collaboration

Measure 2: Percent of ideas considered and adopted.

Citizen engagement isn't just asking for input. It's doing something with it. Does it matter if a million people contributed ideas, if you don’t accept any of them? Does that make citizens feel part of their government? Does that increase public trust? On the other hand, if only 10 people contributed ideas, but you adopted 6 of them, isn’t that what collaboration is all about? Isn’t it about working together to make government better?

I know. Sometimes we get ideas from the public that just seem silly or are obviously (to us) unworkable. I wonder - is that their fault or is it partly our fault for not really collaborating…for not giving them enough background and guidance so that they can submit ideas that are possible and useful…for not giving them enough opportunities to discuss ideas – go back and forth – so the outcome can be a legitimate option?

It seems to me that the goal of engagement and collaboration is to come up with – together – positive outcomes. So I’d measure success by the percent of ideas accepted. And I’d make it my goal to improve the collaborative process so that percentage of positive outcomes increases over time.

Objective: Transparency

Measure 3: Percentage of data systems available to the public.

Yes, I know that transparency isn’t only about data. It’s about decisions and ideas under discussion, too. But I don’t know how to measure those. I do know that every agency is supposed to have (and publish) an inventory of all its data systems. So what percent of those systems (or the data from those systems) is available to the public?

And A Fourth Measure

My friend, Jeffrey Levy, at EPA ran an interesting little contest a couple of weeks ago. He asked us to guess what percentage of the EPA website is viewed at least 100 times a month. The answer? 3%. Does that number seem stunning to you? Probably not, if you’re a government web manager who routinely reviews his/her stats.

And there’s the flipside…what percentage of the website isn’t viewed EVEN ONCE during a month? I’ll bet it’s higher than you think.

Maybe one of our goals should be to improve that ratio. Increase the percentage of the website that is used often, and decrease the percentage that isn't used at all. Focus our resources on making the most-used content truly great. And stop wasting resources maintaining content that is never used. Make that content available upon request. I think citizens would view that as a good value and responsible government. So…

Measure 4: Percent of the website used by at least 100 people every month compared with percent of the website not used at all.

Why do we care about numbers? For bragging rights? To compare ourselves to others? Or to help us make our websites better? I’m going with that last one. Gwynne is right. Numbers can be deceiving. Let’s use numbers that really measure our success in doing the things that matter.