what is the difference between research process and research methodology, how to create a research framework, how to draw a research framework, how to construct a research framework
JohenCorner,France,Professional
Published Date:02-08-2017
Your Website URL(Optional)
Comment
Chapter 10
Research10
So far, we’ve focused on concepts and components. Now we’re going to shift gears
and explore the process and methods for creating information architectures.
If it were just a matter of whipping up a few standard blueprints, our jobs would be
easy. But as we’ve explained, information architecture doesn’t happen in a vacuum.
The design of complex web sites requires an interdisciplinary team that involves
graphic designers, software developers, content managers, usability engineers, and
other experts.
Effective collaboration requires agreement on a structured development process.
Even for smaller projects, when teams are tiny and individuals fill multiple roles,
tackling the right challenges at the right time is critical to success.
The following chapters provide an overview of the process and the challenges you’ll
encounter along the way. Our focus on the early stages of research, strategy, and
design, rather than the later stages of implementation and administration, belies our
consulting background. While the vast majority of our experiences have involved
strategy and design for fast-paced information architecture projects, we are true
believers in the importance of nailing the details in implementation and building sus-
tainable information architecture programs. The dedicated in-house staff who pro-
tect and perfect information architectures over the long haul are the unsung heroes
of the field.
231Process Overview
In the early days of web design, many companies employed a one-step process called
“Code HTML.” Everyone wanted to jump right in and build the site. People had no
patience for research or strategy. We remember one eager client asking us in the mid-
dle of a planning session, “So when are we going to start the real work?” Fortu-
nately, after several years of painful lessons, there’s a growing realization that
designing web sites is hard work and requires a phased approach. Figure 10-1 illus-
trates the process of information architecture.
Project
Research Strategy Design Implementation Administration
Program
Figure 10-1. The process of information architecture development
The research phase begins with a review of existing background materials and
meet-
ings with the strategy team, aimed at gaining a high-level understanding of the goals
and business context, the existing information architecture, the content, and the
intended audiences. It then quickly moves into a series of studies, employing a vari-
ety of methods to explore the information ecology.
This research provides a contextual understanding that forms the foundation for
development of an information architecture strategy. From a top-down
perspective,
this strategy defines the highest two or three levels of the site’s organization and navi-
gation structures. From a bottom-up perspective, it suggests candidate document
types and a rough metadata schema. This strategy provides a high-level framework
for the information architecture, establishing a direction and scope that will guide
the project through implementation.
Design is where you shape a high-level strategy into an information architecture,
cre-
ating detailed blueprints, wireframes, and metadata schema that will be used by
graphic designers, programmers, content authors, and the production team. This
phase is typically where information architects do the most work, yet quantity can-
not drive out quality. Poor design execution can ruin the best strategy. For an infor-
mation architect, the meat is in the middle and the devil is in the details.
Implementation is where your designs are put to the test as the site is built,
tested,
and launched. For the information architect, this phase involves organizing and tag-
ging documents, testing and troubleshooting, and developing documentation and
training programs to ensure that the information architecture can be maintained
effectively over time.
And last but not least comes administration, the continuous evaluation and
232 Chapter 10: Research
improve-
ment of the site’s information architecture. Administration includes the daily tasks of tagging new documents and weeding out old ones. It also requires monitoring site
usage and user feedback, identifying opportunities to improve the site through major
or minor redesigns. Effective administration can make a good site great.
Admittedly, this is a simplified view of the process. Clear lines rarely exist between
phases, and few projects begin with a clean slate. Budgets, schedules, and politics
will inevitably force you off the path and into the woods.
We don’t aim to provide a paint-by-numbers design guide. The real world is far too
messy. Instead, we present a framework and some tools and methods that may be
useful when applied selectively within your environment.
Before we begin, we’ll offer a word of encouragement. Much of this work looks
tedious and boring when taken out of context. Not all of us can get jazzed up about
poring over search logs and analyzing content. But when you do this work in the real
world, it can be surprisingly engaging. And when that magic light bulb turns on,
revealing a pattern that suggests a solution, you’ll be glad you took the time to do it
right.
A Research Framework
Good research means asking the right questions. And choosing the right questions
requires a conceptual framework of the broader environment.
We have found our faithful three-circle diagram shown in Figure 10-2 to be invalu-
able in shaping a balanced approach to research. It helps us to decide where to shine
the flashlight, and to understand what we see. Consequently, we have used this
model to organize our exploration of the research process.
Business goals, funding, politics,
culture, technology, human resources
Context
Content Users
Document/data types, Audiences, tasks, needs,
content objects, metadata, information seeking behavior,
volume, existing structure experience, vocabularies
Figure 10-2. A balanced approach to research
We begin with an overview of tools and methods for research (see Figure 10-3).
Obviously, it won’t make sense or be possible to use every tool on every project.
And, of course, you should absolutely seek out and try methods we haven’t covered.
Our goal is to provide you with a map and a compass. The journey is left to you.
A Research Framework 233 Background Presentations Stakeholder Technology
Context
research and meetings interviews assessment
Metadata
Heuristic Content
Content and content Benchmarking
evaluation mapping
analysis
Search log
Use cases and Contextual User interviews
Users and clickstream
personas inquiry and user testing
analysis
Figure 10-3. Tools and methods for research
Context
For practical purposes, an investigation of the business context can be a good place
to start. It’s critical to begin projects with a clear understanding of the goals and an
appreciation of the political environment. Ignoring business realities is just as dan-
gerous as ignoring users. A perfectly usable site that fails to support business goals
won’t last long. The term “user-centered design” is valuable insofar as it moves the
pendulum away from executive-centered design, but don’t let that pendulum swing
too far.
Of course, context isn’t just about politics. We also need to understand goals, budgets,
schedules, technology infrastructure, human resources, and corporate culture. Legal
issues can also be important, particularly in heavily regulated industries. All of these
factors can and should influence the shape of the information architecture strategy.
Getting Buy-In
Research is not a one-way street. While conducting your investigation, it’s impor-
tant to recognize the value of building awareness and support for your project. After
all, you’re not a scientist studying rats. Your human subjects will have their own set
of questions and concerns. For example:
• Who are you and why are you asking me these questions?
• What’s information architecture and why should I care?
• What’s your methodology and how does it relate to my work?
The way you answer these questions will influence the level of support you receive
throughout the project. Since most large sites today depend upon interdepartmental
collaboration and decentralized content ownership, it’s impossible to succeed with-
out broad buy-in. For this reason, you’ll want to weave elements of presentation and
persuasion throughout the research process.
234 Chapter 10: ResearchBackground Research
When a project begins, an information architect’s head is filled with all sorts of good
questions.
• What are the short- and long-term goals?
• What’s the business plan? What are the politics?
• What’s the schedule and budget?
• Who are the intended audiences?
• Why will people come to the site? Why will they come back?
• What types of tasks should users be able to perform?
• How will content be created and managed? And by whom?
• What’s the technical infrastructure?
• What worked in the past? What didn’t?
But just asking the right questions is not enough. You need to ask them of the right
people in the right way at the right time. You must be very focused in how you use
people’s time and realistic about who can answer which questions.
Consequently, it’s good to begin with a review of background materials. Sometimes
the best way to learn about the future is to dig into the past. Get your hands on any
documents that relate to the site’s mission, vision, goals, intended audiences, and
content. Also, try to find documents that provide a broader picture of the manage-
ment structure and culture. Organization charts are really valuable if you’re an out-
side consultant, particularly when working on intranets. They capture an important
component of the users’ mental model of the organization and will help you deter-
mine potential stakeholders and user groups for interviews and testing.
A revealing exercise is to compare the vision that preceded the current web site with
the actual site itself. In some cases, we’ve seen elaborate PowerPoint presentations,
hundreds of pages long, that paint a tremendously ambitious picture of what the web
site should be. And then we’ve looked to the Web and found a small, poorly
designed site with limited functionality. This gap between vision and reality is a red
flag, suggesting misunderstanding between the managers who produce the slides and
the team who must build the site. Great visions are useless without the time, money,
and expertise to implement them. In these cases, you’ll need to rein in expectations
quickly.
Introductory Presentations
When you’re kicking off an information architecture project, it’s worth taking time
for an introductory presentation. It’s good to get authors, software developers,
graphic designers, marketing folks, and managers all on the same page in under-
standing the following issues.
Context 235 • What is information architecture and why is it important?
• How will the information architecture relate to the other components of the site
and to the organization itself?
• What are the major milestones and deliverables?
These presentations and the discussions they provoke can identify potential land-
mines and foster productive relationships between teams. They are especially useful
in building a common vocabulary that helps people communicate with one another
more successfully.
Research Meetings
In the early 1990s, we held full-day marathon meetings with our clients’ web teams
to learn as much as possible about mission, vision, audience, content, and infrastruc-
ture, and to begin fleshing out a framework for the information architecture. In those
days of small, centralized web design teams, one mammoth research meeting would
often suffice. Today, the design and production of web sites is often more compli-
cated, involving several teams drawn from different departments. This distributed
reality may call for a series of targeted research meetings. Consider the following
three meetings and their agendas.
Strategy team meeting
In many organizations today, there’s a centralized strategy team or working group
that’s been tasked with management of the web or intranet effort. It’s this strategy
team that sets the high-level goals, defining the mission, vision, intended audience,
content, and functionality. This is the group that deals with the big balancing act
between centralization and autonomy.
Because of the need to establish trust and respect, face-to-face meetings with this
team are essential. Only by having these meetings will you learn about the real goals
of the project and the hidden landmines in your path. And only during face-to-face
conversations will you reach a comfort level that allows both you and your col-
leagues to ask the difficult but necessary questions.
It’s important to keep these meetings small and informal. Five to seven people is
ideal. If the group gets too large, political correctness takes over and people won’t
talk. As far as the agenda goes, you’ll want to hit on some of the following questions:
• What are the goals for this site?
• Who are the intended audiences?
• What is the planned content and functionality?
• Who will be involved in this effort?
• When do you need to show results?
• What obstacles do you anticipate?
236 Chapter 10: Research However, the key in these meetings is to follow your nose. Be ready to dig deeper
into the most interesting and important topics that come up. The worst thing you
can do is rigidly stick to a formal agenda. Think of yourself as the facilitator, not the
dictator. And don’t be afraid to let the discussion wander a bit. You’ll learn more,
and everyone will have a more enjoyable meeting.
Content management meeting
The content owners and managers are the people you’ll want to engage in detailed
discussions about the nature of the content and the content management process.
These people typically have lots of hands-on experience and a perspective more
informed by bottom-up realities. If you can establish a rapport, you might also learn
a lot about the culture and politics of the organization as well. Questions for these
folks include:
• What are the formal and informal policies regarding content inclusion?
• Is there a content management system that handles authoring and publishing?
• Do those systems use controlled vocabularies and attributes to manage content?
• How is content entered into the system?
• What technology is being used?
• What content does each owner handle?
• What is the purpose of the content? What are the goals and vision behind this
content area?
• Who is the audience?
• What is the format of the content? Is it dynamic or static?
• Who maintains the content?
• What future content or services are planned?
• Where does content originate? How is it weeded?
• What legal issues impact the content management process?
Information technology meeting
You should meet with the system administrators and software developers early on to
learn about the existing and planned technical infrastructure that will support the
web site or intranet. This provides a good opportunity to discuss the relationships
between information architecture and technical infrastructure, as well as to build
trust and respect. Remember, you depend on these folks to forge the connection
between ideas and implementation. Questions include:
• Will we be able to leverage content management software (CMS)?
• How can we create a metadata registry to support distributed tagging?
• Does the CMS handle automated categorization of documents?
Context 237 • What about automated browsable index generation?
• What about personalization?
• How flexible is the search engine?
• Will the search engine support integration of a thesaurus?
• Can we get regular access to search logs and usage statistics?
Unfortunately, the IT groups in many organizations are swamped with work and
don’t have the time to support information architecture and usability efforts. It’s
important to identify this problem early and develop a practical, realistic solution.
Otherwise, your whole effort can stall when implementation time arrives.
Stakeholder Interviews
Interviews with opinion leaders or stakeholders are often one of the most valuable
components of the business context research. These interviews with senior execu-
tives and managers from a variety of departments and business units allow for
broader participation in the process and bring new perspectives, ideas, and resources
to the table.
During these interviews, the information architect asks the opinion leaders open-
ended questions about their assessment of the current information environment and
their vision for the organization and its web site. It’s worth taking the time to explain
your project to these folks—their political support may be more important in the
long haul than the answers they give during the interview. Sample questions for an
intranet project include:
• What is your role in the organization? What does your team do?
• In an optimal world, how would your company use the intranet to build com-
petitive advantage?
• In your opinion, what are the key challenges your company intranet faces?
• What enterprise-wide initiatives are occurring that the intranet strategy team
should know about?
• Do you use the existing intranet? If not, why not? If so, what parts of the intra-
net do you use? How often?
• What incentives exist for departments and employees to share knowledge?
• What are the critical success factors for the intranet?
• How will these factors be measured? What’s the ROI?
• What are the top three priorities for the intranet redesign?
• If you could tell the intranet strategy team one thing, what would it be?
• What question should we have asked that we didn’t?
As with the strategy team meeting, these interviews should be informal discussions.
Let the stakeholders tell you what’s on their minds.
238 Chapter 10: ResearchTechnology Assessment
In our dream world, we would design our information architectures independent of
technology, and then a team of system administrators and software developers
would build the infrastructure and tools to support our vision.
In the real world, this doesn’t happen very often. Usually, we must work with the
tools and infrastructure already in place. This means that we need to assess the IT
environment at the very beginning of a project so that our strategies and designs are
grounded in reality.
This is why it’s critical to talk with IT folks up front. You’ll want to understand
what’s in place, what’s in process, and who’s available to help. Then you can per-
form a gap analysis, identifying the disconnects between business goals, user
needs,
and the practical limitations of the existing technology infrastructure.
You can then see if there are any commercially available tools that might help to
close these gaps, and you can initiate a process to determine whether it’s practical to
integrate them within the context of the current project. (We’ll discuss tools for
information architects in more detail in Chapter 16.) Either way, it’s much better to
come to terms with these IT issues early on.
Content
We define content broadly as “the stuff in your web site.” This may include docu-
ments, data, applications, e-services, images, audio and video files, personal web pages,
archived email messages, and more. And we include future stuff as well as present stuff.
Users need to be able to find content before they can use it—findability
precedes
usability. And if you want to create findable objects, you must spend some time
studying those objects. You’ll need to identify what distinguishes one object from
another, and how document structure and metadata influence findability. You’ll
want to balance this bottom-up research with a top-down look at the site’s existing
information architecture.
Heuristic Evaluation
Many projects involve redesigning existing web sites rather than creating new ones.
In such cases, you’re granted the opportunity to stand on the shoulders of those who
came before you. Unfortunately, this opportunity is often missed because of peo-
ple’s propensity to focus on faults and their desire to start with a clean slate. We reg-
ularly hear our clients trashing their own web sites, explaining that the current site is
a disaster and we shouldn’t waste our time looking at it. This is a classic case of
throwing out the baby with the bathwater. Whenever possible, try to learn from the
existing site and identify what’s worth keeping. One way to jump-start this process is
to conduct a heuristic evaluation.
Content 239 A heuristic evaluation is an expert critique that tests a web site against a formal
or
informal set of design guidelines. It’s usually best to have someone outside the orga-
nization perform this critique, so this person is able to look with fresh eyes and be
largely unburdened with political considerations. Ideally, the heuristic evaluation
should occur before a review of background materials to avoid bias.
At its simplest, a heuristic evaluation involves one expert reviewing a web site and
identifying major problems and opportunities for improvement. This expert brings
to the table an unwritten set of assumptions about what does and doesn’t work,
drawing upon experiences with many projects in many organizations.
This practice is similar to the physician’s model of diagnosis and prescription. If your
child has a sore throat, the doctor will rarely consult a reference book or perform
extensive medical tests. Based on the patient’s complaints, the visible symptoms, and
the doctor’s knowledge of common ailments, the doctor will make an educated guess
as to the problem and its solution. These guesses are not always right, but this single-
expert model of heuristic evaluation often provides a good balance between cost and
quality.
At the more rigorous and expensive end of the spectrum, a heuristic evaluation can
be a multi-expert review that tests a web site against a written list of principles and
guidelines. This list may include such common-sense guidelines as:
• The site should provide multiple ways to access the same information.
• Indexes and sitemaps should be employed to supplement the taxonomy.
• The navigation system should provide users with a sense of context.
• The site should consistently use language appropriate for the audience.
• Searching and browsing should be integrated and reinforce each other.
Each expert reviews the site independently and makes notes on how it fares with
respect to each of these criteria. The experts then compare notes, discuss differ-
ences, and work toward a consensus. This reduces the likelihood that personal opin-
ion will play too strong a role, and creates the opportunity to draw experts from
different disciplines. For example, you might include an information architect, a
usability engineer, and an interaction designer. Each will see very different problems
and opportunities. This approach obviously costs more, so depending on the scope
of your project, you’ll need to strike a balance in terms of number of experts and for-
mality of the evaluation.
For a good example of such a list, see Jakob Nielsen’s Ten Usability Heuristics
(http://www.useit.com/papers/
heuristic/heuristic_list.html).
240 Chapter 10: ResearchContent Analysis
Content analysis is a defining component of the bottom-up approach to information
architecture, involving careful review of the documents and objects that actually
exist. What’s in the site may not match the visions articulated by the strategy team
and the opinion leaders. You’ll need to identify and address these gaps between top-
down vision and bottom-up reality.
Content analysis can take the shape of an informal survey or a detailed audit. Early in
the research phase, a high-level content survey is a useful tool for learning about the
scope and nature of content. Later in the process, a page-by-page content audit or
inventory can produce a roadmap for migration to a content management system
(CMS), or at least facilitate an organized approach to page-level authoring and design.
Gathering content
To begin, you’ll need to find, print, and analyze a representative sample of the
site’s
content. We suggest avoiding an overly scientific approach to sample definition.
There’s no formula or software package that will guarantee success. Instead, you
need to use some intuition and judgment, balancing the size of your sample against
the time constraints of the project.
We recommend the Noah’s Ark approach. Try to capture a couple of each type of
animal. Our animals are things like white papers, annual reports, and online reim-
bursement forms, but the difficult part is determining what constitutes a unique spe-
cies. The following dimensions should help distinguish one beast from another and
build toward a diverse and useful content sample:
Format
Aim for a broad mix of formats, such as textual documents, software applica-
tions, video and audio files, and archived email messages. Try to include offline
resources such as books, people, facilities, and organizations that are repre-
sented by surrogate records within the site.
Document type
Capturing a diverse set of document types should be a top priority. Examples
include product catalog records, marketing brochures, press releases, news arti-
cles, annual reports, technical reports, white papers, forms, online calculators,
presentations, spreadsheets, and the list goes on.
Source
Your sample should reflect the diverse sources of content. In a corporate web
site or intranet, this will mirror the organization chart. You’ll want to make sure
you’ve got samples from engineering, marketing, customer support, finance,
human resources, sales, research, etc. This is not just useful—it’s also politically
astute. If your site includes third-party content such as electronic journals or ASP
services, grab those, too.
Content 241Subject
This is a tricky one, since you may not have a topical taxonomy for your site.
You might look for a publicly available classification scheme or thesaurus for
your industry. It’s a good exercise to represent a broad range of subjects or top-
ics in your content sample, but don’t force it.
Existing architecture
Used together with these other dimensions, the existing structure of the site can
be a great guide to diverse content types. Simply by following each of the major
category links on the main page or in the global navigation bar, you can often
reach a wide sample of content. However, keep in mind that you don’t want
your analysis to be overly influenced by the old architecture.
Consider what other dimensions might be useful for building a representative con-
tent sample for your particular site. Possibilities include intended audience, docu-
ment length, dynamism, language, and so on.
As you’re balancing sample size against time and budget, consider the relative num-
ber of members of each species. For example, if the site contains hundreds of techni-
cal reports, you certainly want a couple of examples. But if you find a single white
paper, it’s probably not worth including in your sample. On the other hand, you do
need to factor in the importance of certain content types. There may not be many
annual reports on your web site, but they can be content-rich and very important to
investors. As always, your judgment is required.
A final factor to consider is the law of diminishing returns. While you’re conducting
content analysis, you’ll often reach a point where you feel you’re just not learning
anything new. This may be a good signal to go with the sample you’ve got, or at least
take a break. Content analysis is only useful insofar as it teaches you about the stuff
in the site and provides insights about how to get users to that stuff. Don’t just go
through the motions. It’s unproductive and incredibly boring.
Analyzing content
What are you looking for during content analysis? What can you hope to learn? One
of the side benefits of content analysis is familiarity with the subject matter. This is
particularly important for consultants who need to quickly become fluent in the lan-
guage of their client. But the central purpose of content analysis is to provide data
that’s critical to the development of a solid information architecture. It helps you
reveal patterns and relationships within content and metadata that can be used to
better structure, organize, and provide access to that content. That said, content
analysis is quite unscientific. Our approach is to start with a short list of things to
look for, and then allow the content to shape the process as you move forward.
242 Chapter 10: ResearchFor example, for each content object, you might begin by noting the following:
Structural metadata
Describe the information hierarchy of this object. Is there a title? Are there dis-
crete sections or chunks of content? Might users want to independently access
these chunks?
Descriptive metadata
Think of all the different ways you might describe this object. How about topic,
audience, and format? There should be at least a dozen different ways to describe
many of the objects you study. Now’s the time to get them all on the table.
Administrative metadata
Describe how this object relates to business context. Who created it? Who owns
it? When was it created? When should it be removed?
This short list will get you started. In some cases, the object will already have meta-
data. Grab that, too. However, it’s important not to lock into a predefined set of
metadata fields. You want to allow the content to speak to you, suggesting new fields
you might not have considered. You’ll find it helpful to keep asking yourself these
questions:
• What is this object?
• How can I describe this object?
• What distinguishes this object from others?
• How can I make this object findable?
Moving beyond individual items, also look for patterns and relationships that emerge
as you study many content objects. Are certain groupings of content becoming
apparent? Are you seeing clear hierarchical relationships? Are you recognizing the
potential for associative relationships, perhaps finding disparate items that are linked
by a common business process?
Because of the need to recognize patterns within the context of the full sample, con-
tent analysis is by necessity an iterative process. It may be on the second or third pass
over a particular document that the light bulb blinks on and you discover a truly
innovative and useful solution.
With the exception of true bottom-up geeks (and we use the term respectfully), most
of us don’t find content analysis especially thrilling or addictive. However, experi-
ence has proven that this careful, painstaking work can suggest new insights and pro-
duce winning information architecture strategies. In particular, content analysis will
help you in the design phase, when you begin fleshing out document types and meta-
data schema. But it also provides valuable input into the broader design of organiza-
tion, labeling, navigation, and searching systems.
Content 243Content Mapping
Heuristic evaluation provides a top-down understanding of a site’s organization and
navigation structures, while content analysis provides a bottom-up understanding of
its content objects. Now it’s time to bridge these two perspectives by developing one
or more content maps.
A content map is a visual representation of the existing information environment
(see
Figure 10-4). Content maps are typically high-level and conceptual in nature. They
are a tool for understanding, rather than a concrete design deliverable.
Customer
Content sources Marketing Legal
Support
Content model Product Process Reference
Contact System
Content types
Handling How To
Content templates Steps Table
Figure 10-4. A small slice of a content map
Content maps vary widely. Some focus on content ownership and the publishing
process. Some are used to visualize relationships between content categories. And
others explore navigation pathways within content areas. The goal of creating a con-
tent map is to help you and your colleagues wrap your minds around the structure,
organization, and location of existing content, and ultimately to spark ideas about
how to provide improved access.
Benchmarking
We use the term benchmark informally to indicate a point of reference from which
to
make comparative measurements or judgments. In this context, benchmarking
involves the systematic identification, evaluation, and comparison of information
architecture features of web sites and intranets.
These comparisons can be quantitative or qualitative. We might evaluate the num-
ber of seconds it takes a user to perform a task using competing web sites, or take
244 Chapter 10: Research notes about the most interesting features of each site. Comparisons can be made
between different web sites (competitive benchmarking) or between different ver-
sions of the same web site (before-and-after benchmarking). In both cases, we’ve
found benchmarking to be a flexible and valuable tool.
Competitive benchmarking
Borrowing good ideas, whether they come from competitors, friends, enemies, or
strangers, comes naturally to all of us. It’s part of our competitive advantage as
human beings. If we were all left to our own devices to invent the wheel, most of us
would still be walking to work.
However, when we take these copycat shortcuts, we run the risk of borrowing bad
ideas as well as good ones. This happens all the time in the web environment. Since
the pioneering days of web site design, people have repeatedly mistaken large finan-
cial outlays and strong marketing campaigns as signs of good information architec-
ture. Careful benchmarking can catch this misdirected copycatting before it gets out
of control.
For example, when we worked with a major financial services firm, we ran up against
the notion that Fidelity Investments’ long-standing position as a leader within the
industry automatically conferred the gold standard upon its web site. In several
cases, we proposed significant improvements to our client’s site but were blocked by
the argument, “That’s not how Fidelity does it.”
To be sure, Fidelity is a major force in the financial services industry, with a broad
array of services and world-class marketing. However, in 1998, the information
architecture of its web site was a mess. This was not a model worth following. To
our client’s credit, they commissioned a formal benchmarking study, during which
we evaluated and compared the features of several competing sites. During this
study, Fidelity’s failings became obvious, and we were able to move forward without
that particular set of false assumptions.
The point here is that borrowing information architecture features from competitors
is valuable, but it must be done carefully.
Before-and-after benchmarking
Benchmarking can also be applied to a single site over time to measure improve-
ments. We can use it to answer such return-on-investment (ROI) questions as:
• How much did the intranet redesign reduce our employees’ average time finding
core documents?
• Has the web site redesign improved our customers’ ability to find the products
they need?
• Which aspects of our redesign have had a negative impact on user efficiency or
effectiveness?
Content 245 Before-and-after benchmarking forces you to take the high-level goals expressed in
your statement of mission and vision, and tie them to specific, measurable criteria.
This forced clarification and detail-orientation will drive you toward a better infor-
mation architecture design on the present project, in addition to providing a point of
reference for evaluating success.
Following are the advantages of before-and-after benchmarking, as well as those of
competitive benchmarking:
Benefits of before-and-after benchmarking
• Identifies and prioritizes information architecture features in the existing site
• Encourages transition from broad generalizations (e.g., “Our site’s navigation
stinks”) to specific, actionable definitions
• Creates a point of reference against which you can measure improvements
Benefits of competitive benchmarking
• Generates a laundry list of information architecture features, bringing lots of
new ideas to table
• Encourages transition from broad generalizations (e.g., “Amazon is a good model”)
to specific, actionable definitions (“Amazon’s personalization feature works well
for frequent visitors”)
• Challenges embedded assumptions (e.g., “We should be like Fidelity”) and avoids
copying the wrong features for the wrong reasons
• Establishes current position with respect to competitors and creates a point of
reference against which to measure speed of improvement
Users
They’re called users, respondents, visitors, actors, employees, customers, and more.
They’re counted as clicks, impressions, advertising revenues, and sales. Whatever
you call them and however you count them, they are the ultimate designers of the
Web. Build a web site that confuses customers, and they’ll go elsewhere. Build an
intranet that frustrates employees, and they won’t use it.
This is the Internet’s fast-forward brand of evolution. Remember the original Path-
finder web site from Time Warner? They spent millions of dollars on a flashy, graph-
ical extravaganza. Users hated it. A complete redesign followed months after the
original launch. This was an expensive and embarrassingly public lesson in the
importance of user-sensitive design.
So, we’ve established that users are powerful. They’re also complex and unpredict-
able. You can’t blindly apply lessons learned by Amazon to the information architec-
ture design of Pfizer.com. You’ve got to consider the unique nature of the site and of
the user population.
246 Chapter 10: Research
There are many ways to study user populations. Market-research firms run focus
groups to study branding preferences. Political pollsters use telephone surveys to
gauge the public’s feelings about candidates and issues. Usability firms conduct
interviews to determine which icons and color schemes are most effective. Anthro-
pologists observe people acting and interacting within their native environments to
learn about their culture, behavior, and beliefs.
No single approach can stand alone as the one right way to learn about users and
their needs, priorities, mental models, and information-seeking behavior. This is a
multidimensional puzzle—you’ve got to look at it from many different perspectives
to get a good sense of the whole. It’s much better to conduct five interviews and five
usability tests than to run one test ten times. Each approach is subject to the law of
diminishing returns.
As you consider integrating these user research methods into your design process,
keep a couple of things in mind. First, observe the golden rule of discount usability
engineering: any testing is better than no testing. Don’t let budgets or schedules
become an excuse. Second, remember that users can be your most powerful allies.
It’s easy for your colleagues and your boss to argue with you, but it’s difficult for
them to argue with their customers and with real user behavior. User research is an
extremely effective political tool.
Usage Statistics
Most projects today involve redesigning an existing site. In these cases, it makes
sense to begin by looking at data that shows how people have been using the site and
where they’ve been running into problems.
Your site’s usage statistics are a reasonable place to start. Most statistics software
packages, such as Google Analytics shown in Figure 10-5, provide the following
reports:
Page information
The number of hits per day for each page in the site. This data will show which
pages are most popular. By tracking page hits over time, you can observe trends
and tie page popularity to events such as advertising campaigns or the redesign
of site navigation.
Visitor information
Statistics products claim they can tell you who is using your site and where the
users are coming from. In reality, they’ll tell you only the domains (e.g., aol.com,
mitre.org) of those users’ Internet service providers, which is often of limited value.
If you’d like to dig deeper, we recommend reading User and Task Analysis for Interface Design by
Joann
Hackos and Janice Redish (Wiley). And then, of course, there are all sorts of wonderful articles and books
by usability guru Jakob Nielsen (http://useit.com).
Users 247 Your stats software may provide additional views into the usage data, indicating the
times and dates when people are visiting, the referring sites your users are coming
from, and the types of browsers being used, as shown in Figure 10-5.
Figure 10-5. Usage data presented by Google Analytics
The path that users trace as they move through a web site is known as the
clickstream. If you want a higher level of sophistication in your usage statistics,
you
can buy software that handles clickstream analysis. You can trace where a user
comes from (originating site), the path he takes through your site, and where he goes
next (destination site). Along the way, you can learn how long he spends on each
page of your site. This creates a tremendously rich data stream that can be fascinat-
ing to review, but difficult to act upon. What you really need to make clickstream
data valuable is feedback from the user explaining why he came to the site, what he
found, and why he left. Some companies use pop-up surveys to capture this informa-
tion as users are leaving the web site.
Search-Log Analysis
A simpler and extremely valuable approach involves the tracking and analysis of que-
ries entered into the search engine. By studying these queries, you can identify what
users are looking for, and the words and phrases they are using. This is fantastic data
when you’re developing controlled vocabularies. It’s also useful when prioritizing
terms for a “Best Bets” strategy. (You’ll learn more about Best Bets in the MSWeb
case study in Chapter 20.)
248 Chapter 10: Research At a basic level, search-log analysis will sensitize you to the way your users really
search. Users generally enter one or two keywords, and you’re lucky if they spell
them right. Looking at search logs provides a valuable education for information
architects who are fresh out of school and all steamed up about the power of Bool-
ean operators and parenthetical nesting. You can achieve the same effect using a live
search display such as Metacrawler’s metaspy, which shows the terms that real peo-
ple are using to search right now (see Figure 10-6).
Figure 10-6. A public search voyeur service
But with your own site’s search logs, you can learn much more. At a bare minimum,
you should be able to get a monthly report that shows how many times users
searched on particular terms during that month, as shown here:
54 e-victor
53 keywords:"e-victor"
41 travel
41 keywords:"travel"
37 keywords:"jupiter"
37 jupiter31 esp
See http://searchenginewatch.com/facts/searches.html for more on live search
displays.
Users 24930 keywords:"esp"
28 keywords:"evictor"
28 evictor
28 keywords:"people finder"
28 people finder
27 fleet
27 keywords:"fleet"
27 payroll
26 eer
26 keywords:"eer"
26 keywords:"payroll"
26 digital badge
25 keywords:"digital badge"
But hopefully, you can work with your IT group to buy or build a more sophisti-
cated query-analysis tool that allows you to filter by date, time, and IP address.
Figure 10-7 shows a good example of such a tool. This tool can help you answer the
following questions:
• Which popular queries are retrieving zero results?
• Are these zero-hit users entering the wrong keywords, or are they looking for
stuff that doesn’t exist on your site?
• Which popular queries are retrieving hundreds of results?
• What are these hundred-hit users actually looking for?
• Which queries are becoming more popular? Less?
Based on the answers, you can take immediate and concrete steps to fix problems
and improve information retrieval. You might add preferred and variant terms to
your controlled vocabulary, change navigation labels on major pages throughout the
site, improve search tips, or edit content on the site. Note that smart marketing
groups are also getting interested in search logs as a valuable source of information
about customer needs.
Customer-Support Data
In addition to reviewing web site statistics, it’s worth looking to the customer- or
technical-support departments to see if they’ve been capturing and analyzing the
problems, questions, and feedback from the customers of your web site or intranet.
Help-desk operators, call-center representatives, librarians, and administrative assis-
tants can also be rich sources of information; in many large corporations, these are
the people to whom customers or employees turn for answers. That means they are
the people who know the questions.
250 Chapter 10: Research
Advise:Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.