Using Data to Inform Information Architecture and User Experience

Editor's Notes

• #2: Good morning everyone I’m Matthew Edgar. Thanks for being here with me this morning to talk about using data.
• #3: I’m guessing some of you have been in conversations like this as well. For me, this is pretty much a daily occurrence. When I work with clients, there is this idea that if you just get the right data—big, small, great, or otherwise—then all the problems will magically be solved!
• #4: —Big Data
• #5: Small Data
• #6: Great Data
• #7: or just data.
• #8: It is easy to understand this mindset. The whole "Big Data" movement is on the rise. It is everywhere. This is a trend graph from Google about searches for big data, which appropriately is based on a big data set. You can see this has really been trending up since 2012.
• #9: The truth is there is lots of data available today about our users in so many places from data in web logs or email campaigns, to mobile apps and increasingly social media channels. Plus, you can download big data sets from software tools and other libraries. Not to mention all the data you have internally in your CRM or from surveys, interviews, focus groups, and user testing. With all this data around us, it really isn't surprising people tend to assume that data holds the answers to increasing user satisfaction, increasing conversions, or whatever else we may desire.
• #10: Except… …despite its popularity, hardly anybody is really using big data…
• #11: Only 15% of Fortune 500 companies use “big data”
• #12: Just over a third of startups aren't even considering Big Data.
• #13: Other studies show that only 8% of companies are regularly using data.
• #14: Companies who use data are only somewhat successful in doing so.
• #15: The reality is more often than not the data is getting ignored and usually gets ignored in favor of opinions. I’m guessing some of you have witnessed this first hand. I know I’ve been in situations where we’ve collected and analyzed data that shows a super clear direction that needs to be taken to best support users and support business growth yet the results gets ignored. Why? There are usually justifications for ignoring data, like…
• #16: the recommendations aren't "in line with our brand“…
• #17: …or they would be "too technically hard" to implement…
• #18: or “our customers or users won’t like that”, but rarely are those justifications backed by any real proof…it is just opinions shooting down data.
• #19: I think we can all agree that ignoring data in favor of opinions isn't exactly the best approach to take. We are dealing with complicated problems that need to factor in business and user goals, and opinions alone aren’t going to help us answer the questions.
• #20: So, let’s get to the real problem. We all know data is important and that it can be useful…there are lots of examples of it and clearly there is a lot of talk about putting data to work. But, in most cases, data just isn't getting used as effectively as it ought to be. So, instead of talking about all the great things you can do with data, let's talk about the real problem and how to solve it…
• #21: The real problem is why isn’t data used within our organizations? Why are opinions winning out over data? How do we help our clients or employers or our teams use data to its fullest potential or for that matter just use data at all?
• #22: No matter what company size you are in or how much time or money you have, the question we need to figure out is how do you work in any amount of data into the day-to-day processes? My goal with this talk is to share with you a simple and easy plan for how you can start working data into your organization to get away from opinions leading the way.
• #23: So, when companies start thinking about using data, a lot of companies start big, usually by adopting a "Data Driven" process. On the surface, this "Data Driven" approach sounds good – why not use all the data we have to make decisions for us? I don't have to bother with figuring out my content or this sitemap, instead I'll just let the data tell me what to do! And it is tempting to adopt this big data-driven approach because it sounds easy to let data take control. Opinions are bad, let’s get rid of them all.
• #24: Unfortunately, "Data Driven" doesn't really seem to work all that well.
• #25: An example I've seen numerous times is in regards to navigation. Various sets of data—from tests I've run to tests others have run—clearly show that removing navigation from a page can increase conversion rates. Users who see the page without navigation are more likely to convert because there is less distraction.  
• #26: Makes sense. So, with this data in the driver seat we know exactly what we need to do: remove all the navigation from the entire site. Now, that might sound crazy to some of us in this room, but I've had clients do this and they do get more conversions but those same clients have also gotten lower customer satisfaction and retention rates in the future. Why is that? They followed what the data said but removing the navigation not only increased conversion rates in turn it also frustrated the people who visited the site.
• #27: So, the lesson here is this: letting the numbers— such as conversion rates—blindly drive your decision may sound like a good idea but it may also lead you toward a bad decision.
• #28: This is not to say “don’t use data”. You need to use data effectively. However, if some data clearly shows you a path to take, that doesn't mean you necessarily must follow that path. Your opinions and expertise do have to factor in here as well.
• #29: The Chief Scientist at Accenture Technologies said it best: improper use of data can lead to poor decisions made with high confidence. That’s the real problem that happens when you let data drive the decisions.
• #30: So, opinions can be problematic—we all know that. But, data isn't always exactly clear either. Instead of favoring one over the other, it is about finding a balance between data and opinions to avoid poor decisions made with high confidence.
• #31: So to use data properly and to strike that balance between what the data says and what your experience indicates, I want to review the framework I use. This framework starts with a subtle shift in our focus on data.
• #32: Instead of thinking of data as the driver, we want to think of data as the navigator.
• #33: What do I mean by that? Well, you definitely want at least some data to help inform your decisions but you also need to factor in intuition, opinions, and experience.
• #34: Let’s go back to the navigation example I mentioned earlier, where the data told us we need to get rid of the navigation to increase conversions. I'm guessing some of you were a little hesitant of the idea of removing the navigation—after all, navigation does serve a valid purpose. So while, yeah, the data says navigation can hurt conversions, your experience working with users probably would tell you that removing the navigation might not be such a great idea and maybe there are some other things we want to check first. Keeping data as a navigator, alongside your opinions and expertise will lead you toward the best decisions.
• #35: So, what does a data navigated world that balances opinions and data look like? It isn’t all that complex or intricate. Really it comes down to four steps to using data:
• #36: First, you need to know what question you need to answer.
• #37: Second, you need to know how that question relates to different perspectives.
• #38: Third, you need to collect your data.
• #39: Fourth, you need to review the data in context and make a decision.
• #40: The first step sounds simple, but I’ve found it is the hardest: what question am I trying to answer?
• #41: This is the step where people fail because if you ask the wrong question, you will end up collecting the wrong data, which will lead you to making "poor decisions with high confidence".
• #42: The problem with this step is that people want to skip ahead and just collect data and look at the numbers…because that is the fun part! But before we get the data, we need to know what question we're really asking so that we know what data we should even collect.
• #43: Let's walk through the page and navigation example I used before. What question are we asking?
• #44: Now, you could say that the question is "How do I get more conversions from this page?" – the reason this page exists is to get people to sign up or buy our client or employer’s service. I’m guessing your boss or client thinks that is the question to ask. But, we know if we ask the question this way we’ll end up taking away the navigation and could end up frustrating our users and hurting our company longer term. So, ask yourself and your team, is the only purpose of this page to drive conversions? Isn’t there more to it? Conversions matter but…the bigger question to ask is…
• #45: What is the best way to design this page? That is a that gives us many different ways to answer it and allows us to look at more data points than just conversions. Asking the question this way also lets us look at the navigation, but also measure other aspects of the page when it comes time to collect the data. Not everybody may agree on the bigger question, and that is okay because the goal here is to make sure that everybody at least somewhat knows what the question is that you are trying to answer.
• #46: Once we have that bigger and broad question, the next step is to break it down and determine what the different perspectives, or ways of looking at that question, really are. PAUSE AND DRINK SOME WATER!!! So, to keep on with that example of our page, we want to know what are the various perspectives—or pieces—of designing our page?
• #47: One perspective is certainly about conversions—the business case. The goal of the business is to drive conversions—that might be sales, leads, donations, sign ups, applications, or whatever we care about for our clients or employers. The business case almost always applies to any question we're going to ask or to any project we’re working on..
• #48: However, we should also factor in the user's perspective, which is really about achieving user satisfaction. Do your users feel satisfied when they reach this page? Not just looking at if they can find the call to action button, but also measuring if they really getting what they need and want from this page.
• #49: Another perspective to this page in question might be brand loyalty—a marketing case. Are people who reach this page more or less likely to have a positive perception of our company? Are people who sign up through this page more or less likely to remain loyal customers of our company for years to come?
• #50: We can come up with way more perspectives in this second step. . I tend to limit this to 5, as to not overwhelm yourself and get stuck on this step indefinitely..  The point of gaining perspective on the question is to see it from multiple angles, not necessarily to see it from all angles.
• #51: To  bring all that together for the example of the page, the question we're asking is: What's the best way to design this page?
• #52: And the perspectives for that question are: business case which wants to drive conversions, the user case to drive user satisfaction, the marketing case to drive brand loyalty. 
• #53: So, once you stepped back and more clearly understood the question being asked, it is time for the fun step: collecting data points for each perspective. We need a way to measure each piece of the question we're working on.
• #54: The first problem when collecting data is making it far more complicated than it needs to be and you end up collecting way too much data. I’m totally guilty of this problem. But, it is key to remember the idea isn’t to collect everything, just to collect something that matters. Just so long as you can get at least a couple meaningful data points for each perspective of the question you are asking.
• #55: Another pitfall to avoid is to ignore data you may already have available. It is tempting to start collecting new data, but taking a moment to look at the data you already have available can save you time and effort.
• #56: Another pitfall I see is that people don’t collect a mix of quantitative and qualitative data. To really get meaningful answers to our question, we want to aim to collect both.
• #57: All right, with those pitfalls in mind, how do you collect the data? The truth is no one data point I could show you how to collect would work in every situation because the data shifts depending on the question you are asking. There are lots of ways you can collect the data and within your organization there is likely already plenty of the data available that you need. Let’s walk through some examples…
• #58: Starting with quantitative data, the numbers, which is often easily found within your organization…
• #59: Analytics tools show web and mobile usage…you can usually obtain these from your IT or marketing team.
• #60: Then there are details about product usage, including features used or not used, longevity of use, frequency of use, churn rates, and more are usually available as well.
• #61: Server logs can be obtained from the IT department, giving even more details about users on your website or web app...
• #62: There are also numbers about how frequently customers reached out for support or the number of open issues that were resolved or not resolved.
• #63: Of course, you also have your employer's favorite numbers: sales…how much money came in the door? Or the amount of sign ups or donations?
• #64: Meanwhile, there is lots of qualitative data available too. These stories about our customers or users probably exist in some form …
• #65: If you are conducting user tests surveys, or interviews, you already have lots of information about your users. But even if you can't do user testing or extensive interviews, there are really cheap ways to do this now through services like Survey Monkey, Microworkers, Mechanical Turk, Usability Hub, UserTesting.com and more.
• #66: But if you can’t conduct user tests or pay for those services, there are also lots of people you can talk to in an organization. One great resource is the sales team. Sales teams have stories about customers and users at their fingertips—that’s the qualitative data we need. It is usually much cheaper and easier talking with your sales team instead of running a user test. It is an indirect source, but helps you get the qualitative data you need to understand your users, without having to arrange for dozens of user interviews.
• #67: Again, the point is to look at the data you already have and collect a mix of different types of data. You don't want qualitative data to get pushed aside in favor of the "cold hard numbers". But you also don't want to get so lost in qualitative data running survey after survey that you forget to look at the quantitative data.
• #68: And, as before, keep it simple. A little data collected quickly that can help you easily navigate the decision making process is far better than spending months setting up elaborate collection methods or extensive tests generating data nobody will use or understand.
• #69: To make the collection part more practical, let's walk through an example of collecting data in our example of that page, where our question was:
• #70: What's the best way to design this page—navigation or no navigation? And the perspectives for that question are: business case which wants to drive conversions, the user case to drive user satisfaction, the marketing case to drive brand loyalty. 
• #71: So, we aren’t going to worry about collecting every single piece of data possible—just one or two data points for each perspective.  
• #72: So, we know that one important perspective of our page question is the business case, which is all about getting more conversions.
• #73: We can track this in a web analytics platform  to see how many people are converting from this page—in this screenshot, this is Google Analytics goal reporting on conversions over a time period. To help clarify business intent, this report also shows us the channel that led to the conversion.  
• #74: But, we can use other data points too to understand conversions. You could certainly test users for this data. Or, a simpler answer would be to use heatmaps to see if people can spot your call to action buttons.
• #75: Or you can you use form analytics  to see what problems people have with the form…
• #76: The next perspective of this question is about the user’s thoughts on the page…
• #77: My guess is most of you in this room has done some form of user test. When doing so, you'll certainly ask questions that help you understand how users convert on this page. But, you also want to make sure to ask about the user’s needs too—what does the user want out of this page?
• #78: If you can’t run a test on your users for a particular page of the website to ask about their needs and expectations, a good substitute  is to look at the prior steps that led people to this page. Let's say some of the people arriving here came from a Google search result. We could look at what search results led people to this page, which helps us get an idea of user’s expectations and needs.
• #79: Or, if people are visiting a page within your website, you can also look at prior steps. This screenshot shows that report in Google Analytics.
• #80: Along with qualitative user testing, you could also inform this perspective of the question by looking at quantitative numbers like how long people spent on the page and how many visitors exited the page . You can bring the quantitative and qualitative data together to show that a large number of users are frustrated by this page—the design changes might have increased conversion rates, but now the group who didn’t convert is more frustrated with this page.
• #81: Our goal is to have a few data points for each perspective of the larger question we are aiming to answer—and not to worry so much about having every data point we possibly could.  Remember, the point of using data isn’t just to have lots of pretty Excel charts that get ignored in favor of opinions. Instead, the point is to make a decision that is at least somewhat informed by data.
• #82: Once you've collected data, you can move to the fourth step of the process of using data: making a decision. Unless you are a team of 1 with unlimited power to make all decisions yourself, you probably need to share the data you've collected with other people at the organization and they'll be involved in making a final decision.
• #83: This is where using data can get messy. There is a strong temptation to just dive into the data and make decisions outright on the findings. But before you look at any numbers you need to step back and remember the question and the different perspectives.
• #84: It is easy for the people making the decision to only look at one perspective of the question when coming up with an answer. Each decision maker probably cares the most about one perspective more than another. Most of us in this room are probably very user-centric, and we'll favor the user perspective. However our client or boss might only be interested in the business perspective—they want sales and conversions above all else. But we want to start our decision making by at least acknowledging that multiple perspectives exist. And here’s why…
• #85: Let’s go back to that example of our page, where we are trying to decide whether to keep the navigation or not. The big question is how to design a page. The data from the business case shows that the best page is the one that increased conversions that had no navigation. But, if you look at the data from the user case, the best page is the one with navigation that was more satisfying for our users. In other words, the data conflicts here…
• #86: Most companies facing this situation with conflicting sets of data fall back on opinions to make a decision—they’ll go with whatever perspective best matches the opinion they already had. Data gets ignored and we fall back on opinions…
• #87: There is another response I’ve seen at some companies, at that is to respond to conflicting data by collecting more data. These companies believe somewhere out there is the Holy Grail data point that will tip the decision clearly in favor of one perspective over the other. Usually at some point a paralysis sets in and no decisions are actually made but lots of reports are generated—they are using data, but not really benefiting from using data.
• #88: So, I’m guessing some of you have been in these situations before. And all that data starts to look like a waste…opinions seem like a better option. With these problems in mind, you can understand why only 8% of companies are using data…
• #89: What we need is a better way of determining what perspective of that data we need to use.
• #90: The way to do that is to work with all decision makers to determine what perspective of the question really matters most, whether that is users, conversions, brand, technology, or some other perspective of the question. You and all decision makers need to figure that out before anybody looks at the data.
• #91: Making this determination requires working with decision makers to understand where the question fits as part of a bigger whole. The visual framework I've found helps the most is the funnel.
• #92: This is a basic business funnel. People are aware of your company, they get interested enough to take action, and once people take action you have to support those customers so they become loyal advocates for your business. When you talk about whatever question you are trying to answer, you need to be clear on what level of the funnel or part of the business are you working at – what is the point of the page/screen/product/whatever? Figuring that out is where your opinions and expertise really come into play.
• #93: At the top of the funnel, the point is to get people’s attention and make them aware of your company. At this level, you want to base decisions on the data that tells you what people think about the brand and if they want to connect with you.
• #94: As you reach the next steps of the funnel where the priorities center on engaging users, getting them interested in your company, and building desire leading toward a purchase, data about user satisfaction matters more.
• #95: At the action stage of the funnel is where it becomes about the business case. Conversions matter here—no matter what that conversion is.
• #96: But once the customer is on board, then it becomes important to support that customer, working toward building customer loyalty. This is where creating a strong product matters, meaning the user case likely takes a higher priority.
• #97: Once you’ve gone through the funnel, you’ll know what set of data matters most. Then—and only then—should you and the decision makers look at the data set, whether that is the data from business case, the user case, or something else.
• #98: So, let’s go back to our example one last time where we are trying to determine whether or not we should include the navigation on this page. We want to know what level of the funnel the questions falls into. In this case, maybe the page is the first page people will ever see from our organization, and we want to keep the navigation to better satisfy our users. Alternatively, this page might be geared at people who are already familiar with our organization and are almost ready to take action in signing up or making a purchase or whatever that conversion is. At that level of the funnel, we want to follow the data that tells us to drop the navigation.
• #99: To wrap this up, there is a ton out there about data, the truth is you don't have to try for something huge right away. What I’d encourage all of you to do after you get back from this conference is to look at the projects you are working on and go through this process. You can download this worksheet that will walk you through these steps. Remember to keep it simple. Your goal here is to balance data with opinion to make a better informed decision.
• #100: Thanks again for being here with me today. I am excited to hear your thoughts and questions. Certainly if you have more than we can cover in the next few minutes, do feel free to email me or I’m happy to find a few minutes to chat with any of you this afternoon.