Metrics matter: Harnessing usage data for adoption, coverage, efficiency, and quality

Maya Hampton at REI

Hi, and welcome to the Design Systems podcast. This podcast is about the place where design and development overlap. We talk with experts to get their point of view about trends in design code and how it relates to the world around us. As always, this podcast is brought to you by Knapsack. Check us at Knapsack.cloud. If you want to get in touch with the show, ask some questions, or generally tell us what you think. Go ahead and tweet us @theDSPod. We'd love to hear from you.

Chris Strahl:                      Hey everyone. Welcome to the Design Systems podcast. I'm your host, Chris Strahl. Today I'm here with my Hampton, who's the senior product manager for Design Systems at REI. We've been having this REI theme on the podcast apparently, but I'm really excited that you're back because I loved our first conversation and I just want to say thank you for coming back around again.

Maya Hampton: Yeah, thank you for having me back.

Chris Strahl: So some things have changed in your world and I think that before we get to those changes, I wanted to set up this conversation in a fairly specific way. Now you more than most people I know in the land or design systems really care about metrics. So talk to me a little bit about why you care about metrics and why these things matter.

Maya Hampton: I think part of that stems from my role just in product management. So we tend to think a little bit more about the impact of the work the design system team does and how we can communicate that throughout the business. This helps drive adoption of this system. It helps us get the right feedback to know what's working, what's not, where there may be gaps in the systems if we see low adoption for example. And it also helps with that ongoing funding of the design system team to continue that good work. So there's a lot of great information that we could get from metrics, and I think the reason that it keeps coming back up is because it's kind of tricky to pin down what that really ideal metric is for a design system.

Chris Strahl: So what is your ideal metric for a design system? I think that there's so many that get thrown out there and there're at varying degrees of value and also varying degrees of difficulty to measure. So what do you look at?

Maya Hampton: Yeah, so acknowledging that kind of what's possible to measure and what kind of gets to the question we want. The one that we keep coming back to is adoption. And right now I'm really focusing on adoption as well as kind of system usability and user sentiment. And I think those get at a couple of really important parts of how the design system is used. By looking at adoptions, we can correlate not only where the system is being used, but if we can look at how the teams that have adopted the system are able to ship faster, deliver more consistent experience, deliver more accessible experiences, et cetera, we can start to again, correlate increasing adoption can increase the outcomes at that team level. Really fuzzy math in there. It's tricky to get from that kind of hard number of usage into some of those business KPIs around efficiency and speed to market, but I think there's a strong connection there.

The other metric is looking more at our design system consumers. So you had someone that on the show recently who was talking about enthusiastic adopters, which I love because if we have people that see so much value in the system that they want to use it and they're not being forced to use it because there's a mandate, it's a very different environment. And so if we can look at things like how usable is the system, where are those pain points? Do we have those feedback loops set up with our users? What I'd like to try to get into next is really tracking that sentiment of users as well. Are you satisfied with what the system has? Getting some of that more quantitative data gives us some numbers to work with and helps us try to see trends over time, see the impact of, hey, if we add this new capability, how does that impact the overall usability and sentiment of the system? And the hypothesis is theirs. As those numbers improve, adoption is going to improve as well. We really want those enthusiastic adopters.

Chris Strahl: Yeah, I think that's great. Thinking about adoption as a causality is a really interesting direction with it because all the studies get published and all the things come out that basically say adoption is the core, it's the king of what you want. But the reality is very often what builds that adoption metric is all these different qualitative things that measured over either a long enough timeframe or a large enough group of people become really great quantitative data as well. When you think about that idea of sentiment, one of the things I love is that most of our engineering tools and how we measure engineering productivity have shifted over to these sort of sentiment metrics to basically say, does my job feel like there's low friction or am I excited about the work I'm doing or is the things that I'm building do I feel like they have impact or meaning all of those being able to be represented at systems at scale, obviously the adoption needs to be there for the value to be captured across the organization, but the thing you really care about is that people are happy and stoked and doing really interesting creative work.

Maya Hampton: Exactly. And if we can solve some of those really common design decisions and we free up teams to solve those more challenging, more unique problems, and that's one of those core value prop of a design system is how can we flow work to these more strategic areas and get people out of that kind of production cycles.

Chris Strahl: And a lot of this has changed recently because we're just coming out of a once in a century pandemic where all of a sudden you had a bunch of organizations that were like, oh shit, I have no digital presence that's actually meaningful to users. And so much of product was about how much we can produce and it was all about how do I ship more things faster? And that sense has started to change and we've all noticed industry-wide. I think this shift away from hire every designer and engineer you can at all costs to an industry that's actually facing a little bit of contraction, especially in lots of technology groups. I think most people would agree it's harder to find that career pathway in designer engineering than it has been in a while. And as you look at metrics is a way of understanding the value of a design system and understanding the continued need to invest in it from both the design and engineering and even product side. What do you really look at in terms of that? How do you represent that in your organization that showcases why you exist and why you're important?

Maya Hampton: And that's a great example of how the work that the team does can shift as well. A couple of years ago in that big hiring surge, we were really focused on onboarding. Hey, we have a bunch of people that are joining the team. Everybody's working remotely. Suddenly there's a lot more need for better onboarding and better self-service documentation. And so we buckled down and really put out some more robust onboarding guides and did the work to redesign our documentation sites so that it would have more information, be more intuitive, cleaned up a lot of the content, built it on a new platform that's scalable, really trying to invest in that self-service documentation for a growing audience of users. Now to your point, earlier this year, we've seen things tighten up a bit and so we're in a different place where we're now trying to do more with less and we're not looking at goals.

We want a new designer to start and be able to build an experience using design system components and understand how to use that within their first day. It's so easy to do that. We can onboard people super efficiently. That's less important from now. Hopefully we do at some point pick that back up again and I have no doubt we'll be in a good position for that. And I think there'll be an opportunity to flex into some more interesting roles again. So now what we're looking at is really around efficiency. That's the theme of the year for so many companies and we're no exception there. We're all trying to do more with less. So we're looking at things like speed to market, right, velocity. I know you were talking a little bit about kind of output versus outcomes, which is what we love to track, but at the end of the day, the teams that we support that use the design system, they're often tracking their output and they're running tests and experiments and they need to be able to move quickly to do that.

So what pieces from the design system are they using to do that? Where do they have to deviate because the design system doesn't offer that. This goes back to that conversation about adoption and coverage and where can we find some of those gaps to understand if this is something that makes sense to bring back into the system and redistribute. If a team tests a new variant of an existing component and the test performs really well, we want to make sure we have a pipeline to bring that back into the system and make those changes globally across our platforms. So it's a lot more focused on that effectiveness at the team level and making sure that teams are still able to work at that velocity that they need to do in order to hit their goals. It all comes back to adoption and it's so multifaceted that we can break it down any number of ways to really get at how are we delivering value and how does it line up with the priorities of the business today.

Chris Strahl: Yeah, I think one of the interesting things that you just brought up is that you don't just look at your metrics from a design system centric point of view. You actually really reach into your consumers and understand how they're using the design system and the value that they derive from it. And I think about that in a similar way. And look, there's a big debate still raging about our design systems of product, et cetera, et cetera. The idea of you have your internal users of your product that are creating that centralized system and you have metrics and ideas of what they need, what contributions look like, what documentation looks like, what onboarding looks like, all these different things. But you also have that consumer of that design system that has a different set of metrics and has a different set of goals. And one of the things that I really like about REI in particular is how you federate a lot of, not just the data, but the decision-making between your consumers, your core design systems team, and lots of different disciplines along the way. Do you want to talk a little bit about that and how that relates to the way you think about data?

Maya Hampton: Yeah, and I will definitely caveat this by saying we're still learning and trying to improve on this. I think there's only room to go up in this area, but you mentioned contributions and how can we grow the system with contributions from our consumers? And we've had some ups and downs with contributions. It's more complicated than it may seem, but I think one of the big stepping stones to get to those quality contributions is this meaningful collaboration. And so how can we ensure that a design system team, we're not working just in a silo. We see we work across a lot of different product teams as well as engineering teams, platform teams, and start to develop a more holistic picture of how everyone works together and where those needs are. And so by really tapping into these different groups, we can start to federate some of that decision-making, which leads not only to evolving the design system with those best practices, but it also starts to get buy-in from those people who contribute those ideas.

And so if we look at a scale of contributions like contributing ideas and feedback and that collaboration with teams is hugely important in making sure that we are up to date with the work that those teams are doing. And especially as we look cross divisionally into groups like marketing and at REI, the brand team sits within marketing, we want to make sure the design system is aligned with our core brand. We want to make sure that the omnichannel experience for an REI customer feels the same whether you're in a physical store, whether you're using the website, make an order online, pick up in store, we want that all to be as seamless as possible. So we're kind of constantly looking for ways to bring those groups more into the fold, have that collaboration, but then ultimately having that single source of truth and decision maker because if not everything just devolves into consensus and things don't happen, it's crucially important to still I think have that central team but federating out some of those ideas and really ensuring we have feedback and collaboration opportunities with our consumers, with our users and stakeholders just helps keep everything moving.

Chris Strahl: And if you want to catch a really interesting episode about specifically how REI thinks about brand application, Jay Smith and I dove into it in really fun kind of interesting detail, especially y'all's application of color is kind of a remarkable system. And so that was a great episode. We'll go ahead and link it in the show notes. I think that when you talk about this idea of lots of contributors coming from all these different teams and all these different places, you're not just federating decision-making, you're actually federating the power for who has the ability to change the system. And I think that it's really interesting to explore the idea of power structures in design systems because traditionally it's kind of like a ball being passed or I like to sometimes equate it to a river where we can all kind of argue about exactly where we stand on the bank of a river.

But almost always design is upstream of engineering who is upstream of your users and where everybody else stands is sort of fungible. But the reality is is that one way river is something that represents who owns the power of decision-making at any given point and up high kind of away from the users. A lot of UX professionals and designers, maybe a little bit closer to those users is engineers, maybe being a little closer to them is QA people. But you end up in this situation where everybody holds power at a very different moment. And I think one of the really cool things about a design system in terms of these structures is you really truly federating power and you're also breaking down a lot of the traditional ideas of scope of influence. So a designer may feel comfortable making a change to code because there's a system that shows them how to do that. And likewise, an engineer might be comfortable making a change to design because there's a system that assists in that. And that contribution methodology is I think something that you all live very close to your heart and I'd love to have you talk a minute about how you view that contribution or how you view that power structure.

Maya Hampton: This is one of the really cool perks about working on a design system team is it's really a cross-functional team of engineers, designers and content designers as well. And it's been really interesting to see those roles within even the ecosystem of our team shift around. For instance, our content designer was recently working on helping the engineer with some token, some semantic token naming. And so that helps not only increase that kind of awareness of what are tokens and how do I care about them, but it also starts to create semantic names that are going to be more durable and intuitive and make more sense to those end consumers. We've also been able to see mostly internal products, but engineering teams spin up a whole new site using our components without even needing to have a designer and being able to put together a pretty good experience that's on brandand performance accessible and pretty good quality out of the box.

And the more that we can again kind of centralize and democratize these common decisions, then they're accessible to everybody. And the more we have kind of the shared language around the UI and tokens and what is a component, the more effective the collaboration is within these cross-functional teams. And to me, that gets back to these, this is how we can really unlock efficiency for teams is if we have designers and engineers speaking the same language and engineers empowered to make design decisions because they're well documented in the system or encapsulated within a spacing token, and we're not having to reinvent the wheel every time, but we're also speaking the same language and just having that shared language is such a huge benefit.

Chris Strahl: Do you feel like your corporate culture is really aligned with that model as well? I think that there is a part of this that is an unlock for you all specifically because it's how you all think.

Maya Hampton: I think aspirationally we know this is the direction we want to head. What is pretty cool about REI, I've been here for five years now. Jay's been there for 10 years. People have stayed through REI for, we have people that have been here for 25 plus years. It's very common. And we were talking earlier about the cyclical nature of things. Guess what? We've seen some of these same problems come back over and over again as new people join. And if we want to break the cycle of that, then we need to break down some of those barriers and think and work in a different way. Design systems can kind be those change agents, whether it feels like it directly or not. Again, by creating that kind of shared language for everyone to work around, then we can kind of break those borders down a little bit and find more effective ways of working, which is going to make people happier to be working, maybe doing better work because they're excited to show each day and not have to just do a bunch of production work or fix accessibility issues, that type of thing. We're trying to streamline as much as we can there, and there's a lot of benefit to doing that

Chris Strahl: Because REI is a co-op. And because of the nature of the company that you work within, that sort of notion of cooperative decision-making of shared ownership of really a shared responsibility for the success of the company, do you feel like that mirrors a lot of the goals and values that you place inside of the design system?

Maya Hampton: Yeah, I do. And that's where looking at some of these more democratized models around breaking down some of those power structures, but also making sure that we're enabling employees to make the best decisions. So as a cooperative, one of our key values is how can we go further together? And REI is owned by our members and similarly within our employee organizations, there's a lot of pride and accountability that people take in their work. And really it grounds back to that mission about getting people outside that pretty much everyone can relate to at the company, but being able to work together cooperatively helps to accomplish our business goals, but also in a way that's good for our customers, that's good for our employees, that's good for our members, it's good for the business in general, really taking a more holistic look at ways of working beyond just trying to hit shareholder value for a certain quarter. We are accountable to ourselves, and so that gives us some more freedom to carve out the time to build more durable solutions and not get stuck in that kind of short-term gains to kind of show that value to a shareholder group.

Chris Strahl: So I think that there's another part of that that is an interesting additional effect, and that is the idea that you're creating a durable system that can weather a storm. I think that there's a lot of people in the industry right now that are feeling some rocking waves and some thunder and lightning. Maybe they're being reorged, maybe there's a major shift in their corporate culture that's happening. Maybe there's something that's happening where their team is being consolidated. That sense of durability, regardless of what's going on at REI persists as a part of that democratized, federated way of making decisions.

Maya Hampton: Exactly right. We want the solutions that we're putting out today to continue to be durable tomorrow, maybe the value of the token changes, but the tokens will still be there. What can we do today that's going to have that lasting impact,

Chris Strahl: That surviving of any given direction or whim or even manager? I think for some people you don't want to necessarily have a system that can change at the whims of the storm. You want to have that system that feels grounded, that feels stable. And I think that you found that durability largely because the federation of who makes decisions, there's not any one person that can really make a major change without the agreement of a whole lot of other people as a part of that structure.

Maya Hampton: Exactly. We're all working together. We want to shield ourselves from what happens at a lot of companies where a new creative director will come in and they want to put their stamp on things and propose a whole big redesign. Well, guess what? We're not going to do a big redesign. We can do incremental changes. We're going to release it through the system, but we have a lot of partners that we work with and we want to democratize that decision making as well. So it's not just to your point, the whim or the current trend, but it's really a more proven and tested pattern that we agree through what we've learned and what we've learned from other companies, that this is the decision to go forward. And it also empowers us to say, no, we're not going to change things because systems are like if we want it to be durable, we don't want it to be changing that frequently.

There's sometimes certain areas of the business such as campaigns, marketing do need to work at a quicker pace, and so we can work on solutions that enable that sort of flexibility, themeing and pallets. For example, for a sale, we want it to look different than the core. We want it to stand out and really highlight some of these campaign objectives. That's very different from functional ui. So we're not going to build the same solution for both of those, but rooted in those same principles. We can build solutions that are going to be agnostic of where they ultimately live, whether it's in Figma, whether it's in code, whether it's in whatever design tool comes after Figma or whatever framework we use after view. Those decisions will still be durable and grounded in logical reasoning.

Chris Strahl: Thinking about how we bookend this conversation with the idea of metrics, we've kind of gone from this place of trying to understand adoption, trying to understand contribution, to try to understand how to make something that's lasting and durable. What are the things that you look at to really assess that idea of lasting and durable? And I mean adoption is still likely a part of this conversation as well, but when you start to think about what makes that impact not just for this quarter, not just for this creative director, not just for this team, but when everything is different 10 years from now, what remains

Maya Hampton: So we're trying to look at if adoption or coverage is kind of that north star metric. There's input metrics that we can track. Things like how many components are in the system, how many visits do we get to the documentation, how many bug fixes do we release in a given period, and we can look at some of those kind of consumer satisfaction, usability sentiment that drive towards that goal of coverage and adoption and ultimately get us to those lagging KPIs for the business around efficiency and quality. The work of design systems, the value is realized not when the design system builds a component. The value is realized, the more that component is used. And so I think usage data honestly is a big part of it. If we built a button three years ago and we haven't changed it, but it's being used widely, then we know that's a durable solution. If we see that teams are kind of consistently having to deviate because they want to put an icon in there or they want a different color or whatever the case may be, that's an indication I would think that that solution is not quite as durable or the variants that are necessary for that are going to introduce some inconsistencies, but it does lead us towards adoption.

Chris Strahl: Well, it's an interesting idea, this idea that you're building a lot of things to last. You're building some of these core conceptual ideas like a thing like a button, right? The easiest to pick on, most dead simple idea of a component you could have, but then you also have these things that are less durable, that may be more fleeting, and those are things like a stepper flow that has a lot of really detailed form logic inside of it. That may be a component now, but ultimately user preferences or behavior might change that renders that obsolete at some point in the future. The durable part of something like that is the way that you build up from your more durable things to create that experience, even if that's not lasting, the concept of how do I get there is?

Maya Hampton: Yeah, exactly. And if we look at the atomic design methodology, the atoms and subatomic parts that come from a system, those are going to be a lot more durable than the molecules and organisms that are created with them. We create the ability to compose those kind of organism level components, which might show up as a stepper or a film strip carousel. Maybe those come back in style, maybe they go out of style, but those subatomic parts still exist, right? The color, the typography, the buttons, the back and forth arrows, those parts are still there. And so when we do want to try a different kind of composition of those smaller parts, we can still reuse them more broadly. And then if the brand color changes, we can just change out that hex value and we don't actually even have to change the tokens. We can continue to use those smaller parts, kind of extend them as broadly as possible and then build new things on top of them that can be a little quicker changing than kind of those core parts that really shouldn't change that often.

Chris Strahl: So one thing I want to wrap up with is you've created this durable lasting system. You've federated a lot of your power structures. You've democratized the way that people use and contribute to the system. That makes for a really resilient program that is again, beyond the people or managers or even content of the system. When you think about somebody that's in that position right now that is in the middle of the storm and their design system ship is rocking up and down and things maybe don't feel so steady, what advice do you have to that person?

Maya Hampton: We've all been there and with design systems, part of the work is so exciting, so many different ways we could go. And so I think my advice in this case would really be to not get distracted by all the shiny new things and really focus. If there's one thing you can track that thing, if you can talk to some of your stakeholders and understand what do they want to see? How are they tracking things? Somebody on the engineering side is probably going to be looking at different numbers than somebody on the business side, which is going to be different than somebody on the marketing side. And so trying to understand who do I need to communicate these sort of value propositions with? What is the data that's going to help me make that story? And kind of realistically looking at what data do I have availability to.

It can be really hard to get some of these data points around how efficient teams are and benchmarking before and after using the design system, especially if you've had a design system around for a while. And so I would really focus on core metrics around, again, adoption and coverage, and then try to layer on additional KPIs or OKRs that you can line up to specific initiatives. Say if you are introducing a new capability for theming, what's some before and after data that you can try to gather to help validate that that was a valuable use of time? Often that's going to come back to usage and coverage, and so I wouldn't discount looking at adoption, and I would advise trying to slice that data in different ways that are going to tell the story that you need to tell at that time and be open to change because everything's going to change.

Chris Strahl: Awesome. Well, thank you so much, Maya. I really appreciate you being here. This has been the Design System podcast. I'm host Chris Strahl. Have a great day, everybody. That's all for today. This has been another episode of the Design Systems podcast. Thanks for listening. If you have any questions or a topic you'd like to know more about, find us on Twitter @theDSPod. We'd love to hear from you with show ideas, recommendations, questions, or comments. As always, this pod is brought to you by Knapsack. You can check us out at Knapsack.cloud. Have a great day.

Get started

See how Knapsack helps you reach your design system goals.

Get started

See how Knapsack makes design system management easy.