R080 - Evidence-Based Product Development, with Guido Lonetti

There are few things more human than our tendency to assume. We assume, fill in the gaps, tell ourselves stories that seem to make sense, but in product development, this can be a silent curse. Blind faith in intuition, trendy frameworks, or methodological dogmas has led more than one product to crash before even taking off. What if success didn't depend on how well we think we know what the user wants, but on how rigorously we discover it? Today we will talk about evidence-based product development, the art and science of seeking the truth first, even if it's sometimes uncomfortable. Because in the end, in the real world, what counts is not what we believe, but what we can prove. To discuss this, we are joined by Guido Lonetti, Head of Product at DLocal.

What is the real world to you?

To me, the real world is nonsense, a cosmic accident. It shouldn't have happened, but due to energy and matter issues, we appeared. It's going to last very little. The history of humanity will be just a centimeter in the vastness of the universe. We shouldn't take ourselves so seriously. It's impressive how, if you zoom out and see the earth, life, the Milky Way, and the vastness of the universe, you realize how small we are. So why take the day-to-day problems so seriously? It's like that phrase: “Dance like nobody's watching. Have fun, make mistakes, try more things.”

What does evidence-based product development mean?

Evidence-based product development seeks to reduce risk when building products. It is based on the scientific method: we start with a hypothesis, use empirical methods, and observe data and facts. The goal is to avoid failure when building products, which is achieved by collecting evidence that allows us to validate our assumptions before proceeding. It is a continuous learning process to avoid self-deception and make informed decisions.

In the world of product management, there are very easy predetermined answers.

Upcoming Certified Course in Lean Product Discovery

Agile teams focused on impacting Business. A course created by Jeff Gothelf and Josh Seiden, authors of Lean UX, taught in Spanish by Carlos Iglesias and Laura Polls.
From June 4 to 25. Online.
More information and registration here.

Just Enough Research

How do you identify when the evidence is sufficient to move forward with a decision?

The key is understanding the risk. If the risk of failure is high, it's worth doing deeper discovery. If the risk is low, you can act with less evidence. What helps me is using a framework called "truth curve," which compares the amount of evidence with the effort needed to obtain it. This way, you can decide whether to continue with the research or proceed with what you already have.

How much do you trust the evidence you have to make decisions?

There are people with more risk aversion and others with greater tolerance. Some have a very strong intuition and don't need as much evidence because they trust their sense. That's what happens sometimes. While evidence is key, some trust intuition more, and that has its place. But the key is not to make decisions blindly. Intuition should be used as a hypothesis and then validated with evidence.

What role does intuition play in this process?

Intuition plays an important role when you have a lot of experience in an area, such as product management. Over time, we develop an intuitive muscle. However, it's crucial to use that intuition as a strong hypothesis that must then be validated. We should not let it guide our decisions without evidence, because the risk is high. It's a balance between art (intuition) and science (evidence).

What has been a situation where you thought a hypothesis was correct and the evidence proved otherwise?

At N26, we were working on the investment product. Initially, we thought users wouldn't adopt managed funds, which were more conservative. But after conducting user interviews and obtaining evidence, we discovered there was a great demand. In fact, the product turned out to be one of the most successful in terms of adoption. This experience taught me that sometimes our hypotheses can be wrong and that evidence can really surprise us.

The problem is how we use intuition, whether we use it blindly to make decisions without seeking evidence or if we use it as a strong hypothesis that we need to validate.

Common Mistakes in Product Discovery

What other common mistakes do you see in discovery processes?

In the discovery process, one of the most common mistakes is confirmation bias. This bias occurs when people tend to search for and focus only on evidence that supports what they already believe, rather than being open to the possibility that their assumptions might be wrong. It's very easy to fall into the trap of confirming our ideas, especially when we are emotionally invested in them. When this happens, we are not truly listening to what users are telling us; we are only looking for signals that validate our previous beliefs.

Another very common mistake is talking to the wrong people. Sometimes, especially in startup environments or small teams, designers or product managers end up validating ideas with people close to them or with experts on the topic, but not with the real users the product is aimed at. Experts can be very useful in some phases of the process, but they don't necessarily represent the needs and behaviors of your target audience. For example, you can talk to designers or people in the same field who have a very specialized view, but if the product is intended for a general audience or users without technical experience, their responses may not reflect the market reality. Validating ideas with the wrong audience can lead to a misinterpretation of user needs.

Evidence-based product development is based on the scientific method: hypothesis, empirical method, and data observation.

A third important mistake is doing too much research. This may sound contradictory, but it's true that many teams fall into the trap of doing more research than necessary, unnecessarily prolonging the discovery phase. This can be particularly harmful because, by investing too much time in research, you can delay the project's progress. Sometimes, excess data doesn't help make faster or more effective decisions, but rather, it can create analysis paralysis. The ideal approach is to do enough to obtain evidence that validates or refutes the hypotheses, but not so much that it delays you or makes you constantly doubt what you have already learned.

The key here is to find a balance between the amount of research and development speed. In many cases, the best evidence comes from small, quick tests, like user interviews or minimal proof of concepts, rather than exhaustive research that delays the development process. A product should be built and tested with real users to obtain more relevant and concrete data, rather than relying solely on theories or assumptions.

In summary, errors in the discovery process can be caused by not listening to the right people, falling into confirmation bias, or conducting research that, although exhaustive, doesn't help make quick and precise decisions. It's important to be aware of these biases and errors and ensure that the discovery process is as efficient as possible, always maintaining a user-centered focus and a proper balance between research and action.

Uncertainty is the quantification of risk, and we must use more economical techniques to reduce that risk before launching into product creation.

Evidence and Regulation

What particularities do you see in the use of evidence in highly regulated sectors like fintech?

The use of evidence in sectors like fintech presents several unique challenges due to the strict regulation and operational constraints that characterize the financial industry. Unlike other tech sectors, where tests and experimentation can be conducted with more freedom, in fintech there are significant limitations that affect how and what type of evidence can be generated.

One of the main challenges is the inability to conduct certain types of experiments due to regulations. For example, in some cases, we cannot do A/B testing with prices or product conditions, as that could create differences in user experience and, in highly regulated sectors like banking, this can be a legal risk. An example of this in my experience was when we were trying to find the optimal price for a product we wanted to sell, but due to regulation, we couldn't simply set two different prices and see which was more effective. This type of experiment is prohibited by regulations, as it could lead to unfair commercial practices.

Therefore, in fintech, evidence cannot always come from direct experiments or traditional market tests. Instead, we must step back and look for alternative ways to gather information, such as user interviews or direct observation of how users interact with our products in specific contexts. We learn a lot from these conversations and how people use our tools, and we can adjust the product experience based on that data.

The goal of evidence-based product development is to reduce the risk of failure by validating our assumptions before proceeding.

In my experience at DLocal, for example, we learned a lot by talking directly with users about their needs, especially when we started developing products like cryptocurrencies or investment products. Initially, the hypotheses were uncertain, but by validating our ideas directly with users, we were able to obtain concrete evidence that there was a real demand for an easy-to-use investment product, which led us to shape our value proposition. This also taught us the importance of educating users, as many didn't know how to start investing, which became a key point of evidence for designing our strategy.

Another important aspect is that, although restrictions exist, evidence remains key to making informed decisions. Even in a highly regulated sector, if we don't have clear data on what works or not, it's very difficult to build successful products. Therefore, although direct tests are not always possible, we always look for indirect ways to obtain the necessary information: from qualitative interviews to user behavior studies. The combination of qualitative and quantitative techniques allows us to build products that, despite the restrictions, truly meet market needs and comply with regulatory standards.

In summary, the particularity of using evidence in fintech is that, due to regulatory restrictions, we cannot always apply the same experimentation techniques as in other sectors. However, this doesn't mean that evidence is less important. On the contrary, having reliable evidence based on real user behavior remains crucial for the success of any fintech product, and we must be creative and meticulous to obtain it in alternative ways that align with the sector's legal requirements.

How has your way of working changed over the years?

My way of working has evolved significantly over the years, especially in how I approach product development and the use of evidence to make decisions. When I started in product management over 15 years ago, the dynamics were very different. Back then, "gut feeling" or intuition played a central role in decision-making. We had to rely more on what we believed the user knew and what we thought would work, without having as many empirical validation tools as today.

A good designer thinks about the user and has their work centered on the user. And a great designer does it considering the business as well.

In my early years, evidence was not the star of the product development process. Many projects were based on assumptions and intuitions that came from both the founders and the early product managers. We often made decisions based on personal experience and what we thought users wanted, which wasn't always accurate.

However, over time and especially after working with Marty Cagan, Ben Horowitz, and other influential figures in the field of product management, I realized the importance of integrating objective evidence at every stage of the development process. We began to incorporate more data, analytics, and research to validate our ideas before launching them to the market. It was a significant shift from working with intuitions and opinions to being more systematic in seeking data to support our decisions. This change was very marked with the advent of the "product discovery" idea and the more scientific approach adopted in the sector.

In this stage of greater maturity, I also learned to recognize that we don't always have to wait for perfect evidence before making decisions. The combination of science and art is what defines product management. The art is in developing an intuition that allows you to identify good product opportunities, but the science is what helps you confirm those intuitions and minimize risks. Over time, evidence has become a fundamental part of my decision-making process. Now, whenever we start building a new product or feature, we try to validate our hypotheses before making a significant investment of time and resources.

As the market has become more digitalized and user feedback has become more accessible and faster, my methodologies have also changed. A few years ago, we spent much more time in deep research to find answers, whereas now, thanks to the tools and technology available, we can conduct faster tests and make more agile decisions. This speed allows us to iterate constantly, without fear of failure, but with the mindset that each iteration brings us closer to the right solution.

In summary, my way of working has shifted from introspection guided by intuition to a practice based on evidence, but without leaving aside creativity and the ability to take certain risks. Now I see "discovery" as a constant process, which must be balanced with a large amount of data, but also with the courage to follow certain hypotheses that can open new doors, even if we don't have all the evidence yet. The change has been gradual but fundamental to evolving and adapting to the new challenges of product management.

Building products without evidence is like navigating in the fog relying only on intuition. We might reach the port, or we might crash against the rocks without even seeing it coming. Evidence-based product development doesn't guarantee success, but it minimizes self-deception, that silent enemy that often leads us astray. In times of noise, urgency, and easy solutions, betting on the truth can be our greatest act of rebellion. If this conversation with Guido resonated with you, I invite you to leave your reflection in the comments, share the episode with someone you think needs it, and subscribe so you don't miss future talks that invite you to think. A big hug and see you in the real world.

Training for your team

At Runroom, we help organizations like yours bridge the gap between tactics and strategy. Through certified training in Lean Product Discovery, we empower teams to think and act strategically from day one.

Learn more: Request a personalized training for your company.

Next online course in June: Certified in Lean Product Discovery

More information and registration here.