What we would have asked differently to help teams improve?
Written with Josh Armitage
Asking questions to uncover high leverage opportunities can show you what could be an ambitious destination for your team
High-impact questions that can uncover high-leverage improvement opportunities for your business
TLDR
Identifying the highest-impact opportunities amidst competing priorities is a significant challenge. While every team has a different starting point and goal, asking insightful questions can uncover valuable insights for transformation.
We (Josh and I) have curated a shortlist of 5 questions based on our experience and chapter 29 from the book "Transformed" by SVPG. These questions serve as a basis for rapidly and inexpensively discovering opportunities and risks related to transformation, rather than product discovery. For each question, we dive into what it can uncover and what is the impact of those observations.
These questions aim to facilitate a joint learning exercise from a caring perspective, without judgment or evaluation. We also cover an example of an illustrative model to help facilitate conversation with your team.
The five questions are:
“What happened when the team released last time? How often do most teams release? Did they release independently, or did they coordinate releases as a single integrated package?”
“What happened last time when the customer encountered a critical problem? What was the mechanism for detecting and correcting it?”
“When was the last time an idea was killed or significantly changed? What was the process by which that happened? In that instance, did leaders or stakeholders need to approve a decision like this?”
“What is the nature of typical items on the roadmap - features, projects or problems to solve? How do leaders prioritise the work?”
“What happened last time when an innovative idea failed? Were people scared to do something that might fail? Did the company understand the techniques to fail fast and cheaply?”
Context matters, so what is our context? After reading Marty Cagan’s latest book, Transformed, we wondered, “What are the five highest leverage questions we would have asked the teams we have worked with?” This is not to judge them but to jointly learn about high-value improvement opportunities. Every team we have worked with, on transformation, has a very different starting point and goal. So it's unfair to compare their starting points but it is fair to compare the questions to start the conversation. The direction after asking the question would be very different each time.
Why five questions? As a transformation or product leader, these work as the basis for rapidly and cheaply discovering opportunities and risks—not product discovery but transformation discovery.
Together with my ex-colleague and fellow transformation director Josh, we reflected on our experience working with enterprises (at least 1,000 people and $200 million in revenue) undergoing digital transformations in the UK and Australia. We both subscribe to the idea of a transformation lighthouse, an initial transformation target that is low risk and size but high visibility and value. These questions are a heuristic for finding said lighthouses, enabling your team to build momentum through continuous small wins, AKA the only way transformation ever succeeds. Big bang transformations in complex organisations require so much behavioural change that they lose momentum or do not achieve the ambitious change.
A note on psychological safety while asking questions. If the leader is asking these questions as a reflection on themselves and their teams, it shows curiosity. This is good for psychological safety and the team might feel safe to be 100% honest. We would ask these questions as an outsider to the team. Hence this is tricky. The intention of these questions is a joint learning exercise and NOT evaluation. We also want to come from the angle of care and not necessarily as an expert. We want to remove any judgment from these conversations.
The common variables we have encountered in this context are:
The organisation is a complex organisation with at least 1,000 people and $200 million in revenue
There is at least one leader in product, technology and transformation that wants to drive the improvements
The current operating model is resisting the impetus for change, leaders are sceptical of the need and the capability for transformation.
Pressure from the shareholders, the board and the market is pressing leadership to make some improvements
Hence, having a shortlist of questions that cut to the crux of the problem is imperative.
For each question, we dive into what it can uncover and what is the impact of those observations. This is based on our experience in the above context.
Caveat: Any 1 question below can help to initiate improvements. Your judgement to start something will depend on business value based on the leadership, context and business opportunities. See below a model linking the questions and potential opportunities, you can do this as a collaborative exercise with your team.
Q1. “What happened when the team released last time? How often do most teams release? Did they release independently, or did they coordinate releases as a single integrated package?”
What can it uncover?
In the teams we help we want to see teams chase objectives autonomously but with the necessary strategic context, so that they feel empowered. Teams being deeply coupled on the path to production works against this aim.
Teams that deploy autonomously are the starting point of any improvement initiative. A single team that can deploy autonomously today, can become the catalyst that drives towards creating the path for teams.
Years of DORA research (a program run by Google Cloud) have discovered elite clustering around multiple daily deployments, for business differentiating digital capability. We have always uncovered various dependencies teams are struggling to deploy autonomously. This observation is consistent for several types of products - Data Platforms, Consumer Products, B2B Products, Commercial off-the-shelf (COTS) software, etc.
This question can uncover capability gaps in testing, continuous delivery & integration, deployment automation, flexible infrastructure, etc. In that case, teams will need help to skill up in these areas. So keep your eyes and ears peeled for several pathways from here on.
What is the impact?
Coordinated deployments are risky and expensive, inhibiting the team’s ability to experiment and pivot. Without sufficiently mature release engineering that focuses on the rapid delivery of safe change, trying to compete digitally is like trying to drive a family sedan at racing car speeds, no one can do it.
Q2. “What happened last time when the customer encountered a critical problem? What was the mechanism for detecting and correcting it?”
What can it uncover?
The “development vs. operations divide” remains “the de facto” standard in many enterprises. It enforces quality by setting up competing teams independently responsible for throughput and stability.
It can reveal previous pushes for operational efficiency that necessitated focusing on understanding systems through their commonalities rather than their specifics. This led to alerting on diagnostic metrics such as CPU load and similar that are, at best, a poor proxy for customer impact.
Understanding “where quality is” is the accountability of the system creators. This leads to teams who are further along the journey of being customer-centric. They can become internal case studies about the power of making quality and stability everyone’s responsibility.
This question can uncover capability gaps in monitoring & observability, testing and other DevOps skills. In that case, teams will need help to skill up in these areas. So keep your eyes and ears peeled for several pathways from here on.
What is the impact?
If engineers are sheltered from the outcomes they’re being asked to chase and optimise, then a key feedback loop is missing. The dev-only teams that dominate enterprises have multiple layers to unlearn to welcome feedback from several sources. Teams operating in a DevOps mode of shared accountability for quality have a shorter journey.
This can mean that it is expensive for teams to respond to incidents. It also results in a loss of trust with customers if incidents take a long time to resolve or there are repeated quality issues. While bugs are inevitable, bugs reappearing is not.
Q3. “When was the last time an idea was killed or significantly changed? What was the process by which that happened? In that instance, did leaders or stakeholders need to approve a decision like this?”
What can it uncover?
This question can reveal a culture behind developing products. An idea that has originated from a senior leader alone, especially as a habit, can blind teams to the risks behind the idea. In this case, there is a risk of designing HiPPO (highest-paid person’s opinion) products. This disempowerment of the teams can detach teams from customers, sinking time into politics and wasteful coordination.
We have witnessed this in many organisations and uncovering assumptions behind these ideas might be the best place to start. For example - what are we hoping to achieve if these ideas come true? This question can also reveal a capability gap that enables surfacing and testing assumptions underlying ideas.
Sometimes these discussions reveal the funding mechanisms in the organisation. Usually, the purse owner is given the highest importance to their ideas. These ideas may be good but running some experiments, “to prove their worth in the long run”, stands the business in good stead.
A team that requires executive approval to pivot, change or discard an opportunity - is probably wasting a lot of time on decisions they could make themselves if they have the right context. If a team is discarding ideas, this question can reveal the principles and practices teams use to discard ideas.
These discussions can also reveal a need for more alignment between various teams with a stake in ideas. They might have different goals and motivations behind the delivery of the product. It’s worth uncovering that.
Hence this question can lead to many different improvement paths. This is, therefore, a harder question to answer and explore. It requires emotional care to explore the right pathway.
Acclaimed author Neil Gaiman states, “When people tell you something’s wrong or doesn’t work for them, they are almost always right. When they tell you exactly what is wrong and how to fix it, they are almost always wrong”. While teams should liberally accept feedback on where there are issues, allowing people from outside the team to design the solution is a recipe for disaster. This is because they may not have all the right context to come up with the right solution.
What is the impact?
If bad ideas are entertained for a long time, teams spend a lot of resources on building them. These ideas may never see any customer traction and business results. This is a very high opportunity cost for business.
The move to actual customer-centricity, i.e., having the product teams regularly talk to and interact with their customer base, goes against historic top-down approaches to pseudo-product development. Leaders and executives need to become comfortable delegating decisions and empowering teams. This transition can be challenging for people who want to feel a strong sense of control, yet without the required executive support, modelling and change, the transformation is doomed.
Building a solution is the most expensive way to validate an assumption or hypothesis. It should be the fallback option, not the defacto. Yet the continuous discovery muscle, embedded in startups and scale-ups, has never gained any traction in many enterprises.
Q4. “What is the nature of typical items on the roadmap - features, projects or problems to solve? How do leaders prioritise the work?”
What can it uncover?
Creating roadmaps has been the standard practice in most organisations. This helps leaders to know the portfolio of work they are funding. Yet, they are often fundamentally disempowering. The more fixed a roadmap is, the closer we move back to waterfall-style approaches (the one where all risks are known). They ignore that our ideas and bets are filled with risks. It is a roadmap with no potholes in the distance. The more one travels on that road the more potholes are revealed that we never knew existed.
This question can reveal the role of engineers in the teams. Engineers are the closest to enabling technology, which makes them a key input to innovative solutions. Telling them what to build instead of co-designing the solution with them is throwing their most powerful value driver in the bin.
This discussion also reveals prioritisation practices in the team. There is no silver bullet or process that fits every situation. It reveals how a team with different views on 4 kinds of product risks (value, viability, feasibility, usability) comes together to decide. Every team might have a different dynamic.
Roadmaps are unfortunately used as planning tools and not as communication tools. This regularly presents problems in alignment across various teams. Roadmaps do not clarify the most important bets the organisation cares about. It doesn’t clarify the high-risk items that need more data from customers, and stakeholders to validate if they are worth working on.
There is confusion in team ownership across the roadmap. Roadmaps could have various parts to them - platform bets, customer experience bets, bets by market segment, etc or just a big jumble of all the work. This data represents how teams are organised in the organisation.
And so this can unravel the main problem that the teams are solving for. It can show the biggest hypothesis that is worth betting on and its connection with the business result.
This question can also reveal the current state of funding mechanisms and product strategy.
Again this is a deep question that can lead to many transformation pathways.
What is the impact?
When an enterprise overcorrects, trying to achieve certainty in a complex space, it squeezes all the capacity for innovation out of the company. The roadmap style speaks to the enterprise’s ability to tolerate risk and embrace the unknown and unknowable.
When features, not opportunities, are handed to teams, this can hamstring your engineers from delivering the full breadth of their value. When prioritisation decisions are not discussed or the rationale is not clear, the team will be incapable of operating optimally.
A fixed roadmap exposes organisations to risks that are not aligned with their risk vs. reward appetite.
Q5. “What happened last time when an innovative idea failed? Were people scared to do something that might fail? Did the company understand the techniques to fail fast and cheaply?”
What can it uncover?
This question is strongly related to number 3.
Innovation stems from learning, and learning stems from trial and error. Innovation is an inherently risky proposition, and enterprises are socio-technical machines that look to mitigate risk wherever possible.
A business case and artefact-heavy process is based on the assumption that we can predict the future with enough upfront investment. Instead, by lowering the cost of experimentation, you can empirically determine what has promise and what appears to be a dead end. Suppose people are chastised because their experiment didn’t show the predicted results. Then in the future, they will not be able to expose the risks before spending all the money on building something.
This question can also reveal leaders' decision criteria to evaluate ideas. It can reveal the rationale behind the type and number of ideas from different parts of the organisation. It is a good sign if they come far & beyond the product team.
It can also reveal that the company has a portfolio of products with very different risk profiles. However, the question is then “Is the culture in teams working on them also dialled up or down depending on the innovation appetite of the product?”
What is the impact?
Cultural change is hard and expensive. Even more expensive is not being able to experiment or accept that “we have failed as a team”. This is emotionally expensive for people.
The hardest transition is accepting, openly discussing and evaluating the risks of developing software/solutions. It’s the biggest skills gap in enterprises looking to adopt a product model and getting a read on how big the gap is gives you an indicator of the scope of the transformation.
Other questions from the book that can help you or could be part of your top 5 questions:
How Products Are Built and Deployed
Who is responsible for ensuring the new capability works correctly? Is this a largely automated process, or is it largely manual?
What is the level of autonomy in the teams incl. dependencies and interactions to get simple things done?
When new features are deployed are they routinely instrumented so that the data is collected?
What is the level of technical debt?
What is the perception in the organisation regarding speed, quality and trustworthiness of the engineering organisation?
How Problems Are Solved
How is the work provided to product teams? Is it in the form of roadmaps of features and projects? From where do those features and projects originate?
When the team gets to work on something they have been assigned, what is their process? What are the key roles and responsibilities? Who determines the detailed requirements, and how is that done? do they generally have evidence for their decisions or just opinions?
When and how do engineers enter the picture? are product designers involved?
What is the role of the stakeholders in coming up with the details of the solutions? How is the team ensuring that the solutions meet the needs of stakeholders?
What level of customer interaction is involved? Are potential solutions tested with customers before the decision is made to build?
What is the definition of success - shipping a feature or shipping a feature on time or measurable outcomes associated with the feature? What happens if the feature ships but the outcome is not achieved?
What is the perception in the organisation of the people on these teams responsible for defining and designing these features? Are they viewed as deeply knowledgeable of the customers and the business? Are they trusted?
How You Decide Which Problems to Solve
Who decides what work will get done? Does the company have some sort of annual or quarterly planning process? Is the purpose to establish priorities and funding? How are funding decisions made? who proposes the projects? Does the company fund projects or does it fund product teams - or individuals?
Is there a product vision? Is there a product strategy? If so, are these created at the level of organisation or at the level of individual product teams, or at some level in between?
If the product teams work off a product roadmap, who decides the items on that roadmap? Is the work coming from the sales organisation? Is it coming from stakeholders? Is it coming from the CEO?
Are there desired business outcomes associated with the items of work? Who defines those desired outcomes, and how do they come up with the expectations?
Does the organisation track the percentage of items that deliver the hoped-for results?
Product Model Competencies
How do product managers spend their time? Are they in meetings all day? If so, what type of meetings? How much time are they spending on product discovery?
How well do the product managers know the customers? How deeply do they immerse themselves in the data? How well do they understand the product’s go-to market? How well do they understand the other dimensions of the business? How well do they understand the industry, competitive landscape, and relevant technology trends?
How well respected are the product managers? How well do they collaborate with the designers and engineers? How well do they collaborate with stakeholders? How are these people perceived by the executive team?
Are product managers receiving coaching every week?
Does the organisation have enough product designers or are they jumping between several product teams? Are product designers embedded in the product teams as first-class team members?
When are product designers brought into the process of discovering and designing a solution?
How often are designers creating prototypes? What types of prototypes? What tools do they use? How are they testing those prototypes?
Is there an experienced design manager in place who knows what to look for in a product designer, and who provides coaching to the product designers on a weekly basis?
Does engineering have a role that is explicitly responsible to help with the question of what to build and not just how to build? How often do engineers visit directly with users and customers? What is the level of interaction between tech lead and the product manager?
Is the first time the engineers hear of a product idea at sprint planning? What is the role of engineers in evaluating whether something should be built or not?
Are ideas introduced by engineers, especially technical innovations, well received and carefully considered by the product managers?
Are engineers responsible for quality of their code? Do engineers fix their own bug?
Are any of the engineers outsourced? If so, what percentage? And which specific engineering roles?
How are engineers perceived? are they simply there to build what people request?
Are product leaders responsible for determining the direction and measures of success? Do their responsibilities include the strategic context (product vision, product strategy, team topology and team objectives)?
Do product leaders view coaching as one of their most important responsibilities?
How do individual contributors perceive their managers? Are they viewed as committed to coaching? Are they deeply knowledgeable of the strategic context?
Product Model Concepts
Does the company have durable teams or temporary project teams?
Does the product team feel a sense of agency? Are they vested in what they are building? do they care about the outcomes?
What level of access does the team have to customers, data and the stakeholders?
Which product risks the product team is considering as they do their discovery work? Does the team know how to run quick experiments, both quantitative and qualitative?
What is the level of formal process the company is following? What is the priority: following the process or following the principles?
What is the level of trust? Is it mainly top-down, command and control? Or do the leaders try to push most decisions down to the product teams?
Is the organisation optimising for predictability or is it trying to optimise for innovation? Does the company understand the role of engineers in innovation?
Does the company understand the necessary role of failure in innovation? Are people scared to do something that might fail? Does the company understand the techniques to fail fast and cheaply so that they don’t fail in production?
