AGL: Successfully pretotyping at scale
AGL’s Experimentation Team is leading the way when it comes to pretotyping. The team is led by AGL’s Experimentation Manager Richard Guy, who was trained in pretotyping by Leslie Barry, founder of Exponentially. This has led to pretotyping and rapid experimentation not only being used successfully at scale, but coming to sit at the core of AGL’s product innovation.
Richard sought to implement rapid experimentation and early-stage idea validation within product development so that AGL could deliver more great products to its customers. Pretotyping and rapid experimentation seeks to tame failure by testing ideas before they are built, so that companies can make decisions based on data instead of opinion, and ultimately pursue products and services that indicate trends, de-risk the process, and drive better design. By embedding pretotyping into AGL’s innovation process and using it daily and at scale, Richard calculates that his team has saved AGL over $7.5 million over the last two years.
Embedding and adapting pretotyping at scale
Pretotyping and rapid experimentation is used in product development at AGL, but it is also helpful when it comes to planning product launches. “We can use experimentation to generate early behavioural indicators of interest, in order to improve chances of a successful launch.” It doesn’t matter what you are testing: the key is to trust the data, and use it to make decisions and changes.
“You treat every data point [from an experiment] as a learning event. You know a little bit more about something now than you did the day before the test, and therefore what [managers and executives] should expect and demand from our team is [that we do] as many tests as possible.”
As pretotyping became better understood and more practiced within the company’s innovation process and product development, other teams within AGL worked with Richard’s team to integrate hypothesis-led experimentation into their development process. This allowed the company to test and validate ideas before seeking more investment so that people could make informed, data-based decisions on which propositions to move forward. Moreover, recognising the value of rapid, low cost testing, Christine Corbett, Chief Customer Officer, threw her support behind the development of a lean e-commerce platform to allow the Experimentation Team to rapidly pilot products that showed promise through the experimentation funnel; a step change in the support the team could offer back to the wider AGL business.
Using rapid experimentation to stop building products.
A business proposition was developed at AGL to see if customers were interested in buying smart home devices. A team at AGL wanted to run a pilot program, and Richard’s team stepped in to run a series of experiments - in the form of email campaigns to around 100 customers each - to test the level of interest in the market. They gathered data around the best price points and pricing constructs, benefits of the devices and the best messaging to use.
“What gave our program great confidence was when the pilot was delivered to customers and marketing ran large-scale campaign activity: our experimentation results matched campaign activity like-for-like. That was a breakthrough moment for us because it got people to trust the data that we generated.”
Richard’s team have replicated these results dozens of times. An initial pretotype sent to a small number of customers has garnered the same open and click-through rates as when it was sent to hundreds of thousands of customers in the market. By using rapid experimentation at scale, Richard and his team have been able to prove that pretotyping is an incredibly accurate indication of customer behaviour, and can be used to validate ideas at early stages of product development - or, conversely, avoid developing products and services that, through pretotyping, indicate that further investment is unlikely to pay off.
The AGL site setup & ready for an Exponentially pretotyping workshop!
Managing risk and keeping customers happy
Since pretotypes involve testing ideas with real customers, the innovation team at AGL have developed processes with their legal and marketing teams to ensure that pretotypes manage risk appropriately. As Richard explains: “We start by asking the product teams, ‘What’s our learning objective? What is the decision we are trying to make, and what data do you need to make a decision?’ This is so that everyone is clear about what the data will indicate on the other end. We also establish sound and practical guardrails with legal and marketing so that every test is within our brand and legal risk appetite.”
A different mindset to get everybody onboard
Pretotyping and rapid experimentation is not just a method, but a mindset shift. Successful implementation depends on changing the way teams approach innovation, focussing on failing fast, early, and at low cost in order to avoid building products that might result in expensive, high-profile failures.
“Our team likes to encourage curiosity, challenge every assumption and iterate on ideas,” says Richard. This different approach has had flow-on effects throughout the organisation, as Richard’s team has embedded an experimentation mindset into AGL’s innovation processes. “Part of it is the experimentation [methodology] itself...and part of it is because our team is high-energy [with] and has a great storytelling ability.”
The role of Richard’s team is to run experiments and gather data in order to validate (or invalidate) ideas, but key to this is telling a story to executives that convey the value of pretotyping in an engaging and compelling way, not just hit them with numbers.. “The success of what we do is dependent on the stories we tell with the data we generate, and how we communicate the numbers and collaborate with product teams… [pretotyping] requires humanity and salesmanship just as much as the method itself.”
A challenge that Richard still faces when using rapid experimentation at scale is overcoming people’s confirmation bias. Richard says that rather than accepting the data that comes up as a result of a test, “people look for the data that supports their opinion.” To overcome this, he anchors people to principles that help those outside the innovation team understand that data - not opinion - should drive innovation processes. “There are psychological elements at play and people get attached to ideas, [but] the data is the data, and it reflects the behaviour of the customer… It’s not about saying ‘your idea isn’t great,’ but we are honest about the data and present it dispassionately in order to represent what the data is telling us... Then, we collaborate with people to work out the next steps. We are always constructive.”
Measuring the success and value of rapid experimentation
The measure of success for Richard’s team is not based on how many ideas have been validated, but how many tests have been done. “It’s all about volume… we need to get as much data as possible.” This data is what allows other teams within AGL to make strong business decisions, with reliable information from real-world customers about the product or service they are proposing to roll-out.
“We run 1000 tests a year,” says Richard, “and it’s probably going to be 2000 next year… [and that] really high number of tests is great because it means we’re getting smarter [as a company].” As a result of the experimentation team’s success, AGL has made it permanent, elevated it within the organisation, and committed to growing the team and implementing pretotyping at an even bigger scale. According to Richard, it’s because he has proven his team’s value in two distinct areas: “We are de-risking expenditure, and we support product teams to design great products.” AGL is winning industry awards from Canstar and Finder for products that have been refined by the experimentation process.
Richard calculates that through rapid experimentation, his team has saved AGL over $7.5 million. “This gives us an enormous amount of credibility to do more tests, and to have confidence in stopping things if that’s what the data says. You are smarter when you are making decisions based on behavioural data, not your gut. The data is real.”