Australian Financial Review: The new face of product testing, from pretend chatbots to fake websites

When Melanie Wilson worked with Victoria’s Secret in the US, the lingerie company would buy underwear from a wholesaler, snip off the label, stitch in its own and see what happened.

If the product flew out the door, the company knew there was a market and would buy or design something similar. If not, well, it hadn't lost much.

It's what Wilson describes as a "fake it until you make it" approach to product testing.

"We are the worst possible judge of what good and bad looks like – because we are not the customer," says Leslie Barry of product testing firm Exponentially.

Back in Australia, and from her vantage point working in a number of non-executive director roles, Wilson is concerned that Australia’s innovation efforts could be stymied because established businesses aren't prepared to get "down and dirty" when it comes to product testing.

Start-ups have no choice because if what they are developing doesn't sell, they’re out of business. But many established businesses still have a Field of Dreams approach to product development and think, "if you build it, they will come". They might not.

The importance of good design and product testing through iterative learning and prototyping was highlighted in a recent McKinsey & Co report that tracked the design practices of 300 listed companies in multiple countries and industries over five years.

Companies found to be in the top quartile in terms of design quality outperformed their peers in revenue terms by 32 per cent, and in total returns to shareholders by 56 per cent.

Yet more than 40 per cent of companies don’t talk to their customers when they’re developing something new.

The companies that do so need to talk to customers the right way, says Wilson, who has little faith in focus groups.

"You can ask customers if they are going to buy something," she says, "but until they actually hand over the money you don't know if they are going to buy it."

It's called the Hawthorne effect, which recognises that people's behaviour changes when they know they are being observed.

More reliable results

Greg Booker, chief information officer of insurance and road service business RACQ, says: "If you have people who don’t know they are being watched, you get a more reliable result and you get it more quickly."

That's how RACQ approaches its innovation. For example, it wanted to test the market's appetite for mobility-as a-service, in which consumers buy a taxi ride, a flight, a train trip and an Uber ride on one integrated ticket that gets them door-to-door. Booker says the service has taken off in some parts of Europe.

Booker's team mocked up a website offering the door-to-door service and the company bought three days of Google Ads that were served up when people looked at online flight bookings. Then they waited.

"We had zero uptake," says Booker.

What RACQ did learn quickly, and for little outlay, was that there was no point developing a fully-fledged service in the hope that people would come. The experiment had shown that they wouldn't.

RACQ is a client of Leslie Barry's. He runs Exponentially, a firm that uses the Google-born and Stanford-finessed technique called pretotyping – an inexpensive approach to product testing that lets a business decide whether it should build something instead of whether it can.

Barry has, in his own words, "built five start-ups, sold two and destroyed three". He has also worked for companies, including Sportsbet and ThoughtWorks, which drove him to explore solutions to "innovation theatre, costing millions of dollars with nothing coming out at the end of the pipe".

"My mission is to find the least amount of process, tools and techniques to figure out what we should and shouldn't be building," he says. "The problem isn't ideas. The problem is out of the abundance of ideas, how do you test as many as possible as fast as possible to find the ones more likely to have legs, rather than guessing.

"At the moment we have 100 ideas, we stand at a whiteboard and vote on gut feel. That's broken immediately because we are the worst possible judge of what good and bad looks like – because we are not the customer."

Digital simulation

Pretotyping often involves creating a digital simulation of what the final result might look like, presenting it as if it's real, then seeing if the customer clicks. It means that the 95 duds out of 100 ideas can be quickly eliminated, leaving five ideas worthy of further investigation.

"If you have people who don’t know they are being watched, you get a more reliable result and you get it more quickly," says RACQ's Greg Booker.

In one pretotype, Barry had a team member pretend to be a chatbot and interact with about 30 customers to gauge their reaction. "Eight minutes later, we failed – we were wrong," he says.

But at least the business knew it was wrong before it built a chatbot.

Digital advertising business REA Group is an enthusiastic pretotyper. The executive general manager of REA's customer line of business, Val Brown, says: "The beauty of digital platforms is it provides immediate feedback – data-driven innovation.

"The iterative process we have here means we can leverage data insights. When you are running lots of experiments, you've got fewer variables.

"Running a lot of experiments more frequently means you can optimise for each market and eventually the quality experience."

REA uses an array of techniques for its experiments – product discovery sessions, face-to-face interviews, diary studies, eye tracking on new online platforms – and leverages the online service usertesting.com.

Brown says product testing is inconsistent across the Australian business landscape. "More and more for the digital pure plays it's front and centre, a way of life," she says. "It’s becoming that way for some more mature businesses that have a combination of channels. But compare us with Silicon Valley, and there it's a way of life."

Sandy Walker is an associate professor at Flinders University who notes that only one in seven products is successful in the marketplace and 30 per cent of products fail at launch.

"At the moment we have 100 ideas, we stand at a whiteboard and vote on gut feel," says Leslie Barry.

The problem, he says, is that innovation often starts from the equivalent of "fiddling around in the back shed by people who have not had product training so they don't look outside their own little world, or get out of the office or connect with the customer".

"For some companies, their idea of R&D is to go to a trade show rather than consider who might use their product," he says.

Walker says many companies conduct testing only to ensure that a product meets specified standards rather than to be sure an innovation meets customer needs.

He says there is little excuse for not properly testing, adding that 3D-printed mock-ups of products or digital technologies that enable virtual simulations allow product testing to be conducted relatively inexpensively and often.

He has been working on exhaustive product testing with a start-up called PROHAB, which recently won a Design Institute of Australia Premier's Award and Gold Product award for a clip-on device and app that measures the force used with rehabilitation bands in physiotherapy.

"We made in excess of 100 prototypes and engaged regularly with physios and athletes and patients to get their feedback," Walker says. "We have had 3D-printed prototypes, titanium machined prototypes, we've got breadboard pieces of electronics – each of those have given us a different piece of information.

"The whole reason for that was to manage risk. If you are being truly innovative and creating something that really hasn't existed before, you don't know how people are going to respond to it, you don't know if it’s going to work in the way you expect it to work.

"Every prototype we've built, we've been able to gain some really valuable feedback which has informed the next step.”

"The beauty of digital platforms is they provide immediate feedback – data-driven innovation," says Val Brown, executive general manager of REA's customer line of business.

The same goes at REA Group, where Brown says that product testing for a new app will involve releasing an early version to a small group of consumers – perhaps 5 per cent of REA's customer base, then 15 per cent, then 25 per cent – while learning and iterating at every stage.

Exponentially's Barry believes there is still something of a bunker mentality to innovation and product testing in Australia.

"Enterprise Australia is set up in a way that is super risk-averse – the default in this market is 'how do we protect what we have and move when we are under threat?'" he says.

"In other markets it’s 'how can we get as much as we can and move as fast as we can and own the market and respond rapidly to competition?'

"We have a bunkered down 'we are OK until we are threatened' mentality."

In an era of globalisation, digital disruption and unprecedented pace of innovation, that clearly isn't sustainable.

Myriota's satellites

Myriota's technology is out of this world. Literally. The Adelaide-based company uses terrestrial sensors to collect data – often in remote and hostile environments – then transmits that back to base via low earth-orbiting satellites.

They are deployed in remote cattle farms to monitor water tank levels and in the oceans to track currents, water temperatures and barometric pressure.

Prohab chief executive Lyndon Huf, left, and Associate Professor Sandy Walker, from Flinders University, have designed and developed the PROHAB healthcare system.

"Product testing is very important to us and everything starts with the customer," says David Hayley, chief technology officer of mini-satellite company Myriota.

Haley says the drifter was designed then tested on land first before testing at sea, where a robust enclosure was created to protect both sensors and antenna.

Taste-testing at Attica

Melbourne's Attica was ranked the 20th-best restaurant in the world in 2018. The food is cutting-edge delicious, the service exemplary.

For many years, Attica ran its Chef's Table every Tuesday. Restaurant-goers were degustation guinea pigs, testing Attica’s gastronomic experiments for a cut-down price. Tuesday still plays a big role in Attica's innovation, but these days product testing is performed behind closed doors.

"The trial dinners were about not having the resources at the time to do the dishes always the way you wanted to," Attica owner and chef Ben Shewry explains. "We didn't have any resources to just develop – so decided to do the development in the restaurant and reduce the price.

"We did that for many, many years, but as the restaurant evolved and became more successful, we were able to invest a lot more money into development and make that a specific thing here.

"Now we have quite a big creative group of eight to 10 chefs and we meet on Tuesdays and we have a document we are working from, split into three or four parts – a list of ingredients that we are not currently using, then a list of preparations we are working on – marinades or sauces or plates or preparations of any kind – then a list of dishes in development that are being worked on."

By Friday the ideas will have evolved and the team works together – what needs to be amended, what needs to be eliminated.

"We develop things that we like and hope people eat them – we're never too concerned about how they will be perceived because we have to like it first. We don't ever really reach out to the market to see what they would think of it. We’ve been doing this our whole lives.”

Myriota's transmitters monitor water-tank levels and beam the data to a satellite.

"In the past peoplelv8 c complained about what we were doing – something new takes a lot of self-belief. There were many years where the food was ahead of its time for people’s taste and we were still learning. Not so much now – we work on dishes until they are really good and ready."

Ironskinn's shark suits

John Sundnes enjoys spearfishing and knows there’s an inherent risk when you’re pulling back a line with a struggling, bleeding fish on the end. He also knows that sharks are unpredictable and you can’t stop them biting things. But he thought it might be possible to reduce the impact of a shark bite.

Since 1988 there have been an average of just over 16 shark attacks a year in Australia, according to analysis of the Shark Research Institute's data conducted by finder.com.au.

Working with partner Joe Christie, Sundnes tested hundreds of prototypes before finding a material strong enough to withstand the force of shark teeth but light enough that free divers still need to wear a weight belt.

"We have tested prototype suits which are very pleasant to dive and swim in, and yet afford a high degree of protection from a shark bite," Sundnes says. "We have internally tested various puncture-resistant materials and configurations, and validated that testing with official testing labs."

The company, Ironskinn, is scheduling product testing with various shark species to determine if the material used in the suits can withstand bites from different species. Product testing with professional commercial divers is also slated to help Ironskinn finesse the suits ahead of a commercial launch planned for early 2019.

Seniors living by Aveo Care

It is predicted that by 2030, the number of older Australians needing community support or residential aged care will reach 1.4 million. But as any Baby Boomer will tell you, they may be getting older but they are not old.

Cashed-up Boomers exploring senior living options are a demanding bunch. But as Geoff Grady, CEO of Aveo Care, explains, product testing of different aged-care living options can be tricky. "Testing them in the way you might a new car is difficult because you begin to run up against what consumers don’t know."

"We develop things that we like and hope people eat them," says Attica owner and chef Ben Shewry.

For example, they don't know when they might not be able to live independently and require some degree of care – or even round-the-clock, hospital-grade support. Aveo is one of a number of aged-care providers exploring the concept of vertical villages, where a "continuum of care" is available.

At Newstead, its latest 19-storey development in Queensland, people might invest in a penthouse or unit where they live independently on levels 10 to 18, transition to levels seven to nine when they need care packages of up to 15 hours, and eventually move to levels three and four where they can receive care to the end of life.

Product testing in a conventional sense isn't possible, but Grady says Aveo runs focus groups, exploring what people want both from real estate and contract types.

It has worked with Queensland University of Technology to research the future of aged care, and engages in incremental testing across time – the learnings from one development inform the next. The demand for Newstead, for example, means that Aveo will probably double the number of penthouses in its next development.

"You need to put a price in front of people if you want to test it properly," Grady says. "We opened a sales centre with a mock-up of one of the units, provided a [computer aided] fly-through for all the facilities – the look and feel, the communal areas and the care areas.

"And we did a workshop with a number of prospective buyers as to what they would want in activities, what kind of functionality – we ended up with an app that the residents use to get access to what is on, to get together. Those changes and that innovation was more on the technology side than the physical building – but we received feedback on the layout and flow in one area. We shifted a wall around.”

John Sundnes says he has tested hundreds of fabric prototypes during development of the Ironskinn wetsuit, which is designed to reduce the impact of a shark bite.

Aveo Group chief executive Geoff Grady says product testing of different aged care living options can be tricky.