The Australian National University: experimenting with experimentation

This year, Exponentially worked with Australian National University to embed pretotyping into their approach to innovation, in order to improve their student experience. 

“The aim is to have a world-class student and university experience at Australia’s National University,” says Australian National University’s Deputy Vice-Chancellor Professor Ian Anderson, “and to do that we needed to innovate. We needed to listen to students and staff and be able to work with them on ideas that could solve root problems and realise fundamental change. 

“Innovation was not enough. Any organisation can innovate – but our challenge was determining which of the many ideas being generated would affect change, without significant time and financial investment to find the answer.”

So, the ANU Experience Accelerator (EA) was designed as a proof of concept to demonstrate that it is possible to use rapid experimentation to drive innovation and continuous improvement in universities. It was a collaborative effort involving more than 50 people across 13 teams at ANU, incorporating pretotyping (alongside lean and agile methodologies) into the university’s innovation approach to reduce waste and increase efficacy. Exponentially worked with ANU on a daily basis over twelve weeks, providing training, coaching, mentoring and guidance to the EA team, building their capabilities in pretotyping and rapid experimentation.  

Professor Anderson says that pretotyping and rapid experimentation helped ANU to discover which ideas would work, and which could be killed. “The experimentation part of the Experience Accelerator allowed us to… [validate] ideas and dismiss ‘bad’ ideas without wasting significant investment. A complete innovation engine identifies root problems for students and staff, generates quality, targeted ideas, tests and understands solutions to find those worthy of investment and return and implements the solution. [This is] a simple, logical process that can be repeated time after time.” 

How to build a world-class university experience for students, using pretotyping and rapid experimentation.

Ravinth Prasad runs the EA team at ANU. He is the Project Manager in the Service Improvement Group of the Office of the Vice Chancellor, charged with improving student experience. “We needed to discover creative, innovative ways to improve student experience, and challenge current thinking by seeking to deeply understand the problems they are trying to solve from the perspective of the student. This [rapid experimentation] approach assumes that rather than trying to educate or teach our students how they should be navigating our systems and processes...we need to learn how to test and validate ideas before throwing money and effort towards investments...If we can pick out ideas that we know have a better chance of success, we'll be able to reduce so much wasted investment. Think about all the costs that go into executing projects that don't end up delivering the outcomes that you wanted. This is something that we've struggled with over the years, and this is exactly why rapid experimentation was the starting point for the EA.

“Using rapid experimentation to validate ideas before committing resources almost feels like a cheat code. And since learning this, we've been asking ourselves, ‘Why haven't we been doing this already?’ It just seems so obvious. It's probably because we all think that our ideas are great ideas and we jump straight into implementation. In this pilot, we decided to make experimentation our focus because most organizations spend loads of time, money, and energy investigating and defining problems, and also coming up with ideas to solve those problems. What they don't do is take action and actually test those ideas. And this is because taking action is the hardest part.”

Taking action using pretotyping.

Pretotyping allows us to focus on the customer. Instead of asking ‘Can we build it?’ (to which, the answer is usually ‘yes’) we ask ‘Should we build it?’ To answer this question, we run experiments designed to gather data from real customers in our specific context, to be sure an idea will work. Experiments are designed with the customer at the centre, and we seek to improve the customers’ experience of our product, service or company tenfold. For ANU, the customer is the student. The EA team wanted to find data to help develop products and services to improve the student experience. 

So, they pretotyped.

Pretotyping a pilot program.

One of the hardest challenges of innovation is to stop teams from jumping from ideation to execution. Pretotyping uses rapid experimentation as a means to test ideas before executing, so that time, money, and resources aren’t wasted building ideas that don’t work. The problem the EA team at ANU was trying to solve was: how can we improve student wellbeing by building up their resilience and motivation to continue to study at ANU?

The ideation phase consisted of subject matter experts - students, student bodies, staff, counsellors - offering their opinions on how to improve student wellbeing. This generated about 300 ideas, which was consolidated into just over a hundred. With the guidance of experts, the EA team whittled these down to the most high-value ideas, and used a lean canvas to refine them and understand the problem from the customer’s perspective. “At this point,” says Liam from EA, “we had shrunk our ideas down from over a hundred to 30 ideas, and we knew only 10% of them would actually be good. The aim was to get to a small collection of great ideas. We did this by running experiments - 45 of them, over a 12 week cycle, with 27 completed. And towards the end of that 12 weeks, we had 15 ideas that were still in progress, we’d killed 12 ideas, and we'd validated three.” Through the pretotyping process, the team discovered which ideas would actually work, gathering data (rather than opinion) to validate and iterate ideas, so that they knew which had the strongest business case for investment and execution.

“We consider this pilot to be a huge success. We only had twelve weeks and a small team. We chose a really difficult area to investigate. We listened, heard what everyone had to say, and we took 300 ideas down to three business cases, and we also avoided executing 24 bad ideas... The money we saved with those 24 ideas is a success in itself.” .

Killing ideas, validating ideas.

One of the most high-value ideas discovered during the ideation phase was a chatbot that would assist students to find the information and resources they need to improve their wellbeing. “This was something with a bit of potential, and if it worked it would be a huge weight off the student body’s shoulders, as well as the administration.” 

The EA team ran some pretotypes to test and validate their idea. This is what happened:

 “The interesting thing about experiments is that they are all about context. We ran this in the holiday period, when students do not care about exams or assessment at all - there was no point running the experiment for all students. So we decided to hone in on our customer to students who were seeking special consideration for exams during the holiday period, and were actively searching for help submitting the documentation they needed to provide. This allowed us to gather more reliable information while also solving a problem for students in real time.”

“We found out that out of 194 page views, the bot was used 14 times. That was 7.2%, which was past our threshold for this particular experiment.” The idea had been validated, so it was time to iterate the experiment. “We noticed that they had an affinity for clicking buttons over writing text, so we ran another experiment to see if engagement increased when the bot opens with a welcome message with buttons to click.” 

“We found engagement jumped to 17.5%, which was really promising. But we needed to iterate the experiment again to make sure that the idea was still valid outside of this very specific context and use case. So we made a more general assessment appeals bot that would run for three days. 

“In three days, the bot was used 408 times out of a pool of 2,500 students visited the page, which is about 16%, or one sixth of students. Now, with this data, we can say that the idea has been validated and that there is a really value-adding proposition for ANU.”

By using pretotyping, the EA team was able to comprehensively gather the data and develop the use case for a chatbot, allowing ANU to be confident that pouring time, money, and resources into developing the idea is a solid investment that will improve student wellbeing.

Pretotyping is simple, clear, and lean.

Ravinith says: “Pretotyping is about trying things and being comfortable with the unknowns, but having a clear direction provided by strategy and using customer insights really helps to sharpen your focus and keep the innovation process moving forward. Without pretotyping, the team is slowed down by discussion, instead of just trusting the process, following the formula, and doing things rather than just talking and thinking about them.

“It's very tempting to jump ahead and start designing experiments, which is super fun, but it's important to stay on track and really pull apart the idea and understand the problem - from the customer’s perspective - first. Having a customer focus is absolutely critical. We all try and be student-centric and look at things from the student's perspective, but regularly bump up against what is easiest or practical for us the organization. Like lean concepts, pretotyping puts people at the centre, and that’s really valuable.”

Pretotyping is fast.

Using pretotyping in this pilot allowed the EA team to get things done quickly, even with a small team and in a short period of time. “The faster we designed and ran experiments, the faster we learned lessons, and we were able to iterate and run more experiments. This pace and momentum really helped our team and the people we were working with. It allowed us - and our customers - to see that change is actually possible because the customer's perspective is so important. 

Pretotyping helps kill bad ideas quickly.

“In innovation, it is absolutely necessary to remove opinion and emotion from the idea. We all had ideas that we were rooting for and became very attached to, which made it all the more painful when they were killed, but bearable because we killed them using data and not opinion.”

Pretotyping helps innovation teams.

“What really stood out during this pilot is that we can test all kinds of ideas. At scale, we have the opportunity to be bold with the problems and ideas that we work on to really leverage our team’s full potential. As the weeks went on, we ran more experiments, had more conversations and really learned by doing, how to have confidence in the group and how to trust the process. And now, we’re happy to tackle any problem and any idea.”