You’re simply A(/B) test, better than all the tests?
One of the key drop off points in the Give feedback on care (GFC) journey is on the ‘Give your feedback’ page. This is the main section that allows users to freely write their comments and concerns on a particular service. It’s also one of the key targets to improve in terms of users exiting the website. Over the last couple of weeks we’ve been running an A/B test to analysis potential improvements on this page.
As a GDS requirement, we need to review methods to improve our completion rate, which hovers around 42% since we went into Public Beta. Our completion rate is measured by the number of users who submit feedback divided by the total number of users who click on ‘start now’. By reviewing our analytics, we’ve been able to highlight the part of the journey where we seeing a large number of users exit.
What is the data telling us by exit rates?
An exit rate is the percentage of users that leave the website divided by the total number of pageviews on a particular page. The below table summarises the top 10 exit pages for Give feedback on care ordered by highest exit rate.
The top page by exit rate is the post submission page. This is a natural end to the users journey and we would expect this to have a high exit rate (57.16% exit rate). This is not an area of concern. The second page on the list is the start page (20.42% exit rate). At this point, this give us an understanding of our curious users, but not engaged users who have established that they wish to give feedback. The third page in the list is the ‘Give your feedback’ page (12.45% exit rate).
The premise
Based on user research, we hypothesised that by changing the introductory message on the ‘Give your feedback’ page we could lower the exit rate. In the variant, we stated that inspectors read the information and highlighted the importance of providing feedback. If we are able to lower the exit rate on this key page, we could improve our overall completion rate.
We also amended the prompt of ‘Enter your feedback’ into a bold style for users that would simply scan the page.
Over an initial two week period we would monitor completion rate, exit rate, average time on page, and probability of being the best solution.
Results — what does the data tell us?
We ran the test from the 11th June to the 1st July 2020. The A/B test was extended as the initial two weeks didn’t give a clear indication of a winning page.
However despite more time, the test has proved inconclusive. A total of 2,950 sessions were tested:
- Original — 1,483 sessions
- Variant — 1,467 sessions
To start, the data indicated that the variant was performing best and had a strong probability of being the better performing page. As we collected more data overtime, the conversion rates narrowed and it became clear that neither page outperformed the other.
Both pages had the indistinguishable exit rate of 12.28%.
Other metrics highlighted;
- that there was limited difference in time spent on the page — 9 mins 20 secs approx.
- Overall form completion rate had a margin of 0.12% between pages.
Google Optimize highlighted that no leader was found. The original page had a 62% probability of being the better solution. A clearly winner is typically decided if a page reaches 95% probability.
For the full data report: click here
Rather than continuing with the test I decided that focus needed to be on a new hypothesis.
What’s next?
- No changes are required to the ‘Give your feedback’ page based on this A/B test. The results were unfortunately inconclusive.
- However, it’s not all bad news, as we’ve learnt that this approach doesn’t generate that much of an impact for users at this point in their journey. We’ll bare this in my mind when considering future tests.
- We all love a winning outcome, however neutral tests highlight that the problem with this page might not be what we anticipated in our hypothesis.
- We need to continue trialling new ideas for increasing completion rate and lowering the exit rate on this page.
Key updates
- Digital Take-up continues to increase in Public Beta. The concern volumes through GFC have increased from 34% in Q1 to 34.6% in Q2.
- Changes to how we’re collecting submission data and removal of internal traffic taken affect from 26th July 2020. Next week’s data should give us a more accurate reflection on the performance of ‘Give feedback on care’.
- Completion rate continues to remain around 42%.
- The UX team conducted an A/B test workshop to highlight other pain points of the journey with the wider GFC team and in smaller subgroups we generated new ideas that are due to be tested.
- These will require some develop from the UX team before we implement onto GFC for testing.
