Can we contact you for more information?

For all negative submissions, Give feedback on care (GFC) allows users the option to provide their contact details. The option for anonymity is a user requirement for the service, however providing contact details is also an inspector requirement for the service.

The form has a fine balancing act between allowing the users to submit without contact details, but actively encouraging them to provide an email address and phone number.

Give feedback on care aims to provide quality feedback to inspectors so that concerns with services can be raised and effectively triaged. In the event that feedback requires more information, inspectors will be required to get in contact with the user who provided feedback. However, this is only possible if the user has submitted feedback with their contact details. Note that only users who are submitting a negative experience will be asked whether we can contact them for further information.

Private Beta data showed that 43.79% (October — December 2019) of all negative submissions were provided with contact details. In the first 3 months of Public Beta, this number had risen to 49.49% (January — March 2020). This continues to be an obstacle for inspectors, if contact details aren’t provided.

As one of key performance indicators (KPI), we have been investigating how we can amend the ‘can we contact you for more information?’ page to increase the number of submissions with contact details.

The alternative B page

Hypothesis: By changing the content on the ‘Can we contact you for more for information? page, can we encourage more users to read the content of the page and be willing to provide their contact information.

However, rather than immediately implementing amendments on the live site without understanding their impact, we can action a variant test. An A/B test can be used to divide the user traffic between two distinct pages and monitor their interactions.

Working with the UX team, we devised an alternative Variant page to trial with users.

A/B Test: 50% of users would see the original page and 50% of users would see the variant page

There are two key questions we wanted to understand on this page:

• If we change the answers on the radio buttons from ‘yes/no’ to full sentences will this give the user more context.
• If we move the confidentiality information out of the drop down and further up the page, can we provide reassurance to the user and make them more likely to provide their contact details.

We use Google Optimize — a variant testing tool, Google Tag Manager and Google Analytics to measure the success of the test.

Our primary metric focused on the percentage of users clicking on ‘yes’ radio button.

Whereas, our secondary metric focused on the percentage of users clicking on ‘more information’ drop down.

How long do we run the test for? Until we’ve reached over 1,000 sessions. This should give us enough users to verify the correct approach.

Results — what does the data tell us?

The ‘Can we contact you for more information?’ A/B test ran from the 27th May 2020–10th June 2020. In this time, we collected data from 1,208 sessions:

• Original — 602 sessions
• Variant — 606 sessions

The chart below demonstrates our primary objective on users agreeing to provide their contact details.

Conversion Rate on a day to day basis. Date range: May 27th — June 10th 2020

The data highlights that on a day to day basis the two pages were evenly matched, however…

Users were more likely to click on ‘yes’ from the simple ‘yes’/’no’ options rather than sentences

• Original — 44.68%
• Variant — 40.26%

Original had a 93% probability to be the best. This information is based on statistical model provided by Google Optimize.

Further insight gained:

• Users were slightly more likely to provide their contact details on the Original
• The secondary metric on users clicking on the drop down for more information showed that users were 2x more likely to click on the drop down on Original (9.75% of users) than on the Variant page (4.29% of users).
• Users were more slightly more likely to exit on the Original (difference of 0.47%)
• Users were spending on average 18 seconds on the Original and 22 seconds on the Variant

For the full data report: click here

Where do we go from here?

Although the data supports that the ‘Original’ version performed best for our primary objective. The secondary metric highlighted that double the amount of users were wanting information on confidentiality when it wasn’t immediately visible on the page. For users that did have that information they were spending slightly longer on the page.

What does this means? We need more insight. My suggestion, would be an Alternative C page. This page would take into consideration the following:

• We keep the radio buttons with ‘yes/no’ text
• Amend the copy to feature the confidentiality elements outside of the drop down
• Review on the live, whilst we run another A/B test on the ‘Give your feedback’ section
• This way we can gain insight in two areas, and we can anticipate that the new modified version won’t negatively impact our data dramatically, based on what we’ve already gathered so far.

Our team will continue to monitor the impact over another two weeks and compare to these two datasets, before drawing a conclusion on our next steps.

Other key updates:

Form Starts and Submissions and Conversion Rate charts. Note: Week 19 = week commencing 4th May 2020

• The number of users beginning the GFC journey continues to fluctuate on a week by week basis
• Conversion rate however, is fairly consistent between 40–46%
• Our next focus is trialling amendments to lower the drop off rate on the Give Your Feedback page and increase the conversion rate. This is currently collecting data.
• Further investigation into segmentation. Can we gain further insight by slicing the data into our user personas by monitoring their behaviour?