We’ve reset this form or This page no longer exists?

Session modal tracking was set up through Google Tag Manager to identify how often users’ sessions were timing out, the number of users closing the modal and the number of users being sent to the reset page after their session had timed out.

A session modal in the context of Give feedback on care (GFC) is warning message that appears on the screen to let the user know that they’ve been inactive for 60 minutes, and after an additional 5 minutes their session will be reset.

Session Modal display on Give feedback on care

Within a couple of weeks we were able to identify that there was an issue for mobile users. 82.4% of all session modal pop-ups were on mobile devices. However, very few mobile users (14%) were being redirected to the reset page.

Why is this important? Previously users whose session would time out would experience a ‘This page no longer exists’ message which indicated an error and little explanations as to why they might be seeing this message. The reset page was set up to explain why the session had timed out and encourage users to continue to provider their feedback.

See initial blog post on the session modal performance: https://medium.com/@alex.christensen/the-good-the-bad-and-the-idle-39dddaf73e3f

What we did next?

The data highlighted that the most used mobile device was the Apple iPhone. As a team, we ran tests on this device to investigate what the user was experiencing compare to what we were observing in the analytics data. If the user had opened a new tab in Safari, and left the Give feedback on care form to run for over 65 minutes, the session would continue to time-out, however users were not being redirected. If the user clicked back on the tab, the page would refresh and display that ‘This page no longer exists’.

Image 1: Opens a new tab on mobile, Image 2: Reset page — expected result, Image 3: This page no longer exists — reality

The developers were then able to identify a solution for mobile users. If the user’s session exceeded the timeout, then upon reloading rather than attempting to show the modal, users would simply be redirected to the reset page. This was implemented in Sprint 8 (released 26th May 2020)

Expectations: what do we expect to see in the data?

For desktop users, no major changes in the data.

For mobile and tablet users:

• The number of users taken to reset page should increase
• There should be a decrease in the number of mobile users experiencing a session modal pop-up.
• The number of users taken to ‘This page no longer exists’ should decrease

Reality: What is the data telling us

Reset page

Graph to display the weekly number of pageviews for Mobile users going to the ‘reset page’

Since implementing the changes towards the end of May, the weekly number of mobile users to the reset page has doubled (average of 12 to 28). This is great news, as it indicates that more mobile users were being taken to the right area after their session had exceeded 60 minutes of inactivity.

Session modal pop-ups

Number of times the session modal appears by device category comparing May 5 — May 25 to May 26 — June 15 2020.

There has been a 19.58% decrease in the number of mobile users seeing the modal pop-up. This is a positive indication that the code changes are working as intended, however I would have assumed that there would be a greater drop off of mobile users seeing the pop-up. In terms of raw numbers 475 to 382 events is a large decrease, but still a large number of users being presented the pop-up.

This page no longer exists

There are two ways in which the user may experience ‘This page no longer exists’ message:

• When the users’ session times out
• When the user tries to visit a page after submitting a form

Before we look at the number of users who see the ‘This page no longer exists’ message, it’s important to segment the data and remove any users that have already submitted the form.

Comparison tables of users seeing ‘This page no longer exists’. for segmenting users who have not submitted the GFC form. Table only displays the top 5 mobile devices. Table 1: 5th-25th May 2020, Table 2: 26th May — 15th June 2020

Apple iPhone is the most used device. The interesting takeaway is that there is a 20% decrease in the number of Apple iPhone users who have not submitted the form, but are being taken to the ‘This page no longer exists’ message. This is a secondary indication that the code changes have had the desired result.

Summary:

• Overall decrease in Mobile and Tablet users going to ‘This page no longer exists’
• Mobile users are still disproportionately more likely to see ‘This page no longer exists’ (75%)
• Weekly average on number of pageviews has doubled for ‘Reset form’ for mobile users
• 19.5% decrease in modal pop-ups on Mobile
• When segmenting the data to look at only users that haven’t submitted, 20% decrease in Apple iPhone users going to ‘This page no longer exists’
• Top Page for ‘This page no longer exists’ is ‘Give your feedback’ (20% of all pages)

Where do we go from here?

Although changes made on the session modal have decreased the number of mobile users taken to ‘This page no longer exists’, there are still concerns that some users aren’t being sent to the reset page.

Further investigation is required to understand the impact this is having on mobile users and whether it is a concern for the GFC service.