Optimizing Engagement: A/B Testing Toronto Star’s Homepage
As one of Canada’s largest newspaper companies, the Toronto Star covers national news to local stories across the country. We wanted to A/B test one of The Star's most prominent mobile article designs (the stacked article layout) to improve app user engagement and article clicks.
team
1 Product Manager (me)
2 Software Developers
1 Product Designer
1 Senior Product VP
timeline
Identifying the Problem: 0.5 weeks. Breaking Down the Problem: 1 week. Creating a Floorplan: 0.5 week. Bringing a Solution to Life: 1 week. A/B Testing Our Design: 2 weeks. What Did We Learn. What's Next.
my role
Project Lead (self-initiated project)
Design Challenge
"How might we maximize Toronto Star’s mobile app user engagement and increase our article clicks?"
what's the problem
5/5 Toronto Star app users say the homepage is where they look to find their daily articles. Ensuring that we engage our users on their first view becomes that much more imperative. We needed to improve The Star’s mobile engagement by increasing article clicks.
solution
To do this, we targeted Toronto Star's stacked article layout. Outlining our project and requirements in a floorplan Epic was the guideline for design changes needed. We conducted an A/B test between the new variant and current app design to determine which was the most engaging layout.
outcome
Conclusions to our A/B test deemed the variant stacked design to be the winner. The baseline slightly exceeded on stacked and non-stacked article conversions. However, the variant layout significantly outperformed in "more" button clicks which is essential in keeping users on our app.
Identifying the Problem
"How do we increase Toronto Star's mobile app article clicks?"
The most crucial part of any product roadmap is identifying the situation of concern. In this case, it was how we could increase The Star's mobile app user engagement in the form of article clicks. Higher article clicks mean more ad views through our articles, and increased subscriptions with more users running into our 3 article paywall.
Breaking Down the Problem
Stepping into the shoes of the user
The best way to relate to users is to hear from them firsthand. I was gracious enough to conduct user interviews with some of our mobile app users. This allowed for a better grasp of what our app user experience is like.
Our homepage is the #1 place that users use to find their stories but...
Our homepage is extremely long and takes too much time for users to scroll through to the end. Users would leave the app before being able to scroll to the bottom of the page. This was reflected by our "Parse.ly" analytics, which highlighted that topics at the top of the homepage receive a significant amount of increased user engagement compared to the articles shown later on our homepage.
Competitors engage users by showing them information. A lot of information...
We also wanted to understand the wants/needs of potential users on competitor apps. We deduced that the most successful news apps engage users by bombarding them with content. Using this method, users can view all types of articles with the least scrolling.
Creating a Floorplan
Displaying More Content = Increased Article Clicks
We know we want to increase user engagement on the Toronto Star homepage. Our answer was to increase the amount of content presented within the viewing body (viewing body is the screen area that displays news topics and articles). Our hypothesis emphasizes that by decreasing the scrolling and increasing the amount of content shown, a user’s likelihood of finding articles would increase. With more articles to view, this will ultimately increase the likelihood of article clicks and user engagement.
“As a user who casually scrolls through the homepage as a means to find new articles to read, I want the stacked articles to take up less space. This way, I am viewing more content, increasing my likelihood of clicking on articles.”
Toronto Star's stacked article layout is used for 70% of the ENTIRE app's articles
One of the most prominent mobile article layouts was the stacked article. On The Star’s mobile app, this article design makes up 85% of the first container on the homepage (showing anywhere from 5-12 stacked article designs). Targeting these articles would create the largest impact. By minimizing the amount of space used by stacked articles, we would increase the number of articles we show in the viewing body.
What should our solution achieve?
01. “Decrease the space taken by stacked article to show
more content to users”
02. "Decrease the amount of scrolling that is needed to be done by the user while showing them the same number of articles”
03. "Remove clutter on the homepage”
Bringing a Solution to Life
A new stacked article design
With the completed Epic in hand, it was time to see our solution come to life. Collaborating with a designer, we brainstormed some potential design solutions to our problem. Although various designs were presented, we decided to move forward with a simplistic design change. Due to confidentiality, I can't show the final design changes.
Justifying our design
Keeping a similar design was a requirement to our stakeholders since being consistent with other layouts within the Toronto Star app supported brand personality. Changes to the text size and positioning were made to the proposed variant layout. This kept the changes simple while still increasing the number of articles shown in the viewing body.
5/5 Users mentioned that article thumbnails heavily influence which articles they end up reading
Implementing a smaller article thumbnail decreased the amount of stacked article real estate while providing that visual element to users. The decrease in font size and vertical height spacing also helped decrease space showing more content within the viewing area. Although we were able to show an additional 2-3 articles within the mobile viewing body, our solution to decrease the text size did come with its doubts. Article analytics shows our users often increase article text by one or two indices using our font size slider. Although this was a change on the homepage rather than the article text itself, we wanted to monitor how our users would react to the new design change.
A/B Testing Our Design
What makes a winner?
Our A/B test in Firebase allowed us to compare the new variant design with the current stacked article layout. For this test, both layouts were measured based on three success KPIs.
01. Number of Stacked Articles clicked
02. Number of Non-stacked Articles clicked (all other articles that do not follow the stacked article layout)
03. Number of “More” button clicks (each news topic has a "more" button that brings users to the corresponding topic subpage)
43,000 app users tested our A/B staked article design
For this A/B test, we had a sample size of 43,000 users. This included a total of 32,000 iOS users and 11,000 android users. For both operating systems, half of the mobile users were given the baseline design while the other fifty percent were using the new variant stacked article design. For Apple iOS, this meant 16,000 had our baseline design while 16,000 users used our variant design. Similarly, we had 5,500 Toronto Star Android users using our baseline design and 5,500 users receiving the variant design.
32,000
11,000
After a total of 15 days, the three KPI metrics found an almost equal split between the new variant stacked article design with the current app layout. In stacked article and non-stacked article clicks, the baseline design beat the variant by around 0.45% (-0.85% difference between the two designs). These results were similar for both Android and Apple device users. For the “more” button clicks, we see a different story. The variant design defeats the baseline by approximately 0.3% (+16.3% difference between the two designs) in “more” button clicks.
Another interesting thing that can be seen is that the baseline well exceeded the variant design during the beginning of the test. This is by a much greater percentage difference of -11% engagement in almost all three event metrics. As the test ages, we see that the difference between the two layouts level off, both achieving similar clickthrough rates.
What Did We Learn
Although our baseline obtained higher conversions for stacked article and non-stacked article clicks, our variant design well exceeded conversions for "more" button clicks. For slightly higher stacked article and non-article conversions, one possible reason is that The Star app is catered towards Toronto Star power users. Many of these users are avid supporters. As a result, these users may routinely check The Star reading through every article title regardless of design layout. With the article change on the homepage, increased "more" button clicks indicate that the variant design is pulling more users to section pages to view more articles. With this in mind, the recommendation to proceed with the variant stacked article design seems more beneficial in the long run.
One concern with our final variant design was that decreasing the article title font would frustrate users. We have received many user messages regarding how to increase the font size on articles. Although this change was on the homepage rather than on article pages, our top priority was to ensure that this design change did not negatively impact the user experience. Fortunately, with over 21,000 users using the variant design, we had zero complaints regarding the decrease in font size. Although it doesn’t sound like much, information like this is vital in understanding our mobile app users and creating better features in future design changes.
What's Next
Although quantitative data from the stacked article A/B test has confirmed that changing to the variant design for stacked articles will yield the best user engagement, further research regarding qualitative data must be collected. Internal qualitative user testing with Torstar employees and avid news app users has found the variant stacked article design more engaging and intuitive. Quantitative data is significant, although qualitative user testing is the other half of the coin. Conducting a user test to understand whether the new stacked article design improves the user experience is key in building a product that will maximize overall user engagement and article clicks.
User testing is an effective method to gather data from a variety of viewpoints that represents the Toronto Star mobile user demographic. This may include app power users, new users, and users from different ages, backgrounds, and areas. Here is a timeline estimate for a future user test.
Day 1: Preparation, including brainstorming questions and key points that we want to answer through our user test
Day 2 and 3: Organize and gather a panel of users for user testing
Days 4, 5, and 6: User interviews and testing, dependent on how many users will be participating
Day 7: Analyze and summarize findings of the user tests
