Product Management Case Study: Improving Loket.com Engagement Rate

This case study is part of RevoU Fullstack Product Management to fulfill one of the requirements to graduate from a 3-month apprenticeship program.

Introduction

Hello, my name is Fardil Khalidi. I’m currently a product marketing manager for an early entertainment startup company. I enrolled in RevoU’s Fullstack Product Management (FSPM) program with the goal of strengthening my product management skills, as well as pursuing a job in the top tech industry as a product guy.

To graduate from an FSPM apprenticeship, one of the prerequisites is to complete a Group Final Project based on a real-life case study of a growing startup. I joined Group G and was given the chance to conduct research on Loket.com, an Indonesian ticket management system company. Our goal is to find some initiatives to boost its business performance and make it more scalable.

Project Brief

From this final project, I received a brief to conduct a follow-up initiative based on the findings of the previous group project. We were given approximately 6 weeks to carry out a series of given tasks, including making PRD, making prototypes (low-fi and hifi), conducting usability testing, making development plans, as well as making a launching strategy and success criteria. The output is in the form of findings that are delivered in the form of a deck.

About Loket.com

Quick introduction about LOKET.COM, it is the pioneer of Ticket Management Service (TMS) in Indonesia since 2013 LOKET commits to providing end-to-end event technology solutions in creating the best experience in every event, both for the event creators and audiences. Also, to complete their solution by providing a SELF-SERVICE platform for every event creator to utilize their technology in creating the hype for their customers’ event.

~LOKET is a one-stop solution for all your entertainment needs~

Product vision

LOKET product vision is to provide equal distribution of digital technology for event organizers of various scales.

Product strategy

The product strategy is:

• Ticketing management System: Loket offers ticketing management services technology to support EO needs (from distribution to ticket management)
• Event analytics: to support EO performance, Loket also provides event analysis reports at the end of the event so that they can evaluate
• Turn offline queuing into online: Provide solution for people to buy ticket online (rather than queuing in counter)

User persona

LOKET’s user persona is more likely to be someone who is highly enthusiastic about attending a concert but not willing to buy the tickets because they have to queue for a long time. Besides, someone who likes to attend online events to study or network is also their target user.

Background — Previous Findings

On the previous research we found that some of users not really engage with LOKET because they just come and buy specific ticket that advertised by the EO. There’s no many things offered for user to spend more time at LOKET to explore other events. So we believed, it become a problem that we plan to tackle

While from the findings we found:

• In accordance with the hypothesis that we made earlier, 92.31% of respondents experienced a flow that was only directed to transactions on certain event tickets, did not have time to explore other LOKET pages.
• The bounce rate is quite high (avg. 55.74% ), which means they only experience a small amount of engagement inside Loket.com after onboarding.
• Our respondents suggest several improvement at LOKET.com and through prioritization we’ve already find the solution to be developed

Initiatives

So we come up with two initiatives: (i) enhance the recommendation feature, and (ii) make a wishlist feature

1. Recommendation

Objectives

We make innovations on the recommendation feature by adding some new parameters and make recommendation section on the after-payment page. The expectation is that users are not longer just come and go after buying tickets, but can see recommendations tailored to their interests, so that they can choose it to their wishlist (relate to initiatives 2: wishlist)

Problem

we identified the problem as in follow:

• 92.31% of respondents know LOKET because they are directed by the EO when buying a ticket
• Then, they (the same number above) don’t have the opportunity to explore other events in LOKET
• The bounce rate shows quite high 55.74% (source: similarweb)

We can conclude that the users are not really engaged with LOKET because they just come and buy, then go

Reasons

The current recommendations are still based on the topic chosen by the user only when registering at the beginning and not updated.

High level approach

We take initiatives by first adding some new parameters to the algorithm so that users can get their personalized recommendation. The following parameters are included for addition: (i) purchased history, (ii) location, (iii) events on wishlist, and (iv) big events/promoted events. The second one is adding a recommendation section after the payment page. Here we create a section of similar events or events they might like (smart recommendation) after-payment page.

Goals & success criteria

Goal #1: Increase user spending time scrolling/exploring the website (User Engagement). The success criteria are:

• Decrease % of bounce rate on the after-payment page
• Increase % of user hover on the recommendation page
• Average time spent by users from the recommendation page

Goal #2: Users interact with the recommendation section. The success criteria are:

• # Average time spent of the user at recommendation section
• #of the users that visit the details event page

Goal #3: Increase business performance. The success criteria are:

• #of total purchase
• #of total revenue

User flow:

The user flows are presented as in follow:

User story:

We set the Epic as: enhance smart recommendations. It contains six user stories. Each user stories has their own acceptance criteria. While the priority is 1 (high).

Epic breakdown into user story

Mockup (Hi-fidelity)

Then our UI/UX guy designed the mockup as presented below:

A/B Testing

After it is deployed, we are conducting the A/B testing. The desired plan is Enhance the recommendation feature with add some new parameters to the algorithm:

• User’s personal type of events they ever purchased (purchased history)
• User’s current location (based on browser cache)
• Big events/promoted events
• Event on Wishlist

While for the success metrics we expect to:

• Decreased bounce rate by 8.6%
• Increase 8.6% of user hover on the recommendation page
• Increase average time spent by users from the recommendation page

We also set the hypothesis for A/B testing as in follow:

If we enhance the recommendation feature by adding some new parameters to the algorithm, it will help our users to get their recommendations more accurate and attractive which will decrease the bounce rate by 8.6% because we want to make users explore our events more by providing better recommendation.

While for the sample size, we divided it as follows:

• Weekly users: based on their total monthly visitor count of 871.2k, we guestimate that their average weekly visitor is 217.8K.
• Baseline purchase rate: The Benchmark is based on research from Wordstream, which shows the average conversion rate across all industries ideally is 2.35 percent.
• Minimum Detecable Effect (MDE) by 8.6%
• Statistical significance by 95%
• Sample size variation: we set 100.000 sampling for variation per week. Thus, we got this number 200.000/217.800 = 0.92, which means it can be done less than a week.

See the Product Requirement Documentation for enhancing recommendation engine here

2. Wishlist

The next initiatives is building a wishlist feature.

Objectives

The reason is because it’s not easy to expect the user to immediately convert to buying an event. We added a Wishlist to create an opportunity to save items “for later” if they can’t commit to a purchase at that moment and find them quickly whenever they return to LOKET.

Therefore, we affirmed that by building a wishlist feature it is expected to raise the CVR by 7% that impact to the revenue

Problem

it aligns with the previous findings that has been explained before which are:

• We found quite small revenue (10% ~guesstimate) of to the whole ticket revenue in ID which reaches US $248.20M per year. (source)
• 92.31% of respondents know LOKET because they are directed by the EO when buying a ticket
• Then, they (the same number above) do nothing after checking out the ticket.

We can conclude that:

When a user is narrowed based on their selected wishlist events, they tend to convert to buying

Reason:

• It’s a common feature that other competitor (EventBrite) has this
• Users tend to be confuse when they onboard to the LOKET, choosing which events they want to buy
• With this wishlist feature, we can sustain our users by reminding them to check out.

High level approach

To solve the problem mentioned, the high-level approach that we want to make is initiating a wishlist feature inside the LOKET.COM Here are some of the initiatives that we want to provide such as:

• Create wishlist feature by adding button ‘❤’ on each items (events): It allows users to create personalized collections of event they want to buy and save them in their user account for future reference by clicking button ‘❤’
• Create Wishlist Page: Users can find all list of event quickly in wishlist page which has been saved by clicking button ‘❤’
• User can purchase list of event in wishlist page: Allow users to purchase list of event which has been saved in whislist page
• Create share event feature in Wishlist page: Will give an opportunity for users to sharing event they like to social media

Goals & success criteria

Goal #1: Increase users spending time scrolling/exploring the website (User Engagement). The success criteria are:

• decreased % of bounce rate by 7%
• #of users visit wishlist.

Goal #2: Users successfully add event to wishlist. The success criteria are

• #of users add events to their wishlist
• #of event on wishlist
• %of users transaction rate from their wishlist

Goal #3:Increase business performance. The success criteria are:

• #of total revenue
• #of total purchase

User flow

The user is presented as in follow:

User story:

We set the Epic as: build wishlist feature. It contains six user stories. Each user stories has their own acceptance criteria. While the priority is 1 (high).

User stories of build wishlist feature

Lo-fi (Wireframe)

Our UI/UX guy designed the lo-fi (wireframe) as presented below:

Wireframe of Wishlist feature

Hi-fi (mockup)

She also did the hi-fi (mockup) as presented below:

Hi-fi Wishlist feature

You can also try the prototype here

Here you can read the Product Requirement Documentation

Usability Testing — Improvement

After we have completed problem and solution alignment, product development plan, and hi-fidelity prototype, we do usability testing. The purpose is to find out how usable our initiatives (recommendations and wishlist).

So we initiate two usability testings:

• Add event to wishlist
• Buy event on wishlist page

Add event to wishlist

Testing goals/hypothesis: The user succeeded in adding events to the wishlist, successfully accessing the wishlist page, and successfully sharing the events on the wishlist.

Participants: We selected participants from previous respondents regarding the prioritization problem, where the respondents were included in the LOKET.com user category who had purchased tickets at least once.

UT scenario:

Kamu adalah seorang fresh graduate yang punya interest di bidang desain. Kamu sedang mencari design course untuk mempertajam skill kamu.

Ketika kamu buka-buka Loket.com, kamu menemukan event “Belajar Design Menggunakan Canva” lalu kamu ingin memasukannya ke wishlist.

Setelah kamu memasukkan event tersebut ke wishlist, kamu dapat mengakses wishlist page dan membagikannya ke teman-teman kamu di Facebook.

What we did expect:

• User find & click button wishlist on event “Belajar Design Menggunakan Canva”
• Login pop-up appears
• User logs in
• User adds event “Belajar Design Menggunakan Canva” to wishlist
• User open page wishlist by clicking CTA wishlist displayed on pop up
• User share the “Belajar Design Menggunakan Canva” event by clicking the share icon
• Users share event via facebook.

And here are the result:

It’s quite usable with some minor defects

Recommendation from our participants:

• The BUY TICKETS button on the wishlist page looks smaller and less cramped
• Need to add sharing platforms like Instagram and Tiktok.
• The Wishlist icon looks blurry

Buy event on wishlist page

Testing Goals/Hypothesis: The user has successfully purchased the event that has been saved on the wishlist page and can save the event to the wishlist page from the recommendation event on the payment successful page.

Participants: We selected participants from previous respondents regarding the prioritization problem, where the respondents were included in the LOKET.com user category who had purchased tickets at least once.

UT Scenario:

Sebelumnya kamu telah menyimpan event yang kamu sukai di akunmu.

Saat ini sudah mendekati hari pelaksanaan event, kamu ingin membeli event yang pernah kamu simpan, yaitu “Belajar Desain Menggunakan Canva”.

Setelah menyelesaikan pembelian event di halaman yang sama, kamu dapat melihat rekomendasi event yang serupa dengan event yang telah kamu beli sebelumnya. Kamu menemukan event “Kelas Menjadi Youtuber Andal” yang menarik bagimu, dan kamu memasukkannya ke wishlist kamu.

What we did expect:

• User click button wishlist at menu bar
• User click CTA “BUY TICKET” at event card “Belajar Design Menggunakan Canva”
• User choose how many ticket want to buy
• User fill in order data for the ticket
• User successfully buy the event
• User click button wishlist on event “Kelas Menjadi Youtuber Andal” in recommendation section
• Event successful save in wishlist displayed on pop up

And here is the result:

What we did expect:

UT Summary

We found findings & action as in follow:

• We found that the missed click rate on the wishlist icon is quite high (44.4%). So we make wishlist icon more visible
• The missed click rate on the successful payment page is 100%. All of the users told that they’re not aware of the recommendation section on the page. The action is we make the recommendation section to be visible without scrolling on the successful payment page
• We found this during our interview with some users on UT. (i) make the “BELI” button on the wishlist on the wishlist page smaller, (ii) add Instagram as one of the sharing options, and (iii) create notification when the event on their wishlist is near due date

Recommendation from our participants:

• It’s better if the recommendation section can be seen immediately without scrolling on the successful payment page
• It’s good to have pop-up notifications for events that are approaching their due date
• The Wishlist icon looks blurry.

Prioritization

We are doing prioritization based on RICE (Reach, Impact, Confidence, and Effort)

Here is the result:

Action Plan:

Improvement at rank 1st-4th can be done immediately therefore we can attach the new design to this presentation. Meanwhile, improvement at rank 5th will be placed as out of scope at this time.

Improvement after UT:

And here the improvement after UT:

Improvement after UT

Kano model prioritization

We also prioritize utilizing the Kano model more to convince us and choose which initiatives to implement first.

We construct the questions as in follow:

• For initiatives #1:
  (functional/+)How would you feel if you can save event that you like or interested to on the one page?
  (dysfunctional/-) How would you feel if you cannot save event that you like or interested to on the one page?
• For initiative #2:
  (functional/+) How would you feel if you can get more specific recommendation of event according to your interest ?
  (dysfunctional/-) How would you feel if you cannot get more specific recommendation of event according to your interest ?

While for the likert scale we use scoring: (5) I like it, (4) I expect it, (3)I’m neutral, (2)I can’t tolerate it, and (1) I dislike it for functional. While and (5)I dislike it, (4) I can’t tolerate it, (3)I’m neutral, (2)I expect it, and (1) I like it for dysfunctional

The findings: for the initiative #1 for functional we got 77.8% respondents answer ‘I like it’, and 22.2% respondents answer ‘I expect it’. While for the dysfunctional we got 77.8% respondents answer ‘I can tolerate it’, 11.1% answer ‘I dislike it’ and ‘ I’m neutral’.

for the initiative #2 for functional we got 66.7% respondents answer ‘I like it’, and 33.3% answer ‘I am neutral’. While for the dysfucntional we got 77.7% respondents answer ‘I am neutral’, 11.1% for ‘I dislike it’, and 11.1% for ‘I can tolerate it’

Hence, as in picture shown below, we got that the initiative #1 develop the wishlist feature to become our first priority, followed by initiative #2 enhance recommendation engine.

Product Development

After knowing which initiative we develop first, then we breakdown the product development agenda as in follow:

Sprint Sprint Grooming

First we conduct a sprint groom. This already includes what we have determined through the process of research, conceptualization, and prioritization. We decide that the Wishlist Feature will be built first, followed by the Enhance Recommendation Engine.

Sprint Planning

Here we define one sprint as equal to two weeks. We use Jira In the sprint planning, we set a maximum of 20 story points for the engineering team that consists of backend developers, front end developers, QA and testers, and DevOps.

We also define the agenda as in follow:

• Pre development: a set of activities including product research, writing the PRD, prioritization, create flow chart, user story and acceptance criteria. The participants involved are Product Team, Product Design, and Data Analyst
• Development: it is the development process. Using Agile Scrum framework with CI/CD (Continuous Integration and Continuous Development). Participant: Product Team, Tech Team. Here we set the dev environment levels into three: development, staging, and production.
• QA testing: it is the process after development is done and deployed to staging. QA doing the testing. If the scenario runs well, it put into release plan. If the scenario runs not well, it reverts to development phase to be debugged or fixed.
• Internal launch: After everything is okay, then the staging environment can be released to be tested with internal team. It involves ops team, marketing team, product team, sales team, business team, etc. Here the audience is expected to find some minor issues to be fixed soon by dev team.
• Marketing activation: in this phase, product team coordinate with marketing team to determine what activities to implement for GTM (Go To Market), and what is success criteria
• Official launch: in this phase the product is officially launched and can be used massively by the user
• A/B Testing: in this phase product implements two variants of launching and tested to specific ‘control group’ to gain which one running more effectively.
• Feature performance evaluation: in this phase product team employ product development life-cycle as in follow

Product timeline

overall it takes 4 sprints to develop and 2 sprints for launch and evaluation

it can be seen that wishlist feature is developed first rather than recommendation

Launch Strategy

After the launch, we set the plan for gathering improvement feedback as in follow:

• Coordinate with Data Team to monitor event tracking of overall success metrics that we want to track
• Do social media listening to get overview of user satisfaction and experience
• Tracking NPS by distributing questionnaires to know user’s overall experience
• Coordinate with CS Team to gather user feedback
• Distributing questionnaires to know user’s reasoning about why they leaving our platform
• Coordinate with business development to discuss the impact of this improvement on revenue

Project summary

We are confident that by undertaking these initiatives, such as developing a recommendation engine and wishlist feature, we will be able to address our users’ problem where they tend to leave LOKET after purchasing a ticket, and then encourage them to become more engaged with LOKET.

Score Results

Alhamdulillah, we got satisfying results by coming in 2nd place in the group final project pitching battle in the Madrid Section of the Full Stack Product Management program by RevoU.

Group G lying 2nd the best on the Pitching Battle day

Notable thanks

This is a fascinating accomplishment for which we proud of. I’d want to express my heartfelt gratitude to my group-G of fellows. We would not have been able to do this without your contribution. They are Putu Rikky Prasetya as Team Leader, Chriscelda Marpaung as UI/UX, Berlinda Nefertiti as Product Research (recommendation), and Dimas Yudistira as Product Research (wishlist).