How Tony Dinh tried to improve conversion using an in-app survey
Tony Dihn, founder of Blackmagic.so — a tool to manage and grow Twitter audience — decided that he could try to improve the conversion rate by running a simple poll.
Â
In May 2022 he asked people why didn't they complete the sign-in flow.
The answer was unexpected.
According to the survey, the majority of users that did not complete the sign up flow did it because they had concerns about permissions (they were probably thinking the tool would be posting someting on their behalf without them knowing it).
Tony changed the onboarding flow. Blackmagic.so started to ask for Read-Only Twitter permissions when users signed up. When a user wanted to get access to more advanced features that need the “Write” permission, they had to grant it separetely.
Â
Did it help? Nope!
Nothing changed in the sign-up numbers as well as in the conversion rate — if anything it went down.
Â
Source: Tony Dihn monthly newsletter
Â
Why it had an opposite effect? Because surveys are not the best way to get insights into customer’s behaviour. Sometimes people don’t have a relevant answer so they just choose the first option. Sometimes they don’t have time for the survey but because they like the founder they agree to spend 2 min unswering the questions — but again, they don’t take it seriously and choose the first option. Finally, people are really bad in understanding the tru reasons behind them doing something or not doing something. They don’t lie — they just don’t know the reason. Besides, how statistically significant can be the results of a survey like this? Who would agree to answer questions about an app they did not even sign up for? Probably, people who are NOT your targeted audience. Therefore, whatever they say, should be of no concern to you.
Tracking actual behaviour and talking to real customers is a more relevant way to craft a better onboarding.
