This method of evidence is about using the unplanned user feedback a company might receive. These pieces of feedback usually come from sources like enquiry forms, emails, social media, phone calls, or in-person conversations. This is feedback that doesn't just exist on live chat or feedback you have sought out in surveys, it can come in at any time.
With this kind of user feedback you often learn the things that truly bother people, as they have taken the time to seek you out and complain or suggest something. They haven't just waited until asked in a survey, where they may have come up with something for the sake of it.
It is important to have an approach for collecting this evidence as it is the kind of thing that can be easily missed or lost if there is no system for it.
Setting up a system to record this feedback will usually involve a bit of manual work. The majority of your feedback will come into customer service teams or from customer-facing staff, either via email or phone. You will need them to log every time a customer complains about something or makes a suggestion for improvement.
This can be done via CRM software if you have the budget or it can simply be a shared spreadsheet where they can enter a summary of the issue and an identifier for the user that raised it. The feedback this doc gathers should be more long-term and focussed around improving features. These customer service staff need to be able to distinguish when something is just plain broken—as this means a bug ticket needs raising with the development team.
Once you've got a solid system for logging ad-hoc feedback someone (usually a product manager) will need to check it regularly and classify the different issues to make spotting similar ones easier. Either use tags in a CRM or add an extra column of data in a spreadsheet where you specify what category issue each thing is, for example 'password reset emails slow', ‘requests wishlist functionality', 'bigger upload capacity' etc.
When you group issues like this you can keep a running total of the most requested or complained about features. This leaderboard can be a part of your suite of evidence for deciding the priority order for areas of your product to redesign. By checking it every week or so you can see if there are sudden spikes in certain issues or if there’s a leading problem that needs fixing more urgently.
When dealing with this kind of feedback there will always be a bias towards the negative. As humans the frustrations or the bad things that happen stand out more than the smooth easy experiences, and as online user experiences improve people get more accustomed to everything working really well first time.
Just be aware that you're mostly going to see problems highlighted in this feedback so be careful not to throw out everything that is good when redesigning to solve the issues.
Generally, people don't like change. A common time for a raft of complaints is if you release a big redesign of a major feature or part of your website. If it's something lots of people use every day then don't be surprised to see confused and sometimes angry feedback when you first launch. This is to be expected as they adjust.
However if you get an avalanche of negative feedback or users are still making the same complaints weeks after launch then you might have a real problem. At this point you should try to understand why the complaints are coming in (user testing can help here) so you can fix it.
Not all complaints are created equal. Someone quickly sending a moaning Tweet is not as meaningful as someone taking the time to phone up and explain their problem. You might want to weight your feedback to reflect the source it came from.
Be careful not to misunderstand the feedback. It can be hard to truly understand what someone means if they've written a few sentences. People can have all sorts of odd ways of phrasing online behaviour and probably won’t know the correct technical terms.
Flag any feedback you're not 100% sure about and try and clear it up with the person that gathered it. Even better, if you have contact details for the user that raised it, then get in touch with them to get clarity.
Customers and users don't always know what they need so focus more on their problems than suggestions for new features. They may think they need a big all-singing, all-dancing feature but a simple tweak may be just as good. It's your job to define the solution, don't just implement what they ask for.
There are lots of different CRMs out there, and they’re generally not that cheap to implement. Some focus more around sales but usually incorporate customer service too and the most well-known are: Salesforce, Zoho, and Sugar.
The most basic free shared log you can use would be a Google Spreadsheet (a tool I've mentioned a few times in this guide and something I’ve seen many startups be built on!).
To check on and classify about 50 pieces of feedback a week should take an hour or two.