You have never used your own chat as a stranger
Almost everyone responsible for a chat setup experiences it from the inside: signed in, on a desktop, on a page they know, watching a conversation they know is a test. That is not the experience you are shipping. Your visitor arrives on a page they did not choose, on a phone, with a question they are not sure you answer, and with no idea whether anyone is there.
Mystery shopping is the practice of deliberately walking that path yourself, start to finish, and writing down what actually happened rather than what you expected. It is unglamorous and it consistently finds things no dashboard reports, because the problems it finds are not failures — nothing errors, nothing alerts, and every metric stays green while the experience quietly disappoints.
What it catches that transcript review cannot
Quality assurance reads what agents wrote. That is valuable and it is a different question. Reviewing transcripts tells you how well the conversation was handled once it started; it cannot tell you anything about the visitors who never started one, and it never shows you the parts of the experience that happen outside the conversation.
The gaps live in that outside part:
- Whether the launcher is findable on the page a real visitor lands on, at the size it renders on their device.
- What the wait actually feels like when nobody has replied for ninety seconds and nothing on screen says whether anyone is coming.
- What the pre-chat form asks for before a stranger has any reason to trust you with it.
- What happens after the chat — whether a transcript arrives, whether a promised follow-up happens, whether the reference you were given means anything to the next person.
None of that is visible in a transcript, and all of it decides whether the transcript ever gets written.
Run the walk properly
The discipline is to behave like a visitor rather than an administrator, which mostly means resisting the urge to help yourself.
- Use a private window and a device you do not normally test on. Signed-in sessions and cached pages hide exactly the problems you are looking for.
- Start from a real entry page, not the home page. Take one from your analytics, ideally one you have never looked at.
- Ask a real question, in the words a customer would use, including the vague ones. Do not ask a question you have already written an answer to.
- Do not identify yourself, and do not warn the team. A chat that is known to be a test measures the team’s best behaviour, not their normal behaviour.
- Time everything, and write it down as it happens. Memory rounds a two-minute wait down to a moment.
Run it at a realistic hour too. A walk at ten on a Tuesday is a fair test of your best coverage and tells you nothing about the Friday evening a customer actually chose.
The mobile pass is the one that hurts
Do the whole thing again on a phone, on mobile data rather than office wifi, and expect it to be worse. The launcher competes for a corner with cookie banners and back-to-top buttons, the keyboard covers half the conversation, and a pre-chat form that looked reasonable on a wide screen becomes a wall.
Pay attention to what the widget covers when it is open. A chat window sitting on top of the checkout button, the price, or the add-to-basket control is a self-inflicted wound that never shows up as a chat problem, only as a conversion one.
Also check what happens when the visitor rotates the device, or navigates to another page mid-conversation. Losing the conversation because someone clicked a link is the kind of thing that is obvious once seen and invisible forever otherwise.
Test the offline path too
Most teams test chat while staffed, which means the path most visitors take outside business hours goes years without being examined. Deliberately walk it when nobody is signed in.
The questions are simple and the answers are often uncomfortable. Does the widget say plainly that nobody is available, or does it imply someone is? Does the offline form set an expectation for when a reply will come, and is that expectation true? Does the message actually arrive somewhere a person will see it on the next working morning, or into an inbox nobody owns? Send one and find out; an offline form that goes nowhere is worse than no form at all, because it collects a promise you did not keep.
If you run an AI assistant for after-hours cover, ask it something slightly off its documented ground and watch what it does. What you want to see is a graceful admission and a route to a person, not a confident answer that happens to be wrong.
Write down what you found, then fix in order
Record findings as observations with timestamps, not as verdicts. “Waited 3m40s with no acknowledgement, closed the tab” is actionable. “Response times feel slow” starts an argument about whether they are.
Then sort what you found into three piles: things that are broken, things that are confusing, and things you simply do not like. Fix the broken ones immediately, discuss the confusing ones with the team, and be honest that the third pile is preference rather than evidence. A great deal of well-meant chat tinkering comes from that third pile and makes nothing better.
Repeat the walk on a schedule — quarterly is enough for most teams — and after any significant change to your site, since a redesign can move or bury the widget without anyone touching the chat settings at all.
How MyLiveChat fits
Nothing here requires a special mode: you are using your own widget the way a visitor does. What helps is on the other side of the glass. The live visitor list shows your own test session with its referrer, current page and browsing path, which is a good way to confirm the context an agent would really have had. Transcripts let you compare what you experienced with what the agent saw.
Customisation settings are where most of the fixes land — position, colours, the pre-chat form, business hours and the offline message all live in the dashboard rather than in your site code, so acting on what the walk found usually does not need a developer or another deployment.
What to measure
The walk produces observations, and a small number of them are worth turning into numbers you keep: time to first human response, time to a useful answer, the number of fields between a visitor and their question, and whether the offline reply arrived within the window you promised.
Track those across walks rather than within one. A single walk is a story and can be dismissed as a bad day; four walks over a year showing the same ninety-second gap is evidence. Keep the notes, because the most valuable output is usually the comparison, and note who ran it — a walk run by the person who built the setup is worth less than one run by someone with no investment in the answer.