Most chat experiments prove nothing
Almost every team eventually changes the greeting, watches the numbers for a week, and concludes
something. The conclusion is usually wrong, not because the idea was bad but because chat volumes are
small, seasonal and easily disturbed, and a week of data on a small site is mostly noise wearing the
costume of a result.
The three things that ruin most chat tests are all avoidable. Changing several things at once, so no
result can be attributed. Running for a fixed number of days rather than a number of conversations, so
a quiet week decides the outcome. And stopping the moment the numbers look good, which reliably finds
a favourable random fluctuation in any dataset if you check often enough.
None of this means experimenting is a waste of time. It means the discipline has to be slightly
stricter than it feels like it needs to be, because the sample sizes are small enough that ordinary
carelessness produces confident nonsense.
Write down the prediction before you change anything
The single habit that improves chat experimentation most is writing one sentence before you start:
what you are changing, what you expect to happen, and what number would tell you it did.
The prediction matters because it prevents the most common failure, which is finding a result after
the fact. If you change the greeting and conversion does not move but chat volume rises, it is very
tempting to declare a win on volume — and you might be right, but you did not test it, you
noticed it. Noticing is a good source of the next hypothesis and a poor basis for a decision.
Write the stopping rule down at the same time. Decide in advance how many conversations you will
collect before looking, and hold to it. This is the least natural part of the discipline and the one
that removes the most self-deception.
Run on volume, not on the calendar
Chat experiments should be measured in conversations, not days. A site with thirty chats a week
needs a month or more to see anything short of an enormous effect; a site with three hundred a day can
read a result in a few days.
As a rough guide, differences smaller than about a fifth of the baseline are very hard to detect at
small volumes, and you should be honest with yourself about whether your traffic can ever resolve the
question you are asking. If it cannot, the useful move is to test bigger changes — a different
approach rather than different wording — because only large effects are visible at small
scale.
Where you cannot split traffic between two versions simultaneously, run the comparison sequentially:
version A for a fixed number of conversations, then version B, then A again if you can. The repeat is
what protects you from blaming a change for what was really a seasonal shift, a marketing campaign or
a bad week.
What is genuinely worth testing
Some changes reliably move numbers and some are decoration. The ones worth your limited statistical
budget are the ones that alter whether a conversation happens at all, or whether it reaches a
conclusion.
Strong candidates: whether the widget is present on a given page at all, the hours you are available,
how many fields the pre-chat form requires, whether proactive invitations fire and on which pages, and
the wording of the offline message. Each of these changes behaviour rather than appearance, and each
tends to produce effects large enough to see.
Weak candidates at small volumes: the launcher colour, small wording tweaks to a greeting, the icon,
and anything whose effect you would describe as a nudge. These may well matter, but you will not be able
to prove it, and time spent testing them is time not spent on the changes that would have shown up.
Reading the result without fooling yourself
When the numbers come in, check three things before believing them. Did anything else change during
the window — a campaign, a release, a holiday, an outage? Is the effect large enough to matter
operationally, not merely visible? And would you have accepted the result if it had gone the other
way?
That last question is the most useful one on this page. If a favourable result gets accepted and an
unfavourable one gets explained away, you are not running experiments, you are collecting
justifications, and the numbers will slowly stop meaning anything to the people you report them to.
Be willing to record a null result. Most changes do nothing, that is normal and worth knowing, and a
written record of what did not work is what stops the same idea being re-proposed every six months.
What to measure
Measure the outcome you actually care about, not the nearest convenient proxy. Widget opens are easy
to count and rarely the goal; conversations that reached an answer, a booking or a purchase are what a
change is supposed to produce.
Keep a simple log of every experiment: the date, the change, the prediction, the volume collected and
what you concluded. It takes a minute per test and it prevents the slow accumulation of folklore, which
is what a chat setup becomes after a few years of undocumented tweaks.
Re-test the important settings occasionally. An answer that was true when you had one agent and a
thousand visitors a month may not survive a change in either, and the settings people are most confident
about are usually the ones nobody has revisited since launch.