How Much Evidence Is Enough?

Suppose 20 people buy tickets to a pottery workshop.

Eighteen attend.

Fifteen say they would come again.

What can we conclude?

We can say 20 people bought tickets.

We can say 18 attended.

We can say 15 told us they would consider returning.

Can we say Duncanville wants more pottery classes?

Not yet.

That may be one of the most important rules of the Cultural Interest Study.

An observation is where we begin

Research starts with something that happened.

Someone bought a ticket.

A painting sold.

A workshop filled.

An event struggled.

A customer returned.

Those are observations.

An observation does not need to explain everything.

It simply needs to be accurate.

Twenty people bought this workshop at this price, on this date, in this place.

That statement is useful because we know exactly what it means.

The trouble begins when we ask one observation to carry a much larger conclusion.

Then we look for a pattern

Suppose we offer another pottery workshop.

It sells again.

Then we offer a third.

People buy that one, too.

Some of the customers are returning.

Others are new.

Now we know more than we knew after the first workshop.

We may be seeing a pattern.

Those are different levels of evidence.

An observation tells us what happened.

A pattern tells us that something similar has happened more than once.

A finding is a conclusion we believe the evidence is strong enough to support.

We should earn our way from one level to the next.

There is no magic sample size

It would be convenient if there were one number that told us when we had enough evidence.

Fifty people.

One hundred.

Five hundred.

Research does not work that simply.

The amount of evidence we need depends on the question we are trying to answer.

Twenty transactions may be enough to tell us whether a particular 20-seat workshop sold out.

Twenty transactions are not enough to describe the buying habits of an entire city.

The U.S. Census Bureau reminds users of its own survey data that estimates based on samples carry uncertainty. In general, larger samples reduce sampling error, but sample size is only part of the issue.

We also need to know who is represented in the data.

Who participated matters

Imagine we ask people at a live music event what kinds of arts programs they want Duncanville to offer.

Most of them choose live music.

That would be interesting.

It would also be unsurprising.

We asked people who were already attending a live music event.

Their answers can tell us something about that group.

They may not tell us what everyone in Duncanville wants.

Survey researchers pay close attention to how people enter a study for this reason.

When participants select themselves into a survey, the people who respond may differ from the larger population.

That does not make their answers useless.

It changes what we can responsibly say about them.

A study participant can speak for themselves. We need stronger evidence before asking them to speak for everyone else.

A transaction and a survey answer are different evidence

The Cultural Interest Study uses more than one kind of information.

Surveys can tell us what people say.

The Box Office can tell us what people buy.

Attendance can tell us whether buyers actually came.

Repeat transactions can tell us whether they came back.

Each source answers a different question.

When several kinds of evidence begin pointing in the same direction, our confidence may grow.

If they point in different directions, that matters, too.

People might tell us they strongly want theater while very few buy theater tickets.

That difference is not a problem with the data.

It may be the finding.

We should decide what counts before we know the answer

There is another way to protect the study.

Decide what evidence we are looking for before the results arrive.

The Centers for Disease Control and Prevention’s 2024 Program Evaluation Framework recommends defining evaluation questions, deciding what credible evidence is needed, analyzing that evidence, and showing how conclusions are supported by the data.

That order matters.

Imagine we decide before a test that 30 paid registrations would show enough interest to justify another workshop.

Then 17 people register.

We should not quietly change our standard to 15 because we liked the program.

We can still learn from the 17 purchases.

We can change the next test.

We can decide our original standard was poorly designed and explain why.

What we should not do is move the target simply to create the result we wanted.

Small differences may not mean much

Suppose 52 percent of survey participants say they prefer live music and 48 percent choose theater.

It would be tempting to announce:

Duncanville prefers live music.

That may be far more certainty than the data allows.

A small difference can result from who participated, how many people participated, how the question was asked, or ordinary variation in a sample.

This is why serious surveys report uncertainty around estimates.

The Cultural Interest Study may not always need complicated statistical language in its public reports.

But we do need to be clear when differences are small.

Sometimes the correct conclusion will be:

We do not yet have enough evidence to distinguish between them.

That is a finding.

Enough evidence for what?

The amount of evidence we need should also depend on the decision in front of us.

A $500 experiment does not require the same level of evidence as a $5 million building.

Offering another temporary workshop may require modest evidence.

Hiring permanent staff requires more.

Signing a long-term lease requires more.

Building a permanent cultural facility should require much more.

This gives the Cultural Interest Study a practical purpose.

We do not need perfect knowledge before doing anything.

We need enough knowledge for the size of the next decision.

Then we can test again.

Learn more.

And decide whether the next investment is justified.

Our reports should show the limits

A defensible study should explain what it knows.

It should also explain what it does not know.

That means reporting things such as:

Clear limits do not weaken the study.

They make the findings more trustworthy.

The CDC’s evaluation framework includes rigor, objectivity, and transparency among the standards for high-quality evaluation.

That is exactly the kind of discipline the Cultural Interest Study needs.

We do not have to know everything

The Cultural Interest Study is trying to answer a practical question.

What do Duncanville residents show us they are willing to buy?

We may never know the buying behavior of every household.

We do not need to.

We need enough evidence to make useful findings while being clear about their limits.

Some conclusions may become strong quickly.

Others may take months of repeated testing.

Some questions may remain unanswered at the end of the study.

That is acceptable.

An unanswered question is better than an answer the evidence cannot support.

Subscribe to Ron Thompson’s Field Notes

Occasional notes on art, culture, creative economies, and the work of building cultural capacity from the ground up.

Subscribe to Field Notes

By sending the subscription request, you agree to receive Ron Thompson’s Field Notes by email. Your email address will be used to manage and deliver your subscription and will not be sold. You may unsubscribe at any time by replying “unsubscribe.”