In our last paper, we discussed our top tips for recruiting hard to reach markets. Now we want to switch the focus to the analysis portion of the market research process. Here, in this article, we share our best practices for analyzing small samples.

What constitutes a “small sample”?

To begin, let’s first discuss what constitutes a “small sample”. If you are looking for a numerical value, typically any sample under thirty data points is considered small. But there are other criteria to determine if your sample is small. First, if your data is highly influenced by outliers, it is considered small. This can be a problem because if outliers have a strong impact on your data, then your results will be skewed and the possibility of you arriving at the true parameter estimates will decrease.

Second, you should have valid estimates of the sample’s parameters and standard errors. If your sample is small, your estimates will be less accurate because your precision is decreased. Thus, if your sample does not produce valid estimates, it could be because your sample is small.

Third, if you were to have repeated your study multiple times, you should begin to converge on the true estimates. If this is not the case, then the variability from your findings are too great; and thus, your sample may be too small.

Finally, the size of your sample and the size of the effect of the sample should be appropriate. In most cases, the smaller the sample, the larger the effect size you will be able to detect. If your data can only detect large differences, then it may likely be due to a small sample.

Strengths of Small Samples

Although in most cases it is desirable to have a large sample size, having a small sample does have its strengths. To start, studies with small samples can be done more quickly. This can be beneficial if you are on a time constraint or are just looking to pulse your target audience.

Another benefit to small samples is that the cost to do the market research is generally lower as expenses like honoraria, travel and facilities are decreased.

An additional strength to small samples is their ability to be used for exploratory research. Small samples can be a great way to test new hypotheses and gain insights that will help you develop future studies without using up a lot of resources, time and money.

Non-Statistical Considerations When Working with Small Samples

When you are working with a small sample size, there are a couple of non-statistical steps you can take in order to improve your analysis.

First, you may want to make use of secondary endpoints, which can help you draw hypotheses about your primary endpoint. For example, if your primary endpoint is overall survival rate, you may look at reduction in tumor size as a secondary endpoint. Looking at secondary endpoints can help you because they broaden the scope of your research and thus help you detect more differences.

Speaking of differences, another non-statistical consideration to think about when working with small samples is to focus on large differences in the data. When dealing with small samples, it is difficult to detect small changes so make sure you are realistic when developing your hypothesis. Sometimes there may not be a statistically significant difference, but there is a “suggestion” of a difference. It may be important to report on these findings too and consider possibly using these results as a basis for further research.

Finally, you may want to consider rewording your research question in order to accommodate for the small sample size. For example, if your initial hypothesis was Test Group A is going to live 3 days longer than Control Group B; but you have a small sample size, you may want to reword your hypothesis to Test Group A will live 3 months longer than Control Group B. You want your research question and hypothesis to be realistic, so make sure they are reflective of what you can accomplish with your given sample.

Stat Testing on Small Samples

While it may be difficult to do complex statistical testing on small samples, simpler analyses can still be used to detect differences in your data. For example, the two-sample t-test for continuous data and z test for proportions for categorical data can both be used on smaller samples. In particular, the t-test is especially good for small samples and there are tactics you can use to increase the value of t.

One method is to increase your parameter estimate. You can do this by sharpening the focus of the study. For example, if you have a control group and a test group, you can increase the differences between the two groups by making sure your control group receives absolutely no trace of your independent variable.

Another way to increase the parameter statistic is to run statistical tests on secondary variables as their effect may be easier to detect than your primary variable’s effect. You can then see how these secondary outcomes effect your primary variable in the long term.

Besides increasing the focus of your parameter estimate as a way to improve your t-test, you can also decrease your standard error.

One way to do this is by including all data in your analysis, even if it is incomplete. There are many methods available to help you fill in missing data, like the multiple imputation method. By including incomplete data, you will have a larger overall sample size to work with, thus, increasing the power of your statistical test.

You can also reduce the standard error by applying the finite population correction formula. This formula is applicable when your sample consists of more than 5% of the overall population. What this formula does is reduce your standard error and thus increases the power of your statistical tests.

In Summary

Conducting analysis on small samples is no easy task and there are a lot of things you must consider as a researcher or decision marker. Hopefully, this paper has provided you with some insights into our best practices for conducting analyses on small samples so you can feel more confident about your research findings.