Research
Designing a Defensible Quality Sample for Small Outsourced Teams
How to define the population, select the items, calibrate the reviewer, and report the limits of a quality sample in a small Philippines team.
On this page
Key stats
The NIST/SEMATECH e-Handbook of Statistical Methods treats sampling and measurement as linked design choices, so a quality sample is only as defensible as the population and selection rule behind it.
Methodology
This note reviews the NIST/SEMATECH e-Handbook of Statistical Methods, the U.S. Government Accountability Office Yellow Book, and ISO quality-management guidance. It asks how a small Philippines-based team can design a quality sample that supports an operational decision without overstating precision. The method is a practical review design, not a statistical certification. It does not claim that a small sample can prove an error rate for a large population or replace a formal audit.
Evidence summary
Start with the question the sample must answer. A sample that checks whether invoices match the approval record is different from one that estimates the overall error rate for a month. The question determines the population, the unit, and the evidence to collect. Write it down before anyone opens a spreadsheet. A vague question produces a sample that cannot support a decision.
Define the population precisely. Include the queue, the account, the date range, and the status of the items. Decide whether reopened, cancelled, and transferred items belong in the frame. A population that changes while the sample is being drawn cannot support a stable estimate. Record the rule and the count at the moment of selection. Keep the frame count with the sample file.
Build the sampling frame from the system that actually holds the work. A list exported from a report may omit items that never reached a certain status. Compare the frame count with the queue count and investigate the gap. If the frame is incomplete, the sample describes the frame, not the queue. Keep the gap note with the sample file.
Choose the selection method before looking at the items. A random or systematic draw protects against the reviewer picking familiar cases. If the queue is sorted by date, a systematic draw with a random start is easy to explain and repeat. Document the seed or the interval so another reviewer can reproduce the selection. A second reviewer should be able to repeat the draw from the note.
Decide the sample size with the decision in mind. A sample large enough to find one or two examples of a rare defect is different from one intended to estimate a rate with a narrow range. Small teams often need a census for high-risk items and a sample for routine ones. State the trade-off between coverage and review time instead of choosing a number by habit.
Stratify when the queue contains distinct work types. A sample drawn from the total may under-represent the difficult items that carry most of the risk. Split the frame by type, then sample within each group. Report the result for each group as well as the total, because an overall rate can hide a failing category. Report each group separately.
Calibrate the reviewers before collecting the full sample. Give two reviewers the same small set and compare their classifications. Discuss every disagreement against the written criteria, then revise the criteria if needed. Record the calibration result with the sample. A sample reviewed by people applying different standards measures the reviewers, not the work. Keep the agreed criteria version with the sample.
Record the evidence for each item, not just a pass or fail. Note the field that was checked, the value observed, and the source used. A finding without evidence cannot be defended with the client or used to improve the process. Keep the record in the approved system and restrict access to the people who need it. Use the same fields across periods.
Report the limits of a small sample. A handful of items can show that a defect exists, but it cannot prove that the process is stable. Avoid presenting a small-sample rate as a precise percentage. Give the count, the sample size, and the direction of the finding, then say what would require a larger review. Put the limitation in the summary.
Use sequential sampling when the cost of review is high. Check a small batch, and stop early if the result is clearly acceptable or clearly failing. Continue only while the evidence is ambiguous. Write the stop rule in advance so the sample is not extended until it produces the answer someone wanted. Record the point where collection stopped and why.
Write a short protocol that names the question, the population, the selection rule, the sample size, and the reviewer. Keep it to one page so a new team lead can apply it without a training course. Version the protocol when the queue or the definition changes. A stable protocol makes two periods comparable and gives the client a document to review.
Use the result to improve one control rather than to rank the team. A finding that a required field is missing should change the form or the check, not just produce a lower score. Assign the change to the person who owns the process and set a date to retest. The next sample should show whether the change worked, and the record should keep both readings.
Define the population
Name the queue, account, date range, and item statuses included in the review. Decide how reopened and transferred items are treated. Count the frame before drawing the sample. A population that is not fixed cannot support a stable conclusion. Keep the frame count with the sample file. Name the queue and the date range, and count the frame before the sample is drawn.
Choose the selection method
Use a random or systematic draw and document the rule. Avoid letting the reviewer choose the items, even when the queue is small. Record the start point and the interval. The selection should be repeatable by a second reviewer. Explain the rule in one sentence to the client. Record the seed or interval, and record the start point so another reviewer can repeat it.
Set the sample size
Tie the size to the decision and the risk of the item. Use a census for a small high-risk group and a sample for routine volume. State the coverage and the review time. A number chosen by habit is hard to defend when the finding is challenged. Review the size after the first cycle. Use a census for a small high-risk group.
Calibrate the reviewer
Test two reviewers on the same small set before the main review. Discuss disagreements against the written criteria. Revise the criteria when the wording is ambiguous. Record the calibration so the review has a shared standard. Repeat the calibration when a new reviewer joins. Discuss disagreements against the written criteria, and discuss every disagreement against the written criteria.
Record the result
Capture the item, the field checked, the value observed, and the evidence source. Separate a defect from a preference. Keep the record in the approved system and limit access. A result without evidence cannot be used for a client conversation. Use the same fields across periods. Separate a defect from a preference, and separate a defect from a personal preference.
Report the limits
State the sample size, the selection rule, and the population count. Say what the sample can and cannot support. Avoid a precise rate from a very small sample. Recommend a larger review when the decision depends on precision. Put the limitation in the summary, not a footnote. Avoid a precise rate from a very small sample.
FAQs
How many items should a monthly quality sample include?
There is no universal number. It depends on the volume, the risk of the item, and the precision the decision needs. Small teams often combine a census of high-risk items with a sample of routine work, and they state the limit of the result.
Can the team choose its own sample?
The reviewer can draw the sample, but the selection rule should be fixed in advance and the same across periods. Letting the reviewer choose which items to inspect invites bias, even when the intent is honest.
What if the sample finds only one error?
One error can still be important, especially for a high-risk item. Report the count and the sample size, and investigate the cause. Do not present the result as a precise error rate for the whole queue.
How do we compare two months?
Keep the population, definition, selection rule, and sample size consistent. If any of those changed, describe the change before comparing. A difference in method can look like a difference in performance.