The Data Factory Operating System

We can use the Pilot Class Selection tab to select varied classes and mold a rough shape of the data we'd like to build. Once the shape and number of classes have been determined we input the amount of hours required for the pilot. For the worked example we're assuming 100 hours is the minimum threshold for quality output. From here we can see how many hours are required for each class along with the relative difficulty. Our value-add comes from targeting long-tail classes, so we expect a higher-than-average difficulty for annotation.

The business model encompasses both audio contribution and annotation. A varied mix of Wizard-of-Oz (WoZ) sessions, scripted speech, and synthetic TTS is used for the contribution side. Again, focusing on the long-tail requires a larger contribution of WoZ sessions, trading off cost for depth. For the annotation side, we're assuming a base Real-Time-Factor (RTF) of 13.5 labor-hours per audio-hour, which jumps to 15.3 when adjusting for the relative difficulty we calculated from the data shape. Accounting for QA, reworks, and fixed operational costs, we land on a pilot figure of $3,168.33.

We use five tripwires while the pilot is running to determine whether classes within the data shape should be scaled, need to be piloted further, or should be cut altogether. The tripwires include:

1. Elicitation yield — does the scenario even produce the class?
2. Inter-annotator kappa — is it labelable?
3. Count floor — do we have enough to judge lift yet?
4. Marginal lift per +100 labels — does the model actually learn from it?
5. Cost per point of lift vs. median — is it gold per dollar?

Ultimately this acts as the gate that further refines the shape of the data and optimizes value output.

Once the shape has been refined we can model cost at scale, pushing from 100 hours to 1,000 hours. We can reasonably expect improvements in both RTF and yield as our systems become more refined and annotators earn tenure. In this case, we have RTF improving by 10% every time volume doubles. The final cost per delivered hour and program cost accounts for these gains and gives the team a target to measure progress against.

The Capacity Plan tab takes the 1,000-hour target and converts it into labor-hours, then headcount, for both pools. Annotation labor-hours come from produced hours times blended RTF, adjusted for redundancy and adjudication, with a QA pass added on top. Dividing by productive hours per person per month gives us the total in annotator-months, which we can then spread across a shorter or longer calendar. For the worked example, we size the contributor pool against the same timeline, since their output sets the pace annotation can actually run at.

We use the Funnel & Hiring Plan tab to size contributor recruiting against the pilot's raw-audio demand. Sourcing runs through five stages, from an initial response through profile verification to actually producing sessions, and each stage loses some share of candidates along the way. Because our worked example leans on the long-tail, we source by condition (i.e. acoustic environment and fluency profile) rather than as one undifferentiated pool, so a scarce profile like accented speech gets recruited separately from the rest. Contributors have to be producing before there's anything to annotate, so this pool has to start first.

The same Funnel & Hiring Plan tab handles annotator certification, which runs candidates through screening, a calibration batch, and a gold-set attempt before anyone is certified. Each stage carries its own pass rate, so the yields compound down to a fraction of who we started with. The target pool size comes straight from the Capacity Plan's headcount for whichever timeline we picked in the step above, so changing the calendar there moves this number too. Certification takes longer than contributor sourcing, so we start recruiting annotators on day one rather than waiting for contributors to already be producing.