Back to projects

The Data Factory Operating System

RTF Model

The RTF Model acts as a lever: pull the class mix one way and cost, headcount, and recommended pilot scope move with it. That's what makes it useful before a pilot starts and worth revisiting mid-pilot, if actuals start drifting from what was priced.

Explore the live model yourself, the title above opens the source sheet.

The model

DFOS Pilot RTF Model

The cost-and-capacity model the pilot is scoped from. Everything is denominated in one delivered audio hour and driven by RTF. Class mix goes in; blended RTF, cost per hour, headcount, and the recommended pilot scope come out.

Pilot Economics

MetricValue
Recommended pilot hours100
Classes funded12
Estimated pilot cost$154,833
Average hours per class8.3
Estimated pilot RTF 15.3
Unit Economics at ScaleValue
All-in cost per delivered audio hour$1,548.33
Total program cost$1,548,332
Cost per interaction $103.22
Cost per tool-call label$17.20
Blended RTF at scale12.4
Staffing to Deliver 1,000 hrsValue
Annotators required @ 12 months42
Annotators required @ 3 months166
Contributors required @ 12 months4
Contributors required @ 3 months15

DFOS Pilot Class Selection

Ranks capability classes on value ÷ difficulty and funds the top N within a fixed pilot budget, producing the class mix that feeds the RTF model above.

Pilot Class Selection — which tool-call capabilities to fund

Pilot budget (audio hrs)100
Classes to fund (top N)12
Floor per funded class (hrs)4
Incedent Elicitation3
Capacity CheckVALID
Estimated Pilot Cost$154,833
Hour-weighted difficulty of funded mix1.1
Base annotation RTF13.5
Estimated Pilot RTF15.3
Class IDClass nameVOIDVoice dependency (1-5)Traffic freqSeverity (1-5)Model err rateValue scoreDifficulty (x avg)Rig produce (1-5)Instances / hrHour weightPriorityPriority (Difficulty weighted)RankFund?Pilot hrsEst. costInstances deliveredGroundingProduction demand signalDescriptionExample tool-call shapeEst. tierDomainDomain / Category
C-03filled_pause_hesitationFALSE545.0%320%0.271.02520.30.336.86.61YES7.2$11,159146Shriberg 1994um / uh; drives endpointing and barge-in tuningHeadSPOKEN-CONVERSATION PHENOMENADisfluency
C-01dialogue_acts_coreFALSE380.0%212%0.190.77536.00.102.93.72YES5.0$7,772181ISO 24617-2; SWBD-DAMSL (42 tags)Backbone layer on all datainform / request / confirm / acknowledge layer on every turnHeadSPOKEN-CONVERSATION PHENOMENADialogue structure
C-17side_speech_addresseeFALSE517.0%434%0.231.4147.70.434.63.33YES8.2$12,70863Device-directed-speech detection literatureBackground speech not addressed to the agentsuppress actionTailSPOKEN-CONVERSATION PHENOMENARobustness
C-09alphanumeric_sequence_captureFALSE59.0%432%0.120.9054.10.792.93.24YES11.8$18,20048Practical; ATIS-era spelling tasks#1 voice-specific failure mode for tool argsOrder IDs, VINs, emails, confirmation codes spoken/spelled aloudarg for any lookup callHeadSPOKEN-CONVERSATION PHENOMENAASR-hard entities
C-19multi_partyFALSE521.0%336%0.231.5449.50.314.52.95YES7.1$10,93067AMI / ICSI meeting corporaSecond speaker joins the calldiarization-dependentTailSPOKEN-CONVERSATION PHENOMENARobustness
C-10spoken_entity_normalizationFALSE511.0%426%0.111.0255.00.572.92.86YES9.6$14,80147SLU slot-normalization practicetwenty-two grand', 'next Friday' → canonical values22000; ISO dateHeadSPOKEN-CONVERSATION PHENOMENAASR-hard entities
C-04user_correction_post_actionFALSE56.0%538%0.111.5452.70.692.91.97YES10.7$16,60929Schegloff, Jefferson & Sacks 1977Highest-value tail class for agent trainingOther-initiated repair after agent acts — undo/redo the tool callchain: cancel → re-executeTailSPOKEN-CONVERSATION PHENOMENARepair
B-05missing_arg_clarificationFALSE38.0%522%0.090.7753.60.481.31.78YES8.7$13,43031API-Bank; BFCLHallucinated args cause wrong writes — criticalRequired argument absent — ask, never hallucinateclarify → callHeadTOOL-CALL STRUCTURAL PATTERNSStructural
C-05barge_in_interruptionFALSE58.0%430%0.101.4153.60.472.41.79YES8.6$13,37831Spoken-dialogue-systems literatureDuplex / streaming agents live or die hereUser talks over agent TTS; requires state rollbackTailSPOKEN-CONVERSATION PHENOMENARepair
B-08write_confirmation_gateFALSE38.0%518%0.070.6453.60.471.11.710YES8.6$13,31731τ-benchThe core safety behavior for voice agentsRead silently; confirm before irreversible writesconfirm → writeHeadTOOL-CALL STRUCTURAL PATTERNSStructural
C-15asr_error_recoveryFALSE57.0%333%0.071.1553.20.481.71.511YES8.7$13,46127SLU noise-robustness literatureMisrecognition → confirm / repair loopconfirm(slot)TailSPOKEN-CONVERSATION PHENOMENARobustness
B-11cross_turn_slot_accumulationFALSE316.0%420%0.131.4157.20.191.91.412YES5.9$9,06742MultiWOZ DST; SpokenWOZArguments gathered across many turns (dialogue state tracking)state → callHeadTOOL-CALL STRUCTURAL PATTERNSStructural
C-11accent_dialect_coverageTRUE525.0%430%0.300.64311.30.004.57.0-VOID0.0$00Common Voice methodologyFairness + WER tailL1/L2 and regional variation; conjoins with all intentsMidSPOKEN-CONVERSATION PHENOMENASpeaker variation
C-02self_repair_same_turnFALSE56.0%435%0.081.2842.70.001.71.314Queue0.0$00Shriberg 1994; SwitchboardSlot extraction breaks exactly hereReparandum → interregnum → repair ('the red— the blue one')affects slot extractionTailSPOKEN-CONVERSATION PHENOMENADisfluency
C-06backchannelFALSE412.0%218%0.040.5145.40.000.71.413-0.0$00SWBD-DAMSLFalse-trigger preventionuh-huh' that is not a turn — must NOT trigger actionsuppressMidSPOKEN-CONVERSATION PHENOMENADialogue structure
A-24identity_verificationFALSE47.0%516%0.060.9053.20.001.11.215-0.0$00Regulated-vertical practiceGate for every account-touching flowKBA step-up before any write actionverify_identity(answers)Head DOMAIN × INTENT CLASSESAccount & identity
C-20prosodic_meaningFALSE55.0%445%0.091.1532.30.001.41.216-0.0$00Prosody literatureText-identical, label-different pairsStress / intonation changes meaning ('I did NOT order that')affects intent labelTailSPOKEN-CONVERSATION PHENOMENAProsody
B-10tool_error_recoveryFALSE35.0%430%0.060.9052.30.000.91.017-0.0$00ToolBenchAPI error / empty result — retry, reformulate, inform userretry / fallbackTailTOOL-CALL STRUCTURAL PATTERNSStructural
B-06no_call_neededFALSE215.0%314%0.060.6456.80.000.61.018-0.0$00BFCL relevance detectionSpurious calls cost latency + errorsAnswerable without tools; suppress spurious callsno callHeadTOOL-CALL STRUCTURAL PATTERNSStructural
C-18grounding_confirmation_strategyFALSE322.0%215%0.071.0259.90.001.01.019-0.0$00Clark & Schaefer 1989Explicit vs implicit confirmation strategy labelsMidSPOKEN-CONVERSATION PHENOMENADialogue structure
B-12mid_flow_arg_revisionFALSE44.0%432%0.051.2851.80.001.00.820-0.0$00MultiWOZ; SpokenWOZUser changes an earlier slot before executionstate update → callTailTOOL-CALL STRUCTURAL PATTERNSStructural
A-28credential_reset_deflectionFALSE43.0%515%0.020.6451.40.000.50.721-0.0$00Security practiceSecurity-mandatory classNever handle credentials by voice; route to secure resetsend_reset_link(acct)Mid DOMAIN × INTENT CLASSESAccount & identity
B-02tool_selection_among_manyFALSE130.0%310%0.090.64513.50.000.50.721-0.0$00BFCL multiple; ToolBenchChoose the correct tool from several plausible ones1 of N toolsHeadTOOL-CALL STRUCTURAL PATTERNSStructural
A-15payment_intent_deflectionFALSE43.0%518%0.030.7751.40.000.50.723-0.0$00PCI-DSSCompliance-mandatory classDetect payment intent; route to secure channel — never capture card data by voicehandoff(secure_payment)Mid DOMAIN × INTENT CLASSESBilling & payments
C-16emotional_escalationFALSE43.0%528%0.041.0241.40.000.70.724-0.0$00Call-center emotion literature (IEMOCAP-adjacent)Frustration / anger → de-escalate or handofftransfer_to_agentTailSPOKEN-CONVERSATION PHENOMENAAffect
C-08anaphora_coreferenceFALSE214.0%318%0.081.1556.30.000.80.725-0.0$00MultiWOZ DSTbook it', 'the earlier one' — resolve to entity from contextresolve → callHeadSPOKEN-CONVERSATION PHENOMENAAmbiguity
A-25update_contact_infoFALSE44.0%418%0.030.9051.80.000.60.626-0.0$00Write with read-back confirmationupdate_contact(field, value)Mid DOMAIN × INTENT CLASSESAccount & identity
C-07underspecified_requestFALSE27.0%324%0.050.9053.20.000.50.627-0.0$00SGD; clarification-question literatureAmbiguous request → clarifying question before any callask_clarifyTailSPOKEN-CONVERSATION PHENOMENAAmbiguity
B-14result_grounded_followupFALSE210.0%318%0.051.0254.50.000.50.528-0.0$00SGDUser refers to returned results ('the second one')resolve ref → callHeadTOOL-CALL STRUCTURAL PATTERNSStructural
B-13multi_intent_decompositionFALSE35.0%430%0.061.4142.30.000.70.529-0.0$00BFCL parallel-multipleOne utterance decomposes into 2+ intents / callssplit → callsTailTOOL-CALL STRUCTURAL PATTERNSStructural
B-04sequential_dependent_chainFALSE25.0%535%0.091.6642.30.000.70.430-0.0$00ComplexFuncBench; ToolBenchChain breaks = task failureOutput of call N feeds call N+1chained callsTailTOOL-CALL STRUCTURAL PATTERNSStructural
C-14safety_privacy_refusalFALSE33.0%516%0.020.9051.40.000.40.431-0.0$00PolicyCompliance-mandatoryRequests the agent must refuse (PCI / PII / policy)refuse + explainMidSPOKEN-CONVERSATION PHENOMENAScope & safety
A-04cancel_orderFALSE26.0%410%0.020.6452.70.000.20.432-0.0$00τ-bench retail; ABCDCancel with policy eligibility checkcancel_order(order_id, reason)Head DOMAIN × INTENT CLASSESCommerce & orders
A-34escalate_to_humanFALSE24.0%512%0.020.6451.80.000.20.432-0.0$00UniversalFailure here is churn; high severityStructured handoff with context summarytransfer_to_agent(context)Head DOMAIN × INTENT CLASSESSupport & troubleshooting
B-01single_simple_callFALSE145.0%24%0.040.51520.30.000.20.434-0.0$00BFCL simpleBaseline competenceOne tool, all arguments present in one utterancesingle callHeadTOOL-CALL STRUCTURAL PATTERNSStructural
B-09policy_constrained_actionFALSE15.0%528%0.071.0252.30.000.40.335-0.0$00τ-benchComplianceAction valid only under policy conditionscheck_policy → actTailTOOL-CALL STRUCTURAL PATTERNSStructural
A-39send_message_emailFALSE43.0%416%0.021.1551.40.000.40.336-0.0$00MASSIVEIrreversible-send confirmation gateCompose; confirm before irreversible sendsend_message(to, body)Mid DOMAIN × INTENT CLASSESAssistant & device
A-05initiate_returnFALSE27.0%312%0.030.7753.20.000.30.337-0.0$00ABCDEligibility check then RMA creationcreate_return(order_id, items, reason)Head DOMAIN × INTENT CLASSESCommerce & orders
C-12code_switchingFALSE54.0%338%0.051.4121.80.000.50.338-0.0$00Bangor Miami corpus; CS literatureMid-utterance language switchTailSPOKEN-CONVERSATION PHENOMENASpeaker variation
A-17book_appointmentFALSE28.0%312%0.030.9053.60.000.30.339-0.0$00MultiWOZ; SGDSlot search then bookingfind_slots(...); book_slot(id)Head DOMAIN × INTENT CLASSESScheduling & booking
A-21book_travelFALSE33.0%422%0.031.2851.40.000.40.340-0.0$00ATIS; MultiWOZ; τ-bench airlineMulti-slot flight / hotel / train bookingbook_flight(origin, dest, date, class)Head DOMAIN × INTENT CLASSESScheduling & booking
A-08update_shipping_addressFALSE33.0%415%0.020.9051.40.000.30.341-0.0$00τ-bench write-gatesWrite action; confirm before commitupdate_order_address(order_id, addr)Mid DOMAIN × INTENT CLASSESCommerce & orders
B-07near_miss_distractor_toolFALSE14.0%432%0.050.9051.80.000.30.342-0.0$00BFCLPlausible wrong tool present in the schemadistractor presentTailTOOL-CALL STRUCTURAL PATTERNSStructural
C-13out_of_scope_deflectionFALSE24.0%315%0.020.5141.80.000.10.343-0.0$00CLINC150 OOSNot in taxonomy — decline gracefully, never force-classifyno_call + deflectHeadSPOKEN-CONVERSATION PHENOMENAScope & safety
A-14subscription_changeFALSE25.0%412%0.020.9052.30.000.20.344-0.0$00SaaS / telecom demandUpgrade / downgrade / cancel with retention offerupdate_subscription(plan_id)Head DOMAIN × INTENT CLASSESBilling & payments
A-45trade_in_appraisalFALSE42.0%324%0.011.1550.90.000.30.345-0.0$00Conjunction with C-09 (VIN capture)VIN-keyed valuation flow; alphanumeric-heavyget_appraisal(vin)Mid DOMAIN × INTENT CLASSESAutomotive retail (example vertical)
A-03modify_order_itemsFALSE24.0%414%0.020.9051.80.000.20.246-0.0$00τ-bench retailHigh save-rate valueChange items/quantity pre-fulfillment; time-window policymodify_order(order_id, changes)Mid DOMAIN × INTENT CLASSESCommerce & orders
A-18reschedule_appointmentFALSE25.0%318%0.031.1552.30.000.30.247-0.0$00MultiWOZCancel + rebook dependent chain preserving constraintschain: cancel → bookHead DOMAIN × INTENT CLASSESScheduling & booking
A-12dispute_chargeFALSE23.0%414%0.020.7751.40.000.20.248-0.0$00Contact-center demandRegulated timelines (Reg E / Reg Z)Open dispute case with transaction referencecreate_dispute(txn_id, reason)Mid DOMAIN × INTENT CLASSESBilling & payments
A-01check_order_statusFALSE213.0%25%0.010.6455.90.000.10.249-0.0$00ABCD; τ-bench retail; SGDTop-3 contact-center intent by volume in retailLook up order state by ID or phone numberget_order(order_id)Head DOMAIN × INTENT CLASSESCommerce & orders
A-30create_ticketFALSE26.0%310%0.020.9052.70.000.20.250-0.0$00Ticket creation with summary + prioritycreate_ticket(summary, priority)Head DOMAIN × INTENT CLASSESSupport & troubleshooting
A-37smart_home_controlFALSE35.0%210%0.010.7752.30.000.20.251-0.0$00SLURP iot domainDevice on/off/setpoint by device + roomiot.set(device, state)Head DOMAIN × INTENT CLASSESAssistant & device
B-03parallel_independent_callsFALSE23.0%330%0.031.1541.40.000.20.252-0.0$00BFCL parallelCheck both my orders' — simultaneous independent calls2+ parallel callsTailTOOL-CALL STRUCTURAL PATTERNSStructural
A-19cancel_appointmentFALSE24.0%310%0.010.6451.80.000.10.253-0.0$00MultiWOZCancellation with fee/policy checkcancel_booking(id)Head DOMAIN × INTENT CLASSESScheduling & booking
A-40navigation_transportFALSE34.0%212%0.010.7751.80.000.10.254-0.0$00MASSIVE transportDirections, transit queriesnavigate(destination)Head DOMAIN × INTENT CLASSESAssistant & device
A-29guided_diagnosisFALSE25.0%320%0.031.4142.30.000.20.255-0.0$00SGD; telecom demandMulti-turn decision tree with device lookupsget_device_status(); run_diagnostic()Mid DOMAIN × INTENT CLASSESSupport & troubleshooting
A-16promo_applyFALSE33.0%212%0.010.6451.40.000.10.256-0.0$00Validate and apply code; explain failuresapply_promo(code)Mid DOMAIN × INTENT CLASSESBilling & payments
A-13payment_plan_setupFALSE22.0%520%0.021.0240.90.000.20.257-0.0$00Collections demandEligibility-gated payment arrangementcreate_payment_plan(acct, terms)Tail DOMAIN × INTENT CLASSESBilling & payments
A-07refund_statusFALSE26.0%26%0.010.5152.70.000.10.158-0.0$00ABCDRefund state lookup and timeline explanationget_refund_status(refund_id)Head DOMAIN × INTENT CLASSESCommerce & orders
A-02track_shipmentFALSE29.0%25%0.010.6454.10.000.10.159-0.0$00SGD; ABCDWISMO ('where is my order') dominates retail call volumeCarrier tracking lookup + ETA explanationtrack_shipment(tracking_no)Head DOMAIN × INTENT CLASSESCommerce & orders
A-11bill_explanationFALSE115.0%26%0.020.6456.80.000.10.159-0.0$00HarperValleyBank; contact-center#1 call driver in telecom / utilitiesExplain line-item or unexpected chargeget_invoice(acct, period)Head DOMAIN × INTENT CLASSESBilling & payments
A-36timer_alarm_reminderFALSE35.0%26%0.010.6452.30.000.10.159-0.0$00SLURPSet / cancel / queryset_alarm(time)Head DOMAIN × INTENT CLASSESAssistant & device
A-32warranty_checkFALSE33.0%212%0.010.7751.40.000.10.162-0.0$00Serial-keyed warranty lookupcheck_warranty(serial)Mid DOMAIN × INTENT CLASSESSupport & troubleshooting
A-09product_search_availabilityFALSE25.0%210%0.010.7752.30.000.10.163-0.0$00SGDFiltered inventory / availability searchsearch_products(filters)Head DOMAIN × INTENT CLASSESCommerce & orders
A-06exchange_itemFALSE22.0%428%0.021.4140.90.000.20.164-0.0$00τ-bench retailChain failures are costly and commonReturn + reorder as a dependent chainchain: create_return → place_orderTail DOMAIN × INTENT CLASSESCommerce & orders
A-43financing_prequal_statusFALSE22.0%516%0.021.0240.90.000.10.165-0.0$00Regulated disclosuresApplication status; compliance-sensitive languageget_application(app_id)Mid DOMAIN × INTENT CLASSESAutomotive retail (example vertical)
A-33schedule_technicianFALSE23.0%320%0.021.2841.40.000.10.166-0.0$00Booking chain with parts + geography constraintschain: check_parts → book_visitMid DOMAIN × INTENT CLASSESSupport & troubleshooting
A-35media_controlFALSE36.0%16%0.000.5152.70.000.10.167-0.0$00SLURP; MASSIVEPlay / pause / skip / volumemedia.play(query)Head DOMAIN × INTENT CLASSESAssistant & device
A-44delivery_pickup_schedulingFALSE22.0%314%0.010.9050.90.000.10.168-0.0$00Slot booking with logistics constraintsschedule_delivery(vin, slot)Mid DOMAIN × INTENT CLASSESAutomotive retail (example vertical)
A-42vehicle_searchFALSE23.0%210%0.010.7751.40.000.10.169-0.0$00SGD-analog schemaInventory filter by make / model / price / featuressearch_inventory(filters)Head DOMAIN × INTENT CLASSESAutomotive retail (example vertical)
A-38list_managementFALSE33.0%18%0.000.5151.40.000.00.170-0.0$00MASSIVEAdd / remove / read list itemslist.add(item)Head DOMAIN × INTENT CLASSESAssistant & device
A-26account_closeFALSE21.0%514%0.010.9040.50.000.10.171-0.0$00High severityIrreversible; retention flow + confirmation gateclose_account(acct_id)Tail DOMAIN × INTENT CLASSESAccount & identity
A-27preference_updateFALSE12.0%38%0.000.5150.90.000.00.072-0.0$00Notification / marketing consent togglesset_preference(key, value)Mid DOMAIN × INTENT CLASSESAccount & identity
A-22modify_reservation_policyFALSE11.0%426%0.011.1540.50.000.00.073-0.0$00τ-bench airlinePolicy violations have direct dollar costChanges gated by fare rules and change feesmodify_booking(id, changes)Tail DOMAIN × INTENT CLASSESScheduling & booking
A-46registration_title_statusFALSE11.0%328%0.011.1540.50.000.00.074-0.0$00State-dependent rules; long tail of edge casesget_title_status(vin, state)Tail DOMAIN × INTENT CLASSESAutomotive retail (example vertical)
A-20availability_queryFALSE15.0%15%0.000.5152.30.000.00.075-0.0$00MultiWOZRead-only slot lookupfind_slots(filters)Head DOMAIN × INTENT CLASSESScheduling & booking
A-31ticket_statusFALSE15.0%15%0.000.5152.30.000.00.075-0.0$00Status lookup and next-step explanationget_ticket(ticket_id)Head DOMAIN × INTENT CLASSESSupport & troubleshooting
A-41info_query_weather_newsFALSE15.0%14%0.000.5152.30.000.00.077-0.0$00SLURP; MASSIVETool-backed informational answersweather.get(location)Head DOMAIN × INTENT CLASSESAssistant & device
A-10price_match_requestFALSE11.0%216%0.000.7740.50.000.00.078-0.0$00Contact-center demandPolicy lookup + case creationcreate_case(type=price_match)Tail DOMAIN × INTENT CLASSESCommerce & orders
A-23waitlist_joinFALSE11.0%110%0.000.5150.50.000.00.079-0.0$00Join waitlist with notify preferencejoin_waitlist(id)Tail DOMAIN × INTENT CLASSESScheduling & booking

What it means

The point of this model isn't the price of a single delivered audio hour, it's using that price to test whether a capability class is worth funding before committing to it at full scale. RTF is treated as a learning curve rather than a constant, so the pilot's blended cost looks worse than where the program eventually settles, and difficulty multiplies that cost instead of sitting in its own line item, so a hard-tail class shows up as expensive here before it shows up as a blown budget in production.

That's what makes it a go/no-go tool rather than a dashboard to check in on: run a class mix through the sheet, read the blended cost it returns, and decide pilot scope before committing headcount. That decision is what the Capacity Model turns into a hiring plan.

Read-outs

The sheet is the model; these are views onto it, showing whether production is holding to what the model assumes, and the unit it prices.

DFOS Production Dashboard

Open full screen

In-flight dashboard to measure quality and trace fidelity. Allows the team to measure progress and develop operationally tangible action plans.