# Human in the loop AI: when a person approves and when they only watch > Approving every agent reply eats between a tenth and a quarter of the hours the agent was meant to give back. We show three oversight modes, what each costs in hours a month, and the point where you can loosen the reins. - **Canonical URL:** https://appwave.dev/en/blog/human-in-the-loop-ai-human-oversight-in-a-small-business - **Language:** en - **Updated:** 2026-10-06 --- - **Published:** 2026-10-06 - **Author:** Jakub Dulas - Co-Founder, Backend Developer - **LinkedIn:** https://www.linkedin.com/in/jakubdulas/ - **Reading time (min):** 7 - **Topics:** Human in the loop, AI agent, Customer Service ![A stream of small black spheres flows along a sine wave between two dark pillars. One sphere in the middle rests in a cradle on a stand and glows emerald, while the others pass on untouched.](https://cms.appwave.dev/uploads/human_in_the_loop_ai_human_oversight_in_a_small_business_5ce1f88a30.webp) **Human in the loop AI is a setup in which an agent prepares a reply, and a person approves it, corrects it or takes over before it reaches the customer. In a company of 10-200 people, three modes work: approving every reply at launch, approving only cases below a confidence threshold after a few weeks, and reviewing conversations once the agent is stable. Oversight lowers the cost of a mistake, but it does not remove it.** If you are thinking “I’m afraid the bot will say something stupid to my customer,” this text is about turning that fear into settings. You will find the definition in every guide. What is missing is a number: how many hours a month a person in the loop takes, and when you can move them out of it. ## Three oversight modes and the hours each one takes IBM describes human in the loop as a system or process in which a person actively takes part in the operation, supervision or decisions of an automated system. The definition is wide, because it covers labelling training data and it also covers clicking “approve” under a draft email to a customer. A service company cares about the second. In daily work the term splits into three modes. The names and the lines between them are our split, not a settled definition. The modes differ in when the person steps in, and that translates directly into their time. - **In the loop.** What the person does: approves every reply before it is sent. When it makes sense: the first 2-4 weeks, and cases involving money or personal data. - **On the loop.** What the person does: decides only the cases below the confidence threshold. When it makes sense: after calibration, for repetitive questions. - **At the helm.** What the person does: sets goals, limits and handoff triggers, then reads the logs. When it makes sense: a stable agent and growing volume. ![Timeline of an AI agent rollout from day one to month three. Above the axis, three oversight modes: approving every reply, approving only cases below the confidence threshold, and setting limits with a review of the logs. Below it, a curve of cases approved by hand, high at the start and low after calibration.](https://cms.appwave.dev/uploads/human_in_the_loop_ai_human_oversight_in_a_small_business_diagram_34d6035479.webp) The third mode has a name of its own. Paula Goldman of Salesforce calls it “human at the helm.” She argues that approving every step stops working once you use agents, because the whole point of an agent is to travel a multi-step path to a goal on its own. The person sets goals, limits and handoff triggers, and then looks at the audit trail. **Probable conclusion.** This is the opinion of one executive at a software vendor, published by Forbes on 29 September 2026 in a piece marked as sponsored by ServiceNow. It is not a study. It is useful as a name for the mode. It does not prove that you can skip approvals before calibration. ## Three places where a person changes what the agent does Approving is the least interesting part of oversight. More changes in three settings that someone in the company fixes before launch and corrects after it. **The handoff threshold.** In our text on [customer service automation that doesn’t turn your company into a call center](https://appwave.dev/en/blog/customer-service-automation-that-doesn-t-turn-your-company-into-a-call-center) we set out the rule: “Below the set threshold, it doesn’t respond but instead forwards the matter along with the entire conversation history.” Two signals work regardless of the threshold. A customer who rephrased the same question twice did not get an answer, so the conversation goes to a person without further probing. Agitation detected in the message ends the conversation with the bot at once. **The knowledge boundary.** An agent that always answers is more dangerous than one that sometimes admits it does not know. In [BetterCX](https://appwave.dev/en/portfolio/bettercx-saas-platform-for-customer-service-automation) we wrote that “BetterCX responds exclusively based on materials uploaded by the client,” and that when information is missing, it “states this directly and asks the user to contact the company.” The person decides something earlier than approval here: what goes into the knowledge base. **The review after launch.** A mistake nobody reads lasts for months. In our text on [hallucinations and AI agents](https://appwave.dev/en/blog/ai-agent-tools-three-things-nobody-mentions) we write: “Somebody reads the conversations in month three and calls before your customer does.” In the [agent that checks homework](https://appwave.dev/en/portfolio/an-ai-agent-that-checks-math-and-physics-homework-for-the-high-school-graduates-school) we set “a quarterly review of the AI module’s performance,” while “contact with students and instructional decisions remain the responsibility of the school.” ## What a person in the loop costs: 2-5 hours a month Take the company from our calculation of agent costs: about 200 repetitive inquiries a month and about 30 hours of work by the person who answers them. That is 9 minutes per case. The agent takes over 70-80 percent of inquiries, so 140-160 cases, and gives back about 20 hours. The numbers come from our text [AI agent: how much does it cost, and when does it really pay for itself](https://appwave.dev/en/blog/ai-agent-how-much-does-it-cost-and-when-does-it-really-pay-for-itself) of 20 August 2026. If a person approves every draft and spends 1-2 minutes on it, that comes to 2-5 hours a month \[estimate: 1-2 minutes per draft is an AppWave assumption, not a measurement\]. Approving everything therefore eats between a tenth and a quarter of the hours the agent was meant to give back. At 60 PLN an hour that is 120-300 PLN a month. Little, measured against the risk. A lot, if you pay it for a year. Two costs do not show up in the hours. The first is delay: a reply approved by hand waits for a person, so the agent stops answering at 10 p.m., and that was one of the reasons you bought it. The second is habit. Someone who approves the hundred and fiftieth draft of the month easily stops reading it, and from then on oversight exists only in the documentation. That is a hypothesis, not a measurement. So how do you loosen it? Count how many drafts you correct, separately for each category of question. A category where you have corrected almost nothing for two weeks moves to the threshold mode. That is our working rule, not an industry standard. Complaints, personal data and money stay on approval. The first 2-4 weeks are calibration, so do not judge the settings on day three. There is also a conclusion that is awkward for us. Below roughly a hundred inquiries a month the agent has little to take over, so there is little to oversee. In that case we do not build an agent. ## What the EU AI Act asks of human oversight, and what it does not **Probable conclusion.** Article 14 of the AI Act requires high-risk systems to be designed so that people can effectively oversee them. A typical customer service chatbot is usually not a high-risk system, so that article does not cover it. For a chatbot, the transparency duty in Article 50 matters, and it applies from 2 August 2026. We cover it separately in [Chatbot and the EU AI Act: what to check before 28 October 2026](https://appwave.dev/en/blog/chatbot-ai-act-checklist-before-28-october-2026). Regulation 2026/1744, in force since 27 July 2026, moved the high-risk dates: duties for the systems in Annex III start on 2 December 2027, and for the systems in Annex I on 2 August 2028. If your agent assesses job candidates or creditworthiness, that is a different conversation and you need a lawyer. We are not a law firm, and this text is not legal advice. ## How it looks in the tool that launched on 29 September The settings of dots, the background agents in ChatGPT, contain a similar split. According to the OpenAI Help Center, you set the rule for each action to one of four behaviours: take action without asking, take action if pre-approved, ask before taking action, or hand off to you. “Pre-approved” means that you yourself requested the action in the prompt. The first behaviour is work without approval, the second is consent given up front in the prompt, the third is approval for each action, and the fourth is a person taking over the case. It is a similar split to the one above, only in an interface. Who can switch dots on in Europe, and what to set first, we describe in [OpenAI dots in Europe: who can turn them on and what to set first](https://appwave.dev/en/blog/openai-dots-in-europe-availability-and-first-settings). ## What happens when you contact us You talk to a person who knows the rollout from the inside, not to a salesperson. The call lasts 45 minutes. At the end you get a concrete idea and a price range, and if your volume does not justify an agent, we will say so plainly. We build with you, not for you: the handoff threshold and the list of cases the agent does not touch we set together, on your earlier conversations with customers. We start with one process, and the build itself is described on our [AI agents service](https://appwave.dev/en/services/ai-agents) page. A team subscription starts at 3,875 PLN net a month, and you can work it out in the [calculator](https://appwave.dev/en/calculator) before the call. We don’t leave you with a finished product. We stay when you use it. We do not promise that oversight will remove mistakes. We promise to show you where a person should stop them, and to say so plainly if an agent makes no sense for you. [Book a 45-minute call](https://appwave.dev/en/appointment). ## Sources All pages accessed on 5 October 2026. - IBM Think, [What is human in the loop (HITL)?](https://www.ibm.com/think/topics/human-in-the-loop), definition of HITL. - Joe McKendrick, Forbes, 29 September 2026, marked as sponsored content: [Take Humans Out Of The AI Loop And Put Them At The Helm](https://www.forbes.com/sites/joemckendrick/2026/09/29/take-humans-out-of-the-ai-loop-and-put-them-at-the-helm/). - OpenAI, [Getting started with your dot](https://help.openai.com/articles/20001530), Help Center, section on rules and approvals. - Regulation (EU) 2024/1689 (AI Act), Articles 14 and 50, [EUR-Lex](https://eur-lex.europa.eu/eli/reg/2024/1689/oj). - Regulation (EU) 2026/1744 (Digital Omnibus on AI), in force from 27 July 2026, [EUR-Lex](https://eur-lex.europa.eu/eli/reg/2026/1744/oj). - AppWave, [AI Agent: How Much Does It Cost, and When Does It Really Pay for Itself?](https://appwave.dev/en/blog/ai-agent-how-much-does-it-cost-and-when-does-it-really-pay-for-itself), 20 August 2026. - AppWave, [Customer service automation that doesn’t turn your company into a call center](https://appwave.dev/en/blog/customer-service-automation-that-doesn-t-turn-your-company-into-a-call-center), 20 August 2026. - AppWave, [Hallucinations are the smallest of three problems with AI agents](https://appwave.dev/en/blog/ai-agent-tools-three-things-nobody-mentions), 20 August 2026. - AppWave, [BetterCX - our proprietary SaaS platform for customer service automation](https://appwave.dev/en/portfolio/bettercx-saas-platform-for-customer-service-automation), 20 August 2026, and the [homework-checking agent](https://appwave.dev/en/portfolio/an-ai-agent-that-checks-math-and-physics-homework-for-the-high-school-graduates-school), 25 August 2026. ## Questions & Answers (FAQ) ### What is human in the loop in AI? It is a setup in which a person takes an active part in how an automated system runs, supervises it or makes decisions in it. In customer service it means the AI agent prepares a reply, and a person approves it, corrects it or takes over the case before the customer sees anything. ### What is the difference between human in the loop and human on the loop? With human in the loop, a person approves the reply before it is sent. In our split, human on the loop means the agent replies alone, and a person only decides cases below a set confidence threshold or reviews conversations afterwards. The second mode takes less time but depends on a well-set threshold. ### How much time does approving agent replies take? At 200 inquiries a month, of which the agent takes over 70-80 percent, approving drafts takes about 2-5 hours a month, assuming 1-2 minutes per draft. That is an AppWave estimate, not a measurement. At higher volume the time grows in proportion, unless you approve only the risky cases. ### Does the EU AI Act require human in the loop in a customer service chatbot? The human oversight duty in Article 14 of the AI Act applies to high-risk systems, and an ordinary customer service chatbot usually is not one. A chatbot is covered by the duty to tell the customer they are talking to AI. A lawyer, not this text, decides how a specific system is classified. ### When can you stop approving every agent reply? When, after several weeks of calibration, you correct almost nothing in a given category of questions. That category then moves to the threshold mode, and a person stays on cases involving money, personal data and complaints. Categories where you still correct things stay on approval. --- ## About AppWave AI development company from Łódź, Poland. We build custom AI software, automation agents and web applications. We co-create them with the client and guarantee every system we deliver. - **Legal name:** AppWave sp. z o.o. - **Address:** ul. Kolumny 147E/1, 93-611 Łódź, Poland - **Phone:** +48 538 441 413 - **Email:** office@appwave.dev - **Business hours:** Monday-Friday, 09:00-17:00 - [Free consultation](https://appwave.dev/en/appointment)