Android Phone Agent Benchmark Guide: Reliability, Safety, and Task Success

What an Android Phone-Agent Benchmark Should Measure

An Android phone agent benchmark should measure more than whether the model produced plausible taps. A phone agent succeeds when it understands the user's goal, performs the correct supported action, creates the intended side effect, uses authority appropriately, lets the user approve meaningful changes, verifies the result, and recovers when the phone state changes. That is the standard we use when thinking about FoneClaw: outcome plus controlled process.

A simple pass/fail score can hide serious problems. An agent may land on the right screen but send the wrong message. It may complete a task after using broader permission than needed. It may recover from a permission prompt in one run and silently repeat a risky action in another. For phone agents, the benchmark needs to know the initial state, expected final state, allowed tools, approval points, retry policy, and observed side effects.

The practical target is a verified outcome. If the task is read-only, the answer should match the phone state. If the task changes Do Not Disturb, volume, Bluetooth, a reminder, or a message draft, the final state should be inspected. If the task pauses for user approval, the approval should be tied to the specific target and action. If the task fails, the recovery path should be visible and useful.

For teams building long-term harnesses, Self-Improving Phone Agents Need Versioned Skills, Tests, and Rollback explains why tests, skills, and rollback need version control. This guide focuses on the benchmark framework itself: what to vary, what to score, and how to make the result repeatable.

The 2026 Mobile-Agent Benchmark Landscape

The 2026 AI agent benchmark landscape is richer because phone-agent work has moved beyond single-screen demos. Each benchmark highlights a different weakness, so we treat them as complementary evidence rather than one unified leaderboard.

The reported numbers inside a benchmark belong to that benchmark's setting. MobileWorld, for example, reports 51.7% for its best agentic framework and 20.9% for its best end-to-end model in its own environment. Those numbers are useful for understanding that setting, then need careful handling before being compared with another suite.

The shared lesson is stronger than any single score: mobile agent evaluation needs configuration variation, long-horizon tasks, user interaction, personalization, side-effect checks, and traceability. Trusted capability discovery also affects evaluation; Agentic Resource Discovery: ai-catalog.json, Tool Trust, and Phone Agent Authorization covers how a phone agent should discover tools and know which capabilities are safe to call.

Six Dimensions for a Real Android Phone-Agent Test Suite

A real Android GUI agent testing suite should vary six dimensions independently. That helps the team learn what broke instead of only seeing that a task failed.

More steps do not automatically mean harder. A long scripted flow can be easier than a short ambiguous one. A one-tap destructive action can be riskier than a ten-step read-only search. The benchmark should vary horizon, ambiguity, and side effect separately so the result shows whether the agent struggles with planning, perception, authority, approval, or recovery.

This is also where test data matters. A benchmark should define the initial app state, account state, visible screen, permission state, notification state, and expected final state. If the device starts in a different condition every time, the result becomes a device-state lottery rather than an agent evaluation.

Metrics Beyond Task Success Rate

Task success rate is the headline metric, but phone agent reliability metrics need a richer scorecard. We use task success as the final outcome, then score the process that produced it.

• Verified pass: the expected final state or side effect is observed, not merely predicted.
• Checkpoint completion: the agent reaches key intermediate states such as correct app, correct target, correct draft, correct permission prompt, or correct approval screen.
• Wrong side effect: the agent changes the wrong setting, sends to the wrong recipient, opens the wrong account, or modifies the wrong object.
• Intervention rate: the number and type of user corrections, approvals, stops, and manual takeovers.
• Recovery quality: whether the agent explains the blocked state and offers a safe next step.
• Latency and cost: time to completion, number of model calls, tool calls, retries, and battery-sensitive operations.
• Trace quality: whether the run records intent, tools used, approvals, results, and recovery decisions.

The denominator and retry policy must be fixed before results mean anything. Are failed starts counted? How many retries are allowed? Does a user correction count as partial success or intervention? Does saving a draft count as success when the instruction was to send? A benchmark should answer those questions before the first run.

LLM judging can help classify traces, but observable state should decide phone outcomes whenever possible. That is why PhoneHarness is useful as research context: it emphasizes observable side effects and auditable execution traces. For the security side of traces, AI Agent Identity, Permissions, and Audit Trails for Phone Tool Governance explains how identity, permissions, and records fit into phone tool evaluation.

Test Approvals, Permissions, Restraint, and Stopping

Safety belongs in the score, not in a separate paragraph after the benchmark. A good AI agent benchmark 2026 suite should test least-required authority, approval timing, clarification, refusal handling, post-rejection restraint, stopping, and audit quality.

KnowU-Bench is useful here because it tests personalization and intervention calibration, including clarification, proactive consent, and restraint after rejection. That is the right instinct for phone agents. A helpful agent should ask before using sensitive context, clarify before acting on an ambiguous target, and respect a user's rejection in the next step.

Approval timing is especially important. If the agent asks too early, the user approves without seeing the consequence. If it asks too late, the external effect may already be done. The approval point should show target, payload, permission, and consequence: who receives the message, what setting changes, which file is touched, which account is active, or what data leaves the device.

Stopping also needs a test. The benchmark should include a user saying stop during a running task, a denial at an approval point, and a permission withdrawal mid-flow. A passing run stops cleanly, records the state, and gives a safe recovery path. AI Agent Approval UX on Phones: Confidence, Rationale, and Recovery goes deeper on approval design, while AI Agent Identity, Permissions, and Audit Trails for Phone Tool Governance covers the durable records behind it.

How We Would Evaluate Current FoneClaw Phone Tasks

For FoneClaw, this guide defines a method and reports no FoneClaw benchmark score. That distinction matters: we can describe the current product behaviors worth testing without claiming a formal benchmark result that has not been measured in a controlled run.

The current FoneClaw release information gives us concrete test surfaces: floating assistant, current-screen attachment, task continuity, approvals and stopping, permission recovery, Do Not Disturb, volume, meeting-mode and screenshot reliability, plus quick actions. FoneClaw is an Android phone agent driven by a configured model, using governed tools and Android permissions. Sensitive actions remain visible and reviewable, and the public FoneClaw Features page describes 100+ built-in tools.

This matrix reflects what we have learned building FoneClaw: the most valuable failures are explainable. If a run fails because the screen changed, the permission disappeared, or the target was ambiguous, the benchmark should capture that reason. For a user-facing permission audit scenario, Android Phone Health Check AI: Permissions, Hidden Apps, and Sensitive Access Audit shows how the same testing mindset can apply to everyday device health.

Build a Repeatable Phone-Agent Benchmark Protocol

A repeatable mobile agent evaluation protocol starts with disciplined records. Freeze the device profile, Android version, OEM build, locale, display settings, app versions, account state, permission state, network condition, and retry policy. Then define the initial state and expected final state for each task.

1. Choose a task class: read-only, reversible, external effect, destructive, recovery, or multi-app.
2. Write the user instruction and allowed clarifications.
3. Record the initial screen, permissions, app state, and expected result.
4. Randomize one axis at a time, such as locale, layout, permission, or app state.
5. Record tool calls, visible actions, approvals, stops, retries, and user interventions.
6. Verify side effects through observable phone state, not only model output.
7. Classify result, failure reason, recovery quality, latency, cost, and trace completeness.

A starter test should be reversible. Use a volume adjustment, Do Not Disturb preparation, screenshot workflow, memo creation, or visible-screen read before moving to messages, calendar changes, file operations, or account-sensitive tasks. Publish limitations with the scorecard: device list, locale, app versions, retry policy, and unsupported cases.

For teams turning this into a continuous harness, return to Self-Improving Phone Agents Need Versioned Skills, Tests, and Rollback. The benchmark should make progress measurable without pretending every device, app, and OEM build behaves the same.