AI Phone Assistant Brings Vision-Language Automation to LineageOS

Summary

AI Phone Assistant is a privileged system application for LineageOS/AOSP that turns a compatible Android phone into a task-driven, self-hosted AI agent. Built and tested with LineageOS 22.2 and Android 15, it captures the display, sends selected screenshots and the configured task to an OpenAI-compatible vision API, then injects the returned touch or key events. The app supports Android 13 through 15 and uses WindowManager.captureDisplay(), with a mirrored VirtualDisplay and ImageReader fallback. It does not analyze continuously without conditions: the assistant is disabled by default, requires the user to enable it and set a task, and only sends a frame after the screen settles and changes meaningfully, with rate limiting and a post-action cooldown. The model returns one JSON action per round, including tap, swipe, long press, scroll, key press, text input, wait, or none; coordinates from the downscaled image are rescaled to the physical display, and malformed or incomplete actions are ignored. Deployment requires a platform-signed privileged app in /system/system_ext/priv-app/, signature-level capture and input permissions, matching privapp permissions, and SELinux policy included in a system image. The repository provides a LineageOS build path for the POCO F2 Pro and a rooted-device APK deployment script. Users can configure hosted or local endpoints such as OpenAI-compatible servers, Zhipu, Ollama, or vLLM, and change the task or endpoint while the service runs. Important limitations remain: keyboard-based text input is limited to ASCII/Latin text, secure surfaces are black by default, and multi-step tasks require several capture, analysis, and action cycles. Because the system-UID app can read the screen and control the device, screenshots are sent to the configured endpoint and the API key is stored in plaintext SharedPreferences; the project recommends using a device the operator controls and preferably a LAN or self-hosted endpoint.