A phone-powered companion for Sphero RVR and Arduino Uno: expressive pixel eyes, local conversation, familiar faces and a rather enthusiastic interest in shoes.
The phone supplies the screen, cameras, microphone, speaker, motion sensor and AI. The Uno reads the robot's extra sensors and enforces movement limits. The RVR supplies the wheels and robot telemetry.
When connected, landscape mode shows two animated digital eyes on black. First connection asks for the robot's name, repeats it for confirmation, then remembers it. Traits such as bravery, kindness, cheekiness and curiosity change with interactions. Named relationships, fondness and remembered facts influence later replies.
The robot can greet people, approach a visible face in short checked steps, refuse cheekily, and request help for low battery, a possible pickup or being stuck. These are supervised behaviours. This release does not map rooms, find a charging dock or navigate a house independently.
There is no Samsung-only code or Galaxy AI dependency. The S24 Ultra is the intended first physical test phone, not a requirement.
| Requirement | What to check |
|---|---|
| Operating system | Android 8.0 or later. The app targets Android 16. It is an Android APK, so it does not install on iPhone. |
| Processor and OS | A 64-bit ARM Android system, arm64-v8a. A 64-bit chip running a 32-bit version of Android is insufficient. The APK also includes x86_64 for testing. |
| Robot connection | USB host / OTG support and a data-capable connection to the Uno. USB-C alone does not guarantee USB host support. Android USB host reference. |
| Camera and audio | A working camera, microphone and speaker, with camera and microphone permissions allowed. Front and rear cameras can be selected. The app falls back if the selected camera is absent. |
| Speech | Offline speech recognition must be available from the phone's installed recognizer. Android 12 or newer exposes an explicit on-device recognition check, but support and language packs still vary. Typed conversation remains available. |
| Storage and memory | The APK bundles three models, and the language model is copied to private storage on first launch. Allow roughly 2 GB of free storage for download, installation and that copy. RAM and processor speed affect responsiveness; no minimum-RAM phone has yet been certified. |
| Physical fit | The phone needs a secure landscape mount, an unobstructed forward-facing camera, speaker and microphone, and a practical charging arrangement. Phone weight changes drive power and stability. |
Before buying a dedicated phone: the practical compatibility test is local model startup, offline listening, camera recognition and USB communication together. A cheaper Android phone may work, but no particular budget model has been verified yet. Simultaneous phone charging and USB host operation depends on the phone and adapter.
Hugging Face's 360 million parameter instruction model, in Unsloth's Q4_K_M GGUF format. The compressed model is about 258 MiB and runs through llama.cpp on the CPU.
It receives the robot's traits, current person, recent memories and camera labels. It generates short character dialogue. It does not receive raw camera images or issue motor commands.
OpenCV's SFace 2021 December model produces a 128-number face descriptor. ML Kit finds the face and landmarks first. The app aligns the face to a 112 × 112 image, then runs SFace with ONNX Runtime.
The model file is 38.7 MB. Face descriptors are compared with examples saved for familiar people.
Google MediaPipe's float32 image embedding model, version 1, produces a 1,024-number image descriptor. It runs through LiteRT, with a 224 × 224 input. Its file is 4.1 MB.
This is a general visual similarity model. It compares saved shoe appearances; it is not a biometric shoe identity model. ML Kit first checks that a crop appears to contain footwear.
Personality develops through saved traits and relationship history. The app does not retrain model weights. Behaviour rules handle names, emergency commands, familiar greetings and help requests.
Speech uses Android's installed offline text-to-speech voices. Naturalness depends on the selected voice pack. There is no bundled neural voice model, voice cloning or paid speech API.
The language model uses a 2,048-token context, up to 100 generated tokens per reply and a 45-second generation limit. Its small size makes local operation practical, but replies can be repetitive or imperfect. No model API key or AI subscription is required.
Open Faces and shoes in the caretaker screen. Select or add a person, then teach their face or shoes. The preview shows what the model is analysing. Save three slightly different clear views, with that person's agreement. Shoes can have a label such as “blue trainers”.
A single sufficiently large, near-front-facing face is aligned and compared with stored examples. A match must exceed the face threshold, beat another enrolled person by a margin, and repeat over three observations before it can trigger a named greeting.
Unknown, poorly visible and ambiguous faces remain unnamed. Multiple visible faces suppress automatic identification for that frame.
When no face is visible, the app checks three nearby square regions for footwear and compares their appearance with saved shoe examples. Conflicting matches to different people are discarded.
A repeated shoe match asks for confirmation before attaching that person's relationship or facts to the conversation. A shoe hint never initiates an approach to feet.
“Those shoes look familiar! Is that you, Lia? Tiny shoe detective needs confirmation!”
Face recognition can be disabled under Voice and camera. Individual face or shoe examples can be deleted from Faces and shoes. Forgetting a person also deletes their visual examples.
Descriptors use cosine similarity, which is not a probability. The best two enrolled views must agree. Initial thresholds are 0.60 for faces and 0.92 for shoes; the required margins over another enrolled person are 0.12 and 0.045 respectively. Three successive matching observations are required. These conservative initial settings still need trials with the actual people, shoes, lighting and camera angle.
The gallery holds up to 30 people, with up to nine face views and 12 shoe views per person. Older examples are replaced as new ones are added. Appearance matching is not authentication or a liveness check: a photograph can resemble an enrolled person. Different people may share shoes, and unfamiliar shoes cannot identify their wearer.
| Layer | Language and tools |
|---|---|
| Android app | Kotlin, native Android views and a custom Canvas for the eyes. Kotlin 2.2.21, Android Gradle Plugin 8.13.2 and Gradle 8.14.3. |
| Language inference | C++17 through JNI and llama.cpp. Built with Android NDK 28.2.13676358 and CMake 3.22.1. No WebView or Flutter layer. |
| Uno firmware | Arduino C++ with SoftwareSerial and a fresh implementation of the published Sphero packet framing. The old owner's code was used only as wiring information. |
| Memory | Private Android SharedPreferences containing bounded JSON records. Visual descriptors live separately from conversation and personality records. |
| Build and checks | PowerShell and Python for asset preparation and device checks, JUnit for behaviour tests, Android instrumentation for actual inference, and a C++ harness for the firmware. |
| This document | Standalone HTML and CSS, with a small print button. No external fonts, analytics or JavaScript libraries. It can be saved and read offline. |
| API / dependency | Purpose |
|---|---|
| AndroidX Activity 1.10.1 Lifecycle 2.9.1 | Native activity, permissions, lifecycle and background cleanup. |
| CameraX 1.4.2 | Camera2 capture, image analysis, rotation and lifecycle binding. Only the selected camera is active. |
| ML Kit face detection 16.1.7 | Bundled face detection and facial landmarks. Identity comes from SFace, not the detector's tracking ID. |
| ML Kit image labelling 17.0.9 | Bundled object labels and a footwear check for shoe crops. These are uncertain labels, not full visual understanding. |
| ONNX Runtime Android 1.23.2 | Local CPU inference for SFace, using its Java API from Kotlin. |
| LiteRT 1.4.1 | Local CPU inference for shoe image descriptors through the TensorFlow Lite Interpreter API. |
| Android SpeechRecognizer | Microphone speech recognition. Uses on-device recognition when available. The optional installed system service may use its provider's network. |
| Android TextToSpeech | Speech output through an installed offline voice, with listening paused during speech to reduce self-triggering. |
| USB serial for Android 3.11.0 | USB serial communication with the Uno, including supported standard and clone USB serial chips. |
| SensorManager / BatteryManager | Phone acceleration and orientation changes, plus phone battery level. |
| Sphero UART protocol | Wake, motor commands, acknowledgement, robot battery queries and stall notifications. No cloud robot API. |
| Firebase Hosting | Hosts this page, the public APK and QR code. The installed robot app does not connect to Firebase. |
The phone communicates with the Uno at 9,600 baud over USB. The Uno communicates with the RVR at 115,200 baud over UART. Matching Little Rover Uno firmware is required; the 0.1.0 firmware remains compatible with app 0.2.0.
| Uno connection | Role from the supplied wiring map |
|---|---|
| D2 / D3 | SoftwareSerial RX / TX for the RVR link. |
| D4 | Cliff sensor digital input. |
| D5 / D6 | Ultrasonic trigger / echo for forward distance. |
| D7 | Bumper input. |
| D8 / D9 | Existing sensor module power controls. Actual module current and power wiring need checking. |
The RVR UART uses 3.3 V logic; the Uno uses 5 V logic. The actual interface needs suitable voltage conversion and a common ground. The supplied wiring map is not proof that the existing level conversion or sensor power arrangement is correct.
The Uno requires calibrated sensors and fresh robot replies before arming. Motor pulses are bounded to 350 ms, phone heartbeat loss stops motion after 700 ms, and missing or invalid sensor readings block movement. There is no rear range sensor, so autonomous blind reversing is disabled. The language model cannot override these rules.
Pickup detection combines phone motion and tilt clues. Stuck detection uses RVR stall notifications and lack of forward range progress. Gentle, level lifting can be missed. Camera loss, app backgrounding, stop commands and session shutdown cancel movement.
Robot name, personality traits, named relationships, remembered facts, recent conversation episodes and enrolled visual descriptors stay in the app's private storage. Camera frames and enrolment preview photos are not saved. A descriptor is still sensitive information about a familiar person's appearance, even though it is not a photograph.
Android cloud backup and device-transfer extraction are disabled. Installing an update with the same package and signing key preserves app data. Uninstalling, clearing app storage or switching to a different phone does not carry these memories over. There is no memory export or cross-phone sync in this release.
The app has no Internet permission, account, advertising or cloud AI dependency. Android speech services are separate: if you enable the optional system recognizer, that installed provider may send audio over its own network connection. Offline mode is the default. Model files are downloaded during development and bundled before distribution.
Version 0.2.0 passes 18 behaviour and safety tests, Android lint, actual language and vision model inference, and enrollment save-and-cancel checks in an isolated Android 15 emulator. APK signature and 16 KB native-library alignment checks also pass. Existing app memories survive the update. Physical testing with the S24 Ultra, RVR and Uno remains outstanding.
The phone compatibility requirements describe what the software is built to support. They do not certify every Android handset. Recognition checks use a small set of test images, not a real-world accuracy benchmark. Actual recognition quality, camera angle, heat, battery life, voice quality, USB reliability and motor response still need testing on the mounted phone.
Current scope: a personal, development-signed hardware test APK. Familiar face matching and shoe hints are character features. Room navigation, a charging dock, always-on background listening, face authentication, a natural neural voice pack and transferring memories between phones are not implemented.