The Android developer community is witnessing a significant evolution in mobile application capabilities, with a renewed focus on empowering developers to create truly personalized, intelligent, and agentic user experiences. This shift is exemplified by the ongoing "Build intelligent Android apps" blog post series, which systematically transforms a basic Android application into a sophisticated AI-driven tool. Following an introductory post that unveiled "Jetpacker," the demo application serving as the series’ cornerstone, the latest installment delves into the practical implementation of on-device intelligence using Google’s Gemini Nano and the robust ML Kit Prompt API. This article explores how developers can harness these cutting-edge technologies to imbue their Android applications with advanced, privacy-centric features that operate directly on the user’s device.
The Power of On-Device Processing for Enhanced User Experiences
At its core, building intelligent on-device features signifies the capacity of an application to process prompts and data locally, bypassing the need for constant data transmission to external servers. This architectural approach offers a compelling suite of advantages that directly translate to a superior user experience. Firstly, enhanced privacy and security are paramount. By keeping sensitive user data on the device, the risk of data breaches during transit or from cloud storage is significantly mitigated, fostering greater user trust. Secondly, reduced latency is a critical benefit. Eliminating the round trip to a server means that AI-powered responses are delivered almost instantaneously, crucial for real-time interactions and applications demanding immediate feedback. Thirdly, offline functionality becomes a reality. Applications can continue to provide intelligent features even in environments with limited or no internet connectivity, expanding their usability and reach. Finally, cost efficiency is a tangible advantage for developers. Reducing reliance on cloud-based AI services can lead to substantial savings on server infrastructure and API usage fees.
Recognizing these benefits, the Jetpacker application has been enhanced with three key on-device features designed to elevate user engagement: the intelligent summarization of trip itineraries, streamlined expense management, and efficient voice note capture. These features demonstrate the tangible impact of integrating advanced AI models directly into the mobile ecosystem.
Intelligent Itinerary Summarization: From Overload to Insight
Travel planning often involves navigating a wealth of information, from flight schedules and hotel bookings to daily activities and local attractions. The itinerary screen within the Jetpacker app, designed to provide a comprehensive overview of a trip, can quickly become overwhelming for users. To address this, a new "Get ready for your trip" section has been introduced, offering a concise and actionable summary powered by AI.
This feature leverages Gemini Nano, Google’s most efficient AI model specifically optimized for mobile devices. First introduced several years ago, Gemini Nano has since seen widespread adoption, now operating on over 140 million devices globally. The latest iteration, Gemini Nano 4, builds upon the architectural foundation of the recently released Gemma 4 model, further refining its capabilities for maximum battery and performance efficiency.
The implementation utilizes ML Kit’s Prompt API, allowing developers to harness Gemini Nano 4’s advanced model capabilities for prototyping on-device features. The process involves crafting a prompt that feeds the trip itinerary to the model, requesting it to generate a personalized summary, along with practical preparation tips and relevant local phrases.
A typical prompt structure might look like this:
// Dependency declaration for ML Kit's GenAI Prompt API
// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
// Configuration for the Gemini Nano 4 E2B preview model, prioritizing speed.
val previewFastConfig = generationConfig
modelConfig = modelConfig
releaseStage = ModelReleaseStage.PREVIEW
preference = ModelPreference.FAST
// Obtain a client for the Gemini Nano 4 preview model.
val geminiNano2BPreviewModel = Generation.getClient(previewFastConfig)
// Placeholder for the trip itinerary data.
val tripItinerary = "Day 1: Arrive in Paris, check into hotel. Evening Eiffel Tower visit. Day 2: Louvre Museum, Notre Dame Cathedral. Day 3: Montmartre, Sacré-Cœur Basilica. Depart in the evening."
// Constructing the prompt to generate a summary, preparation tips, and useful phrases.
val getReadyForYourTripSummary = geminiNano2BPreviewModel
.generateContent("Given this trip itinerary: $tripItinerary,
generate the following: overall vibe, tips on how to prepare for this
trip, and common short phrases to learn for the trip.")The iteration process for finding the optimal prompt is crucial for achieving desired results. Tools like the AICore app, which allows developers to opt into a developer preview and download preview models like Gemini Nano 4, are invaluable for testing prompts and observing model outputs. Through diligent refinement, the response time for itinerary summarization in Jetpacker was dramatically reduced from an initial 13 seconds to under two seconds, a testament to the efficiency gains achievable with on-device AI. The final code implementation and prompt details can be found in the Jetpacker repository on GitHub.
Local Processing for Sensitive User Input: The Expense Manager
Enhancing the travel experience further involves simplifying the often tedious task of managing expenses. The Jetpacker app now features a simple expense manager that aims to automate the process of sorting receipts and calculating budgets. This is particularly relevant given that receipts can contain sensitive personal information, such as credit card numbers and addresses.
The decision to implement this feature on-device is driven by the need to safeguard user privacy. By processing receipt data locally, users can be assured that their sensitive financial information remains on their device and is not transmitted to any external servers. This adherence to privacy-by-design principles is a cornerstone of modern mobile application development.
Furthermore, Gemini Nano 4’s enhanced multimodal capabilities, especially in image understanding tasks like Optical Character Recognition (OCR) and visual data extraction, make it an ideal solution for extracting information from receipts. For this use case, the prompt is designed to analyze an image of a receipt and extract key details such as a generated title, the total amount spent, and the category of the expense.
To ensure the model outputs this information in a structured and easily parsable format, ML Kit’s Structured Output API is employed. This allows for the seamless generation of a Kotlin data object, pre-defined by the developer.
The structured output definition might appear as follows:
// Dependencies for ML Kit's GenAI Prompt and Schema Compiler
// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
// ksp("com.google.mlkit:genai-schema-compiler:1.0.0-alpha1")
// Data class defining the expected output structure for parsed receipt information.
@Generable("Information extracted from an expense receipt")
data class ParsedReceipt(
@Guide("Generated title for the expense less than 6 words. Based on restaurant or activity name.")
val String,
@Guide("Total amount of the expense. Look for values at the bottom and words like total or balance due.")
val amount: Double,
@Guide("Type of expense", enumValues = ["travel", "food", "shopping", "entertainment", "other"])
val category: String,
)
// Initial prompt to guide the model in identifying and parsing receipts.
val prompt = "Determine if the image is a receipt or expense.
If it is NOT a receipt or expense, output the text 'NOT_A_RECEIPT'.
Otherwise, parse the receipt information."
// Constructing the request with the image and initial prompt.
val request = generateContentRequest(ImagePart(bitmap), TextPart(prompt))
// Requesting structured output as a ParsedReceipt object.
val requestWithStructuredOutput = generateTypedContentRequest(request, ParsedReceipt::class)
// Configuration for the Gemini Nano 4 E4B preview model, prioritizing reasoning power.
val previewFullConfig = generationConfig
modelConfig = modelConfig
releaseStage = ModelReleaseStage.PREVIEW
preference = ModelPreference.FULL // Prioritizing reasoning power over speed.
// Obtaining a client for the Gemini Nano 4 preview model.
val geminiNano4BPreviewModel = Generation.getClient(previewFullConfig)
// Generating content and receiving the parsed receipt data.
val response = geminiNano4BPreviewModel.generateContent(requestWithStructuredOutput)
val parsedReceipt: ParsedReceipt? = response.candidates.firstOrNull()?.responseThis approach not only ensures data privacy but also significantly simplifies the development workflow by providing structured data directly from the AI model, eliminating the need for complex post-processing logic.
Multimodal Input: Seamless Voice Notes with On-Device Transcription
The final on-device intelligent feature introduced in Jetpacker addresses the need for users to easily record audio memos during their travels. This fully on-device voice notes feature utilizes ML Kit’s Speech Recognition API to automatically transcribe spoken words into text. With the transcribed text, the app then leverages ML Kit’s Prompt API to identify which specific trip activity the voice note is associated with, allowing users to effortlessly recap their journey as they review their itinerary.
The ML Kit GenAI Speech Recognition API offers two distinct modes for on-device transcription. The Basic mode, available on most Android devices with API level 31 and higher, employs a traditional on-device speech recognition model. The Advanced mode, however, utilizes Gemini Nano to provide broader language coverage and enhanced transcription quality, currently supported on Pixel 10 devices.
The implementation elegantly combines the Speech Recognition API with the ML Kit GenAI Prompt API:
// Dependencies for ML Kit's GenAI Prompt and Speech Recognition APIs.
// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
// implementation("com.google.mlkit:genai-speech-recognition:1.0.0-alpha1")
// Placeholder for trip event data.
val tripEvents = listOf("Visit the Eiffel Tower", "Explore the Louvre Museum", "Dinner in Montmartre")
// Setting up speech recognition options, prioritizing advanced mode and US locale.
val speechRecognizerOptions =
speechRecognizerOptions
locale = Locale.US
preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
// Obtaining a client for the speech recognizer.
val speechRecognizer: SpeechRecognizer = SpeechRecognition.getClient(speechRecognizerOptions)
// Coroutine function to handle voice note transcription and processing.
suspend fun transcribeVoiceNote(recognizer: SpeechRecognizer)
// Variables to store partial and final transcription results.
var partialTextResponse = ""
var transcription = ""
// Creating a speech recognition request from the microphone.
val request: SpeechRecognizerRequest
= speechRecognizerRequest audioSource = AudioSource.fromMic()
// Starting the recognition process and collecting responses.
recognizer.startRecognition(request).collect response ->
when (response)
is SpeechRecognizerResponse.PartialTextResponse ->
// Update UI with partial transcription as the user speaks.
partialTextResponse = response.text
is SpeechRecognizerResponse.FinalTextResponse ->
// Final transcription received.
transcription = response.text
// Process the transcribed voice note and categorize it.
processAndCategorizeVoiceNote(transcription, tripEvents)
// Function to process the transcribed voice note and link it to trip events.
fun processAndCategorizeVoiceNote(transcribedVoiceNote: String, events: List<Event>)
// Prompt to clean up transcription and match it with trip events.
val prompt = "Given the voice note $transcribedVoiceNote
and the following events for this trip: $events, rewrite this transcription
to remove filler words. Then, identify which events from the
list this rewritten transcription matches to."
// Utilizing ML Kit's Prompt API to process the voice note and tag it with relevant trip activities.
Generation.getClient().generateContent(prompt)
This integration allows for a seamless user experience, where spoken notes are not only transcribed accurately but also contextually linked to specific trip activities, enhancing the overall organization and recall of travel memories.
Conclusion: A New Era of On-Device Intelligence
The advancements demonstrated in the Jetpacker application underscore a significant shift in Android development, empowering developers to build sophisticated, on-device intelligent features. By leveraging ML Kit’s GenAI APIs in conjunction with Gemini Nano, developers can create personalized and agentic experiences that prioritize user privacy, offer reduced latency, and function effectively offline, all without incurring additional cloud infrastructure costs.
The "Build intelligent Android apps" series continues to explore this transformative landscape. Following this deep dive into on-device intelligence, subsequent posts will venture into hybrid and cloud reasoning with Firebase AI Logic, system integration with AppFunctions, and the development of in-app agentic workflows, promising a comprehensive guide to building the next generation of intelligent Android applications.
The full source code for the Jetpacker application is available on GitHub, offering developers a practical resource to explore and implement these cutting-edge AI capabilities. The accompanying video, "Build Intelligent Android apps with Google’s AI," provides further insights into integrating on-device models, cloud-powered reasoning, and agentic frameworks into your own applications.
Learn More:
This article is part of a series exploring the development of intelligent Android applications:
- Part 1: Introduction of the app and a high-level overview.
- Part 2 (This post): On-device intelligence. A deep dive into ML Kit’s GenAI APIs and Gemini Nano for building privacy-first features like itinerary summarization, receipt parsing, and local audio processing.
- Part 3: Hybrid and cloud reasoning. Exploring how to use Firebase AI Logic to ground LLM answers in real-world data like Google Maps and web context.
- Part 4: System integration. Integrating with the Android intelligence system using AppFunctions.
- Part 5 (Coming soon): In-app agentic workflows. Extending the app with an end-to-end booking assistant powered by A2UI and ADK.
For further exploration into Android Development, follow Android Developers on YouTube or LinkedIn.
Copyright 2026 Google LLC. SPDX-License-Identifier: Apache-2.0
