Build Intelligent Android Apps: Leveraging Gemini Nano and ML Kit for On-Device Personalization

The Android development landscape is undergoing a significant transformation, moving beyond mere functionality to embrace intelligent, personalized, and agentic user experiences. In a series of insightful blog posts, Google’s Android Developers team has been meticulously detailing this evolution, showcasing how to imbue basic Android applications with advanced AI capabilities. The latest installment delves into the practical application of on-device intelligence, demonstrating how developers can harness the power of Gemini Nano through ML Kit’s Prompt API to create sophisticated, privacy-centric features directly on user devices. This approach not only enhances user experience but also addresses critical concerns around data privacy and real-time processing.

The foundation of this intelligent transformation lies in the ability to process prompts and data locally, eliminating the need to send sensitive information to remote servers. This on-device processing offers a compelling array of advantages, including enhanced privacy and security, as user data remains within the device’s secure environment. Furthermore, it significantly reduces latency, enabling near-instantaneous responses and a more fluid user interaction. Offline functionality is another key benefit, ensuring that intelligent features remain accessible even without an active internet connection, a crucial consideration for travelers or users in areas with unreliable network coverage. Finally, on-device processing can lead to cost efficiencies by minimizing reliance on cloud-based AI services, which often incur usage-based charges.

Building upon the foundation laid in the introductory post of this series, which introduced "Jetpacker" – a demo application serving as the canvas for these AI integrations – this article focuses on three specific features designed to elevate the user experience within Jetpacker: summarizing trip itineraries, managing expenses, and capturing voice notes. These features, when implemented with on-device intelligence, promise to make travel planning and management significantly more intuitive and efficient.

Enhancing Trip Planning with Intelligent Itinerary Summarization

For travelers, a well-structured itinerary is essential, but an abundance of detailed information can often lead to an overwhelming user experience. The Jetpacker app addresses this challenge by introducing an intelligent "Get ready for your trip" section at the top of its itinerary screen. This feature leverages the power of Large Language Models (LLMs) to condense complex itineraries into easily digestible summaries.

By feeding a trip’s itinerary into an LLM, the application can generate a concise overview of the trip’s essence, coupled with practical advice such as packing tips and relevant local phrases. This is a prime candidate for on-device implementation due to several factors. Firstly, the summarization of text is a computationally feasible task for modern mobile processors. Secondly, and crucially, itineraries can often contain personal details, making on-device processing a significant privacy advantage. Users can be assured that their travel plans are not being transmitted to external servers for analysis.

The engine behind this intelligent summarization is Gemini Nano, Google’s most efficient AI model specifically engineered for mobile devices. Launched a few years prior, Gemini Nano has already seen widespread adoption, powering intelligent features on over 140 million devices. The latest iteration, Gemini Nano 4, builds upon the architectural advancements of the recently released Gemma 4 model, further optimizing for battery life and overall performance efficiency. This makes it an ideal choice for integrating advanced AI capabilities into mobile applications without compromising the user’s device performance or battery longevity.

Developers can harness Gemini Nano 4’s capabilities through ML Kit’s Prompt API. This API allows for the creation of prompts that instruct the model to perform specific tasks. For the itinerary summarization feature, a prompt is crafted to include the trip itinerary and request a summary, preparation tips, and useful phrases. The process involves defining a configuration for Gemini Nano 4, specifying parameters like the release stage and performance preference. The generationConfig block in the provided code snippet illustrates how to set up the model for preview, prioritizing speed with ModelPreference.FAST.

The crucial step of prompt engineering often requires iteration to achieve optimal results. Google’s AICore app, accessible via a developer preview, plays a vital role in this iterative process. By enabling the AICore developer preview, users can download and test preview models like Gemini Nano 4, allowing them to refine prompts and evaluate model outputs in real-time. Through such iterations, the development team successfully reduced the response time for itinerary summarization from an initial 13 seconds to under 2 seconds, demonstrating the impact of effective prompt design and the performance gains of on-device models. The full code implementation and optimized prompt for this feature are available for developers to explore in the Jetpacker GitHub repository.

Streamlining Expense Management with On-Device Data Extraction

The second intelligent feature introduced in Jetpacker is a simplified expense manager, aimed at automating the tedious process of sifting through receipts and calculating budgets. This feature addresses a common pain point for travelers and individuals alike, offering a more streamlined approach to financial tracking.

A key consideration for any expense management tool is the potential for sensitive data to be present on receipts, such as credit card numbers and personal addresses. This makes on-device processing a highly attractive solution, ensuring that such private information is handled locally and never leaves the user’s device. This aligns with the growing user demand for enhanced data privacy and security in their mobile applications.

Gemini Nano 4’s enhanced multimodal capabilities, particularly its proficiency in image understanding tasks like Optical Character Recognition (OCR) and visual data extraction, make it exceptionally well-suited for processing receipts. The model can accurately extract relevant information from images, such as the name of the establishment, the total amount spent, and the category of the expense.

To facilitate the seamless integration of extracted data into the application, ML Kit’s Structured Output API is employed. This API allows the LLM to output information directly into a defined Kotlin data object, eliminating the need for manual parsing of unstructured text. The ParsedReceipt data class, annotated with @Generable and @Guide to guide the model’s output, exemplifies this approach. The prompt instructs the model to identify receipts, extract key information, and categorize the expense, with specific guidance on the format and content of each field. The ModelPreference.FULL is utilized in this scenario, prioritizing the model’s reasoning power for accurate data extraction from potentially complex receipt images. The code snippet demonstrates how to create a generateContentRequest with an ImagePart and a TextPart for the prompt, then uses generateTypedContentRequest to specify the desired structured output.

Empowering Voice Notes with On-Device Transcription and Categorization

The final intelligent feature developed for Jetpacker is a fully on-device voice notes capability, designed to help users easily record and manage audio memos during their trips. This feature combines the power of ML Kit’s Speech Recognition API with the Prompt API to provide a seamless audio-to-text experience.

The ML Kit GenAI Speech Recognition API offers two distinct modes for on-device transcription. The Basic mode, available on most Android devices with API level 31 and higher, utilizes a traditional on-device speech recognition model. For enhanced performance, broader language coverage, and superior transcription quality, the Advanced mode leverages Gemini Nano, currently supported on Pixel 10 devices.

For the Jetpacker application, the Advanced mode is employed to ensure the highest quality transcription. The process begins with setting up the SpeechRecognizerOptions, specifying the locale and prioritizing the advanced mode. The SpeechRecognizer client is then initialized. The transcribeVoiceNote function uses a SpeechRecognizerRequest to capture audio from the device’s microphone. As the user speaks, the API provides real-time PartialTextResponse updates, offering a preview of the transcription. Upon completion of recording, a FinalTextResponse delivers the full transcribed text.

Following transcription, the system utilizes the ML Kit GenAI Prompt API to process the transcribed voice note. The prompt is designed to take the transcribed text and a list of trip events, then refine the transcription by removing filler words and, crucially, identify which of the provided trip events the voice note relates to. This allows users to easily associate their audio memos with specific activities within their itinerary, providing a contextualized recap of their trip. The code snippet illustrates how the processAndCategorizeVoiceNote function constructs a prompt that feeds both the transcribed text and the trip events to the LLM for analysis and categorization.

The Broader Impact of On-Device AI on Android Development

The successful integration of these three intelligent features into the Jetpacker app underscores a significant trend in Android development: the increasing feasibility and desirability of on-device AI. By leveraging Google’s cutting-edge AI models like Gemini Nano and the robust toolkit provided by ML Kit, developers can now build sophisticated, personalized, and agentic experiences that were once the exclusive domain of cloud-based systems.

The implications of this shift are far-reaching. For users, it means more private, responsive, and offline-capable applications. For developers, it opens up new avenues for innovation, enabling them to create richer user interactions without the complexities and costs associated with extensive cloud infrastructure. The privacy-first approach inherent in on-device processing is particularly timely, as users become increasingly aware of and concerned about their digital footprint.

This series of blog posts serves as a comprehensive guide, not just for Jetpacker, but for any developer looking to infuse their Android applications with intelligence. The progression from Part 1 (introduction to the app and high-level overview) to Part 2 (this deep dive into on-device intelligence) sets the stage for subsequent discussions on hybrid and cloud reasoning (Part 3), system integration with AppFunctions (Part 4), and the development of in-app agentic workflows (Part 5, forthcoming).

The availability of the full source code for Jetpacker on GitHub and accompanying video resources provides invaluable practical guidance. As Android continues its trajectory towards more intelligent and personalized user experiences, understanding and implementing on-device AI capabilities, as demonstrated in this exploration of Gemini Nano and ML Kit, will be crucial for developers seeking to build the next generation of groundbreaking applications.

The commitment to empowering developers with these advanced tools signifies Google’s dedication to fostering innovation within the Android ecosystem. By providing accessible and powerful AI solutions, the platform is poised to witness a surge in intelligent applications that not only enhance user productivity and enjoyment but also uphold the highest standards of data privacy and security.