RideGuide

A multimodal conversational interface for passenger engagement and spatial learning in robotaxis

Table of Contents

• Overview
• Usage
• System Architecture
  • Technology Stack
  • Prerequisites
  • Setup
• Study fidelity
• Documentation
• Security notes
• Citation
• License

Overview

RideGuide is a Flutter research prototype for iOS and Android that investigates how a multimodal conversational interface can support passenger engagement and spatial learning in robotaxis. Passengers can use voice or touch to ask about nearby landmarks, explore points of interest, and receive context-aware spoken responses.

Key Features

• Multimodal Interface: Voice, touch, and visual interactions
• Context-Aware: Real-time location-based information and responses using OpenStreetMap/Wikipedia data
• Customizable: Personalized voice gender, chatbot personality, and spoken language (the chatbot's language, while the interface ships English only, see docs/LOCALIZATION.md)
• Map and POI Grounding: Interactive map with POI integration and landmark referencing

Usage

Onboarding: Customize voice gender, language, and chatbot personality

Voice Interaction: Simply start talking (no wake word required)

Map Interaction: Tap on POIs on the map interface for details

Contextual Queries: Ask about surroundings, landmarks, or specific directions

• "What's that building on the left?"
• "Can you recommend restaurants nearby?"
• "What is the history of this area?"
• "What is the entrance fee for that museum?"

The interface and architecture figures are Fig. 1 and Fig. 2 of the published paper, see doi:10.1007/s12193-026-00479-2.

Fig. 1: The RideGuide main interface illustrated: (left) after a directional voice query referencing a visible point of interest (POI), (middle) the default listening, (right) the touch interaction state after tapping a map POI, showing the POI image gallery and the Ask-AI button that grounds a subsequent voice query in the selected location. The map uses Mapbox with OpenStreetMap and Wikipedia place annotations, as well as Wikipedia images (in-app images replaced in figure).

System Architecture

Fig. 2: Interaction diagram: User Input (Voice/Touch) → Intent Detection (GPT-4o-mini) → Chat Intent Category Detection (GPT-4o-mini) → Context Processing → Response Generation (GPT-4o) → Text and TTS Output (tts-1) → Zoom to POI Location on Map.

Technology Stack

Layer	Technology
Framework	Flutter (Dart 3.5+)
LLM	OpenAI GPT-4o / GPT-4o-mini (optionally routed via OpenRouter)
TTS	OpenAI TTS (tts-1)
STT	speech_to_text (on-device)
Maps	Mapbox with OpenStreetMap data
Data Sources	Wikipedia geotagged articles, OpenStreetMap Places
Architecture	Intent detection pipeline with real-time processing

Prerequisites

• Flutter SDK 3.5+
• An OpenAI API key or an OpenRouter key, see docs/REPRODUCIBILITY.md
• A Mapbox account (public pk. access token only, no secret download token is needed for mapbox_maps_flutter ≥ 2.8)

Setup

1. Clone and install

git clone https://github.com/eveetc/RideGuide.git
cd RideGuide
flutter pub get

2. Configure environment variables

cp assets/config/.env.example assets/config/.env

Open assets/config/.env and fill in your keys. See assets/config/.env.example for all required variables:

OPENAI_API_KEY=sk-...
MAPBOX_ACCESS_TOKEN=pk.eyJ1...

3. iOS setup

No extra steps beyond flutter pub get. Run with:

flutter run -d ios

4. Android setup

No extra steps beyond flutter pub get. Run with:

flutter run -d android

5. VLC media server (optional, lab/demo setup)

The prototype can drive a VLC player (video simulation of the drive) over its HTTP interface. Defaults come from .env (VLC_IP, VLC_USERNAME, VLC_PASSWORD) and can be changed at runtime: onboarding → Technical Setup → ⚙ next to VLC Server opens a setup sheet with the last used values pre-filled; changes are persisted on the device.

Study fidelity

Some behavior of this code was intentionally preserved to match the build the published study ran on. This includes the byte-identical prompt wording (and the deliberate role: assistant quirk in two detectors), a known spatial-index defect that drops a small percentage of in-radius POIs, and architectural rough edges such as the StateManagerService god object. These are documented, not oversights: do not "fix" them if your goal is to reproduce the published results.

Documentation

• docs/PROMPTS.md: All prompts live in lib/services/AI/prompts.dart, pinned character-for-character by tests; this document describes each prompt and its role in the pipeline.
• docs/LOCALIZATION.md: Every user-facing string lives in lib/l10n/app_en.arb; how to regenerate localizations and add a language (no code changes needed).
• docs/ARCHITECTURE.md: Project layout and conventions, including the deliberate deviations from idiomatic Flutter that were kept to match the study build.
• docs/REPRODUCIBILITY.md: Exact study configuration (models, endpoints, study vs demo mode, route and vision assets), study implementation details, and known limitations.
• docs/ROADMAP.md: Areas for improvement: interaction, personalization, and data.

Security notes

• assets/config/.env is bundled as a Flutter asset so keys can be read at runtime, this means the OpenAI key is also inside any built APK/IPA. Do not distribute binaries with a personal/unrestricted key; use a restricted key or a backend proxy for production.
• The Mapbox token is a public-scoped pk. token by design (client-side maps); restrict it to your app in the Mapbox dashboard. If you ever committed a real token, rotate it.

Citation

If you use RideGuide, please cite both the archived software release and the accompanying journal article.

Software

Schade, E. (2026). RideGuide: multimodal conversational tour guide for passenger engagement and spatial learning in robotaxis (Version v1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.15699734

@software{schade_2026_15699734,
  author    = {Schade, Eve},
  title     = {RideGuide: multimodal conversational tour guide for passenger
               engagement and spatial learning in robotaxis},
  month     = jul,
  year      = 2026,
  publisher = {Zenodo},
  version   = {v1},
  doi       = {10.5281/zenodo.15699734},
  url       = {https://doi.org/10.5281/zenodo.15699734}
}

The version-specific DOI above identifies the exact archived release. The all-versions DOI is https://doi.org/10.5281/zenodo.15699733.

Journal article

Schade, E. (2026). RideGuide: multimodal conversational tour guide for passenger engagement and spatial learning in robotaxis. Journal on Multimodal User Interfaces. https://doi.org/10.1007/s12193-026-00479-2

@article{schade2026rideguide,
  title   = {RideGuide: multimodal conversational tour guide for passenger engagement and spatial learning in robotaxis},
  author  = {Schade, Eve},
  journal = {Journal on Multimodal User Interfaces},
  year    = {2026},
  doi     = {10.1007/s12193-026-00479-2},
  url     = {https://doi.org/10.1007/s12193-026-00479-2}
}

Machine-readable metadata is in CITATION.cff.

Funding. Swiss National Science Foundation (SNSF), Grant Agreement No 200021_207430.

The accompanying exploratory study involved 12 participants; full methods, parameters and results are reported in the associated article and summarized in docs/REPRODUCIBILITY.md.

License

Copyright 2026 Eve Schade, University of St. Gallen.

Licensed under the MIT License, see LICENSE.

The bundled third-party assets and data are not covered by that licence and keep their own terms; see NOTICE for the full list. In short:

Asset	Licence
Polestar 1 3D model (assets/models/polestar1-white.glb, modified)	CC BY 4.0 (Sketchfab)
OpenStreetMap-derived place data	ODbL 1.0
Wikipedia place descriptions	CC BY-SA 4.0
Mapbox Maps SDK / map tiles	Mapbox Terms of Service

These attributions are also shown in-app: onboarding → Technical Setup → MIT · Licences & attribution, which opens Flutter's licence page (registered in lib/config/licenses.dart) alongside every package dependency.