Generative AI Multi-Modal Output Demo

This Generative AI-powered tutor expands the way users interact with media, allowing AI to not only retrieve information but actively manipulate and highlight content in real time. Whether guiding a learner through technical repair, explaining complex visual concepts, or navigating 3D models, the AI functions as an interactive assistant with direct control over video, images, and 3D objects.

Overview

AI-Driven Interactive Learning
The AI can highlight, zoom, annotate, and manipulate media to make explanations more engaging and hands-on.
Multi-Modal Media Integration
Supports video, images, and 3D models, allowing users to ask questions and see dynamic, real-time responses.
Embedded Intelligence
Media files are pre-annotated with structured data that enables the AI to understand and interact with content effectively.
Built-In Knowledge Retrieval
A custom CMS and vector search system ensures relevant media is surfaced and context-aware to the user’s conversation.

SCREENSHOT GALLERY

AI-powered video assistant highlights key moments dynamically.

AI manipulates a 3D model, acting as a digital twin of real-world equipment.

Users can ask AI about an image, and it will zoom, highlight, and label elements.

A React-based back-end admin to store, and annotate media content.

When new content is created, embeddings are automatically created and stored for a vector database.

The React.js back-end was built on Strapi, the headless CMS. Though I was also evaluating payload.

Highlights

Video Interaction & Highlighting
- Users can ask the AI questions about a video, and it can find and play key moments based on transcription metadata (SRT files).
- The AI can highlight important on-screen elements in real time.
Image Annotation & Analysis
- Images are stored with SVG overlays for AI-driven zooming, highlighting, and labeling.
- Users can ask the AI to explain specific parts of an image, and it will dynamically respond.
3D Model Manipulation
- The AI interacts with GLTF 3D models, reading structured metadata to understand how the object functions in the real world.
- Users can rotate, highlight, and manipulate parts of the model as they would with real equipment.
Smart Content Retrieval
- A Strapi-based CMS lets users upload, label, and annotate media while automatically generating vector embeddings.
- The Weaviate vector database ensures relevant media is surfaced based on the AI’s understanding of the current conversation.
Real-World Use Cases
- A student can explore course materials dynamically, asking deeper questions and getting tailored, interactive responses.
- A technician troubleshooting new equipment can see highlighted steps, manipulate digital twins, and retrieve just-in-time knowledge.

Conclusion

This project reimagines AI-assisted learning by making media not just searchable, but fully interactive. By integrating annotated video, images, and 3D models with an AI-powered real-time tutor, this system provides engaging, on-demand support for learners and professionals alike.