07Machine Learning
Keyo
Data Science / Machine Learning Track Winner, DeerHack 2025
Co-authored IEEE conference paper, DSIT 2025
A voice-guided indoor object-finding assistant for visually impaired users, with related research published in the IEEE DSIT 2025 proceedings.
The idea behind it
Keyo is an indoor object-finding and navigation prototype built for DeerHack 2025. A camera feed is processed with YOLO object detection and MediaPipe pose estimation to locate a requested item relative to the user and generate spoken directions. The system combines person and object tracking, estimated depth and orientation, and remembered object locations, with a FastAPI backend and a browser interface for voice requests and live updates.
What it does
- Voice-activated navigation with hands-free interaction for 80+ object types
- Real-time object detection using YOLOv11 with CUDA acceleration support
- 3D spatial positioning with accurate distance and direction calculations using MediaPipe pose estimation
- Turn-by-turn voice guidance with automatic TTS instructions every 5 seconds
- Multi-method depth estimation combining pose geometry and object class knowledge
- Precise navigation instructions (forward, backward, left, right, up, down) with distance feedback
- Last known location memory for object tracking and position prediction
- Bilingual support (English and Nepali) for broader accessibility
- Responsive web interface with live video feed and object detection overlays
- Real-time WebSocket updates for status and instruction synchronization
- Mobile-optimized interface with touch controls and auto-rotate support
Engineering focus
- Implementing accurate 3D spatial reasoning from 2D webcam feeds with depth estimation
- Designing robust person and object tracking systems with temporal consistency
- Calculating precise turn angles and navigation directions relative to user orientation
- Balancing instruction frequency to be helpful without overwhelming visually impaired users
- Optimizing YOLOv11 performance for real-time processing at ~25 FPS on standard hardware
- Converting technical navigation data into natural, conversational language instructions
- Handling varying camera angles and environmental conditions for consistent detection
Published research
IEEE conference paper · 2025
Real-Time Assistive Navigation for Visually Impaired Using YOLOv12 and MediaPipe Pose Estimation
2025 8th International Conference on Data Science and Information Technology (DSIT)
The paper describes an indoor object-finding system that combines YOLOv12 object detection, MediaPipe pose estimation and voice interaction to provide directional audio guidance. Object-location memory and handling for temporary occlusion help maintain continuity when a target disappears from view.
The public prototype uses YOLOv11, while the published research describes a YOLOv12-based system. The repository documents the prototype implementation; the paper presents the related research.
Outcomes & impact
- Successfully implemented voice-activated indoor navigation for visually impaired users
- Achieved real-time object detection and tracking across 80+ different object types
- Built comprehensive 3D spatial navigation system with MediaPipe pose estimation
- Delivered automatic voice guidance every 5 seconds with Google TTS integration
- Created production-ready FastAPI backend with WebSocket support for real-time updates
- Developed responsive web interface optimized for desktop, mobile, and tablet devices
- Won Track Winner in Data Science/ML category at DeerHack 2025
- Implemented bilingual support making the system accessible to broader user base
Explore further
DeerHack's organizer archive lists TechTacticos as the 2025 Data Science / ML track winner.
Open documentationIndependent news coverage of the results, recognizing Tech Tacticos in Data Science and Machine Learning.
Open article