letras sa senyas

the design

the mission

developed as our capstone thesis requirement, letras sa senyas is a real-time gesture recognition application. this system was built in collaboration with two other developers. as the project manager and lead developer, i directed the end-to-end system design and managed project timelines, while serving as the primary engineer for our custom machine learning pipeline

my core responsibilities

• technical leadership: architected the overall system flow, database structure, and user interfaces, with the help of my team
• ai engineering: handled all primary data collection scripts, model training, and integration of the gesture-recognition neural networks into the production mobile environment

the architecture

• frontend & mobile: nuxt.js, vue, capacitor (for native android/kotlin compilation)
• backend & database: firebase
• machine learning: python, tensorflow, opencv, mediapipe
• ui/ux design: figma

system architecture & logic

to ensure a seamless flow of data between the mobile application, the machine learning inference engine, and the administrative dashboard, we mapped out strict system architectures before writing any code:

machine learning: the hybrid neural network

to accurately translate filipino sign language in real-time, relying on a single type of ai model wasn't quite hitting the mark. i designed and trained a hybrid neural network architecture to handle different types of human movement:

• fully-connected neural network (fcnn): to recognize static gestures (stationary signs like letters or numbers)
• long short-term memory network (lstm): to track and translate dynamic gestures (signs that require continuous motion and sequential time-series data)