SyncBeat: Move. Draw. Hear. Engineering and Music

SyncBeat: Move. Draw. Hear. Engineering and Music

An interactive web application developed during ShiftAppens 25' that allows users to generate music from movement (via webcam) or freehand drawings (on tablet). Designed for inclusivity, SyncBeat turns gestures into rich sonic feedback using AI-powered motion tracking.

This 1 min video briefly shows how to interact with the app and how to control sound using our movement (the video contains sound).

Problem: Music creation often excludes those without musical or artistic training, and few tools combine body movement, drawing, and sound in an intuitive way.

Solution: A real-time web and mobile application that translates motion and gestures into musical expression. Aimed at accessibility, it features audio feedback and voice guidance for visually impaired users.

Technologies:

  • JavaScript, HTML, CSS
  • React Native
  • TensorFlow / Google’s Pose Detection libraries
  • Vite, VSCode
  • GitHub for version control
  • Figma, Canva, Photoshop for design and communication
  • Taikai for challenge collaboration

My role:

  • Led development and integration
  • Recorded and edited the demo video
  • Contributed to accessibility-focused design

Joined by my teammates that focused on business rules and brainstorming ideas:

  • Catarina Pereira — M.Sc. in Telecommunications and Computer Engineering, University of Minho
  • Daniel Cardoso — M.Sc. in Electronics and Computer Engineering, University of Minho
  • Licínia Mendes — Physics Engineering, University of Minho

GitHub: CatarinaT/SyncUs