A vision-based automatic transcription of guitar music from RGBD videos

Guitarists play music by listening or by relying on transcriptions. These transcriptions can use different types of notations (e.g. common music notation, tablature, etc.). Tablature (or "tabs" for short) is a form of notation that is adapted by guitarists and that tell the guitarists the frets to press and on what string. Nowadays, transcription is still manual which makes it a slow process and vulnerable to mistakes. To address this issue, automatic transcription for guitar music videos can be applied; it can be done through audio, vision or a combination of both. However, having the same note in multiple positions on a guitar fretboard makes it more ambiguous for audio, hence it is believed that vision suffices. Unlike previous methods that rely on image processing algorithms and detecting fingertips to process a guitar playing video and extract tablatures, our approach uses an RGBD camera, semantic segmentation, hand pose estimation and optical flow. The RGBD camera captures RGB and depth frames, the segmentation is used for fretboard localization in a frame; the hand pose estimation estimates the location of joints and their 3D coordinates to recognize the different pressings and their places while the optical flow serves as a detector of string picking/plucking.