Playing musical notes with Scilab

Hey! Are you fond with music? If yes, then you made the right choice of viewing this blog. :) Now, in case you're not so familiar with musical notes, you can try and search google about the different notations made available to everyone. A table of values for the correct frequency and pitch are available in the web. You can visit these websites [1] [2] [3] to start learning. Be sure to go back to this page and be fascinated with how powerful image processing is! :)

For the past weeks, I've been blogging about different image processing techniques we have learned in our Applied Physics 186 class. We learned about image enhancement using both histogram and Fourier analysis, morphological filtering, and of course, (color) image segmentation. Now that we are equipped with enough knowledge and experience on these different techniques, it's time to integrate them all in a more challenging activity: playing musical notes.

Given a musical note found on the web, the challenge for us is to play the keys using Scilab. We have to use all the image processing techniques we have learned to be able to segment the image and let the computer read the corresponding notes dictated by the processed image. For this activity, I chose the traditional version of the Auld Lang Syne. I've always been fascinated to this music. I wanted to play the Pachelbel's Canon, but I figured out it's better if I start first with something simpler.

Figure 1. Auld Lang Syne notes

I started out by cropping this image and obtaining just the notes and disregarding the lyrics at the bottom. By using imcomplement(), I obtained the inverse of the image so that the notes are valued 1 and the background are valued 0. My first instinct in removing the unwanted lines (staff) was to apply Fourier transform and creating a mask, but I figured out that applying morphological operations are much easier. I created a structuring element that is vertical and 3 pixels long and applied opening operation to the image to remove the single-pixel horizontal lines. The result is shown below:

Figure 2. Result after cropping, inverting the image and applying the opening operation with a vertical structuring element

In order for me to determine the correct notes, I also isolated the circular-shaped blobs found at the bottom of each note. I determined the centroid of each of these blobs using AnalyzeBlobs(). Also, I cropped the notes from each row and aligned them all into a single row so that the notes are read by the computer from left to right. The image is shown in the following figure.

Figure 3. notes arranged from left to right in correct order

This time, I can now assign the range of possible location of the vertical axis of the centroid for a particular note. The challenge in this method lies in the presence of whole and half notes that do not form as whole blobs in the image. I also need to isolate those that are non-quarter notes. I isolated these notes by getting the correct size of the blobs and filtered them using FilterBySize()

Now my next problem lies on the dots beside the notes. In detecting the additional time, I searched all the blobs first and compute for their corresponding area. The histograms are then plotted to check the distribution of the blob sizes.

Figure 4. Size Histogram of the isolated blobs

The ones below the size of 20 sqpixels corresponds to the dots, while those that are greater than 150 sqpixels are probably that of the eighth note. Those that belongs to the  range of 100-120 sqpixel size corresponds to the quarter notes. The rest of the blobs correspond to the half notes.

Having known this, the code is now ready to read the musical keys, and play the "Auld lang syne".

As you may well observe, it follows the traditional old lang syne. For this activity, I give myself a grade of 10/10 for being able to play musical notes using Scilab.