Community Core Vision v 1.4

Finally the summer is over. Google Summer of Code made this summer one of the most amazing summers I have had. There are many people I need to thank for the successful completion of my project including Pawel Solyga(Mentor), Christian Moore, Jimmy Hertz, Sharath Patali and many other users who took time to test and give feedback for the test version of CCV-COT.

Its a great pleasure for me to announce that my work will be integrated into CCV mainstream development and will be released with this release (1.4).

Note : There has been some major changes in this release, so please read this fully before trying CCV-1.4a. Also if you have not tried out CCV-COT , you may want to read this post prior to reading this.

Changelog (After CCV-COT):

  • Complete Fiducial Support (Integration of ofxFiducialFinder addon by Alain Ramos).
  • TUIO XML messages bug solved.
  • CCV Debug mode
  • Different Filter chain and controls for Fiducial and Object/Fingers.
  • Fiducial settings on config.xml file.
  • Object acceleration calculations included.
  • removal and inclusion of some keyboard controls.
  • Closing of application from the “X” button,minimizing the application.
  • Changes in GUI
  • Blob counter for Finger, Fiducials and Objects respectively.
  • Sample AS3 app to test Custom Object Tracker(COT) added.

Community Core Vision v 1.4a

  • For Fingers + Objects – You can follow the post regarding CCV-COT and get started. The blob counter in the information window shows the count of finger blobs, object blobs and fiducial blobs respectively.

  • For Fingers + Fiducial –
  1. Start the application. Configure(Adjust the filter and calibrate) it for Finger. Enable Fiducials mode in the “Track” Panel.
  2. Press “i”. This will take you to the Fiducial Control Mode. “i” is the filter toggle key. In the debug window you will see The “Filter” tag changing to “Fiducial” from “Finger/Object” (See the screenshot below).
  3. Now you can adjust the filters so that you can see clear fiducials on the Binary Image. Note : If Fiducial mode is not enabled in the “Track” panel , you will see either a blank image or a still image.
  4. For better understanding , think of it like this. The camera image is copied into two images, one of them is analyzed for Finger/Objects and other for fiducials. Initially (when Filter is “Finger/Object”), all the sliders and controls are used to control the image that is analyzed for “Finger/Object”. When you press “i”, now all the sliders and controls are used to control the image that is analyzed for fiducials.
  5. The blob counter in the information window shows the count of finger blobs, object blobs and fiducial blobs respectively.
  • For Fingers+Objects+Fiducials- (**Not recommended for all**) This was something which was disabled initially because Fiducial tracking solves the problem of Object tracking. But again , we decided to give the user the freedom to choose what they have to track.  In this mode, most likely Objects and Fiducial will be confused (e.g. the image below)

But of course you need to be very lucky to get this mode working. A lot of adjustment will be required. But it is worth trying ;).

Tests :

Thanks to jimmy hertz, he has tested this version of CCV with his DI setup. Here are some images from his test.

In case of any doubts/suggestions get back to me by mail/comment/twitter/Facebook or by whatever mean you can. :-). It is now released on NUIGroup. Grab your own copy here :



DOWNLOAD : CCV-1.4.0-lin-32-bin.tar.gz

DOWNLOAD CCV 1.4.0 For your own Platform


Compile it yourself from LATEST SOURCE CODE

CCV-COT : Update

In the past weeks, I have been working on my Last Post’s TO DO List, and this post is to update you with my work.

CCV Debug Mode :

As the CCV users may be knowing , CCV logging was removed in CCV 1.3. This was due to several users reporting a bug  that the logs become very big size(sometimes in GBs). Logs help developers to visualize a crash situation, so its quite an important part. Therefore I have tried a workaround for that. Instead of logging every time, we make the logging optional. This is called CCV Debugging mode. You just have to pass a command line argument -d for that e.g.

"Community Core Vision.exe" -d

And CCV Debugging mode is activated. Also , to make it simpler , in the next release a shortcut to CCV Debug mode will be included with the binaries. As usual this saves the Logs into the “logs” folder with the timestamp as the filename.

CCV Fiducial:

I know, this has been much awaited and many people have tried to do this before. Here is Stefan Schlupek’s repo where he managed to do it properly. I have integrated Fiducial support in CCV-COT.

Things TBD :

  • First of all , I have to test the Fiducial additions ASAP and do all the corrections.
  • Sending 2DObj messages for Objects (right now it is 2DCur) also for Fiducials.
  • The TUIO bug , that has been many reports of TUIO not working properly. Have to look into that.
  • Better Object Tracking algorithm (after all the integrations are done)

Release :

I am not releasing this time, as I have not tested them well. So in next few days, there will be changes going on. I will release them after testing it out. It must be soon though ! In case you are in a hurry to try it out , you can do it yourself by checking out the repository and compiling the code yourself. In case you run into any problems, just let me know via comments/mail.

Also for my previous version , slugmobile(noah) has compiled binaries for Mac OSX . In case you want to try them out, check here.

GSoC 2010 : CCV Template Matching

Hello everyone , its a little long since my last post but yes, I had been busy coding and thought of posting after I got some results. Now that I am getting some results, I thought of posting an article about my work and plans.

Also I want to tell you that there was a slight deviation of my plan/road-map from my last post. In that post I mentioned regarding the two parts, one for template selection and one for template matching. Well, I have changed the plan a little bit. Instead of having 2 tools, one for selection and one for tracking, I integrated selection and tracking into one.So it will be just modification of CCV, which will do both template selection and tracking. Initially I thought it will be a bit slow, but it turned out to be effective. You can follow my work on

Here is a small preview of what I did.

As you can see, in the GUI I have added  a “Track” section where you specify what to track (fingers/objects/fiducial). Also I added a “Template Area” panel , which decided the blue and green rectangle you see in the figure. Also I modified the settings section to have Save templates and Load templates.

Also I have included the TemplateUtils class in the sourcecode which takes care of all the template works. The templates are saved in “templates.xml” file in “data” folder.The ID will be auto assigned. In case you want to change it , you can change in the xml file and load the xml again while the CCV is running.

Right now , I am able to track objects(I am able to differentiate them from finger blob). Next what I will be doing is :

  • Template ID assignment
  • x,y points conversion of the templates.
  • Drawing of the tracked objects
  • TUIO message sending
  • Better Template matching method (Right now it is very simple)
  • CCV shortcut keys for templates
  • Fiducial integration

In next week, I will be implementing some of these functions and testing. Once it is ready for the release, I will have a release package for alpha testers and have a video showing the work. Also then I will be posting a step by step documentation soon how to work with this. Look for the posts here in near future.

Template Matching Invariant of Rotation

The first week of GSoC 2010 is over.In the mean time, I created a project on . All my updates will be in that google-code svn repository, so if you have svn command line or svn client such as Totoise SVN you can checkout the source. I also have committed my openSURF experimentation source-code there in /branch/test folder. All my experimentation will be going into that folder.

As of now I have worked on different algorithms. I am planning to make CCV-TT modular so that any algorithm can be fit into it easily, so that it will be easy for future developments also. I will be writing the structure of the project in my next posts when I start coding for the Template Selection Tool.

So as the topic says I had experimented on contour matching. The source-code can be found on the googlecode repository. It includes the images in Images folder. The test images are :

The first image is the main image where the matching has to be done (say a frame from video/camera). The second is the template provided by the user. I have tried using SURF to find out the feature points and got only single feature point among them.Which was quite slow and will not be good to track (feature point was not at the center). Therefore I moved on to contour matching. Then I realized that contour matching may give very good results [cvMatchShapes(…) returns 0 ] when tried on the same template [the one cropped and transformed from the main pic]. But when tried on different picture’s templates [like the images shown] it gave quite bad results. You can see the difference in the drawing of the contour.

After seeing the contours [drawn in white in both ], the difference can be seen, it can give bad results or say can match to arbitrary curves on the surface/image. Therefore I thought of new plan, say contour-bounding-rectangle matching. The advantages and disadvantages are :

Advantages :

  • Faster – It will involve very basic level math calculation.
  • Saving into files/Loading from files, as these are of CvBox2D data type, it can be saved in files/loaded from files. This will also fasten the process a lot, reducing the time for finding contour for every image saved and calculation.
  • minArea, maxArea for each contour- which will make it even more accurate.
  • Detection and tracking of angles- from the rectangle that has been matched.

Disadvantages :

  • It will not be very accurate.
  • It will be more like shape matching.

As I had done experiments on this, You can see this clearly from results. Or you can checkout the google-code repository and perform the experiment by yourself.

As you can see the result, the actual contour bounding rectangle is matched. In case you want to know the contour matching results, just include the line


in the for loop where the draw_box(…) is used. This will return a double value. For the best detection it will return 0.This code basically has 2 funtions-to match the images, or to see the drawn contours. You can see the match(…) and check(…) for more details as it is documented. For match(…) you have to define PROCEDURE as 1 and for check(…) , you have to define PROCEDURE as 2. It is better explained in source.

Also for the linux users, just copy the main.cpp and you can compile it with g++. But you need to have openCV2.0 installed.

This is all for now. Next I will be coding the template matching tool on OpenFrameworks. I will be coding it as CCV mod. For the progress of the work, watch out this space or the googlecode project.

SURF and OpenSURF : Usage of OpenSURF library

As I said before OpenSURF is an open source implementation of SURF library, which is being used for feature extraction and matching. In my last post I have provided the results of my experimentation. This post is for description of the methods and how to do it all by yourself.

Well, first of all you need to configure your IDE for using openCV dlls and libs. Then add the openSURF library to your project(else for testing you can create inside openSURF project e.g. see main.cpp). All you need is to include “surflib.h” from the openSURF library which contains all the main function you need for the implementation.

The main functions are :

//! Library function builds vector of described interest points  (from surflib.h)
inline void surfDetDes(IplImage *img,  /* image to find Ipoints in */
std::vector<Ipoint> &ipts, /* reference to vector of Ipoints */
bool upright = false, /* run in rotation invariant mode? */
int octaves = OCTAVES, /* number of octaves to calculate */
int intervals = INTERVALS, /* number of intervals per octave */
int init_sample = INIT_SAMPLE, /* initial sampling step */
float thres = THRES /* blob response threshold */)

//To get matches between two Ipoints (from ipoints.h)
void getMatches(IpVec &ipts1, IpVec &ipts2, IpPairVec &matches);

//! Draw all the Ipoints in the provided vector (from utils.h)
void drawIpoints(IplImage *img, std::vector<Ipoint> &ipts, int tailSize = 0);

//! Save the SURF features to file (from utils.h)
void saveSurf(char *filename, std::vector<Ipoint> &ipts);

//! Load the SURF features from file ( from utils.h)
void loadSurf(char *filename, std::vector<Ipoint> &ipts);

So as the very first function says, it calculates all the ipoints(Interest Points) from the image and stores them in the ipts vector.

As indicated by comments, the drawIpoints(…) method draws the interest points found by the previous method, saveSurf(…) saves the ipoints to a file and loadSurf(…) loads the ipoints from a file to a variable.

So once all interest points of the image are found, you can match them to the interest points of another image by the getMatches(), which stores all the matched points to the matches variable. The definition of IpPairVec is

typedef std::vector<std::pair<Ipoint, Ipoint> > IpPairVec;

So in case you want to connect all the match points by cvLine you can use the following code snippet (see the 5th procedure in main.cpp)

  for (unsigned int i = 0; i < matches.size(); ++i)

    const int & w = img1->width;
    cvLine(img1,cvPoint(matches[i].first.x,matches[i].first.y),cvPoint(matches[i].second.x+w,matches[i].second.y), cvScalar(255,255,255),1);
    cvLine(img2,cvPoint(matches[i].first.x-w,matches[i].first.y),cvPoint(matches[i].second.x,matches[i].second.y), cvScalar(255,255,255),1);

So this is all for now, I will keep you updated about my work !

So as the very first function says, it calculates all the ipoints(Interest Points) from the image and stores them in the ipts vector.


GSoC 2010 work has already started. As the very first phase, I have to choose a good algorithm to implement the pattern matching technique.Initially I am concentrating on the pattern matching on images and after this I will be concentrating upon integrating into CCV. The application may need to handle multiple image matching in real-time, therefore a good algorithm is needed. So as my mentor Pawel suggested, I looked around for different algorithms on feature extraction. Features of images remain same on different condition of images. Therefore it seems like a good idea.

The very first thing that I got regarding this was SIFT (Scale Invariant Feature Transform). Then I came across SURF (Speeded Up Robust Features) which is inspired by SIFT and has better performance. Also there are some implementations of SURF around internet which are listed in the wiki page. Even SURF has been integrated into openCV.  But I have come across another open-source implementation named OpenSURF. It was quite nicely documented and had this Notes on openSURF library which has awesome explanation. But you should really check out the source as that has nice examples. Also each header files contain the use of each function.

One more good feature in SURF is that the Interest Points(IPoints) can be saved to a file and loaded for use which takes very less time than calculating interest points everytime. There are some specific parameters of iPoints which need to be saved/loaded. Those are scale,x,y,laplacian and the 64 descriptors acquainted with it.

I had carried out this experiment on 4 set of images. The results can be downloaded here. Each image features are extracted to a file of the same name and with “.txt” extension.In this post I will be analyzing one of the categories in details. You can see the result of all the categories by downloading the link I provided.

The Categories are :(Click on the Image to see it in actual size)

I will be analyzing the last category in which I got good results.

Here are the results in the image form. The matches are joined together with lines.

If you see the third image, out of the 3 matches, only one match is correct. Similarly I have found from my work that, if the images are very much similar then SURF works very well. But in case there is a major viewpoint change or major difference, then SURF fails to identify the features. With the Samples I have provided, see the Ferrari Images [c1.jpg-c5.jpg], there were no matches found in them (1 wrong match found).

The time taken for calculation depends upon the size of the image and also how the image is . In the process of my work, I have seen images with 2 Ipoints to 5k iPoints.As the number of iPoints increase, the time taken is more.For me some images took ~25 seconds and some tool 0.015 seconds for the iPoints calculation.

So this algorithm is not as good as the one used in Google Similar Images. But yes, if the images are quite similar (with rotation/change in lightening), then SURF seems to work awesome !

GSoC 2010 – A New Start !

Hello World GSoC !!!

I think the last line says it all.I feel it is long enough to write the “Hello World” codes. Now I am moving on to a little higher level, of course with the help of (the?) Google .

Yes ,  I have been selected for Google Summer of Code 2010 for NUIGroup (Natural User Interface group).  My project title is “User-Defined Blob Detection And Tracking in CCV”. The full proposal can be downloaded here. My mentor will be Pawel Solyga for the project. I have always heard great things about Pawel, but never got an opportunity to work with him and now it is here. I plan to utilize this opportunity very effectively.

I don’t know how many people I have to thank to for this. But yes, I would personally like to thank Sharath patali, Anirudh Sharma, Rohan Anil,Sachid Swami,Christian Moore, Seth Sandler, Lawrence Muller and finally Pawel Solyga for their support and reviewing my application.And of course, I would like to thank Google for this awesome program for students. Only a GSoC student can know how much can this program help. Of course it is good to get paid for doing what you like with supports from Google.

Now it’s work time. I need to brush up my C++ and openCV skills. Need to read  some papers regarding pattern recognition and feature extraction. And got to finalize the structure of the coding although it is very much inside my head.

For initial information, I will be using openFrameworks as the base framework for building “Template Selection Tool”. I am planning to use moment matching on contours for the shape matching. I still got to try out  different things like SIFT and SURF algorithms which are very good for feature extraction.I got to see the Google Similar Images project from Google Labs from Pawel Solyga. I had my mouth wide open after seeing this project. I was not surprised about the possibility of this, but about the speed. I wonder what algorithm possibly give such an efficient and effective (use it to know) results. Simply Google is God .

[EDIT: Got to know that Google indexes the features of an image and anytime we click “Find Similar” button no feature extraction happens, rather index searching happens . Still Google is God 🙂 ]

And yea, OpenCV-yahoogroups people, I have been getting a lot of hits from you regarding the tutorial on “Template matching invariant of Rotation”. Its not that I am not keeping my promise, I just lost the code accidentally, which I am too lazy to rewrite. But yes, I promise to do that stuff as soon as I am done with my exams as its a part of my project.

Cheers all