John Rousseau - Senior Capstone Experience

Project Journal

January 15, 2003

     Today we received our capstone projects. I was excited to be able to continue my work on the golf cart vision application. The background work and preliminary application work that I did during the Fall 2002 independent study will prove to be invaluable information as I begin the Senior Capstone Experience. Under the direction of Dr. Pankratz, we began to lay out the requirements for the semester. I am confident that I will have no problem achieving all of the goals. The first objective has a fairly simple premise, yet is vital to the success of the project. Task #1 requires me to create an API for the web cam, as it will be used on the golf cart. I need to discuss this and the project in general with Mr. Blahnik before proceeding.

     After meeting with Mr. Blahnik today we set a meeting time for later this afternoon so I could visit the golf cart. During our meeting we discussed the API and the first stage of development. We both agreed that the basic function the API should accomplish is to capture an image and do some pre-processing. Pre-processing should use a technique developed during the independent study called Matrix Processing. After forming the Matrix, the API needs to allow the user (application) discretion at what will done with the 'results'. The preferred programming environment we are using is Visual Basic. The problem with the Logitech QuickCam that we are using is that the interface to Visual Basic is more limited than the interface for MFC and Win32. Solved simply enough, we agreed to develop the application in MFC.

     This afternoon I brought the camera over to Mr. Blahnik's house and the work area of the golf cart. There are logistical issues that we need to work with. The most important of these is to make sure the camera can turn with the wheels. We discussed two methods to process images.

     The first method would be to mount the camera on the front bumper of the car. This method would allow us to always tell where the cart is heading. The major defect in this plan lies in its simplicity. We would only be able to tell where the cart is heading. During a turn, we would not be able to tell where we were heading.

     The more ideal solution would be to allow the camera the ability to turn with the wheels. This gives us a unique logistical issue in that the camera needs to turn with a ratio to the wheels. For example, if the wheels turn 20 degrees in either direction, and the camera turns 40 degrees, we would get skewed results. After making some test equipment out of wood, we decided that this solution would be feasible.

January 16, 2003

     I took the application that I developed during my independent study and began modifying it for the capstone project. In tandem with this work, I began working on making a DLL written in MFC or Win32 that can accomplish the picture to memory function we need. There are two functions that allow for image capture from the camera. The first is a function called PictureToFile. This saves an image from the camera into a 24-Bit BMP File. The PictureToMemory function saves the 24-Bit BMP file to the memory instead of to a file. The problem is the PictureToMemory function does not work in VB. Logitech also offers no support for the SDK and newsgroup support is limited. Logitech claims the PictureToMemory function will not work in Visual Basic, and all internet resources I can find concur.

January 17, 2003

     I met with Mr. Blahnik today to talk about what I found during my research. The application development is not currently a priority as this research will either make or break the project. For example, in order to build a car, you need to make sure the wheels are able to move. Right now, our wheels can't move. I shared a couple of the tech newsgroups sites that I had been searching. It seems like some developers have had success using PictureToMemory in Visual Basic. Mr. Blahnik and I talked about how we could accomplish this.

January 18 - 19, 2003

     I spent the weekend doing more research but getting little accomplished. Some people's comments I have read in the newsgroups even say the using the camera with Windows2000 is impossible. I am currently running Windows2000 and the Logitech camera and it is working properly. I also posted all 3 SDK's so Mr. Blahnik also has them available for research. It seems like if we can send in a pointer to an array, we can get the function to work.

January 20, 2003

     I met with Mr. Blahnik again today to go over the research I did over the weekend. I was able to find a robot developer that was able to get the PictureToMemory function working in VB. I printed off the code and brought it along so we could go through it. The solution they provided was to use the bytArray(0&) syntax to get the pointer. Mr. Blahnik didn't think this solution would work, testing concurred.

January 24, 2003

     I brought everything I have done thus far in with me to visit with Mr. Blahnik. Mr. Blahnik didn't feel like we were getting anywhere with the application, so I brought everything in to work on it together. We setup the camera and we went through the application in VB I had been working on. For about 2 hours, Mr. Blahnik and myself worked on getting the MFC example working, but neither him nor myself was able to get any further than I had.

January 27, 2003

     I sent some more examples I found over email to Mr. Blahnik for review. We also began talking about a way in which we could setup the equipment on the cart laptop.

January 29, 2003

     I met with Mr. Blahnik today to discuss where we should go from here. We are nearly at a standstill with the research, and since so much time has been dedicated to it, the application has not made much progress. We devised a way in which we could attack the problems. We would need to run 3 processes in tandem. The first process would be the continued research of the PictureToMemory function. PictureToFile is just too slow to be productive, so we need the memory function to work. The second process would be to continue develop to application by processing files. It is understood that this is temporary and will be slow, but it will allow us to make progress. The third process would be to think about ways to implement the camera on the cart laptop. If we can find a reasonable proceed exact camera, we should purchase one to allow application development to move faster.

January 31, 2003

     I completed a large portion of the initial application that we talked about today. I stopped in by Dr. Pankratz to get help on the image-processing algorithm that I was working on. I also spent about an hour showing my solution to Mr. Blahnik. At this point I was able to detect colors in an image, display an image map of matched colors, and fill the 'Matrix' with the correct values. The Matrix is a smaller version of the picture divided up in a 15 x 15 grid. Each location on this grid represents 300 pixels on the original image. The Matrix contains a 1 at any given point if there are enough matched pixels in the quadrant to confirm the color exists. Ultimately, this Matrix will be used to tell us where the object is that we are following. If the object is off to the left, we need to move left. All of the work we have been doing thus far has culminated into the Matrix. Mr. Blahnik and myself made plans for tomorrow morning to go over the application a gain and implement some of the ideas Mr. Blahnik came up with as a result of my research. He is thinking we may be close to getting the PictureToMemory function working.

February 1, 2003

     I met at Mr. Blahnik's house today. After going through the application again we began trying to get the PictureToMemory function working. Through some testing / trial / and error, we were able to get the PictureToMemory function working in Visual Basic. Today was a huge step since everyone said this feat was impossible, even Logitech. Now that we were able to get PictureToMemory to work, our original application needs to be modified to use the memory function instead of the file function.

February 2, 2003

     Mr. Blahnik gave me the cart laptop to take with me and install the camera. I had the same install problems with his computer that I had with all of the other installs I have performed. Out of the 4 I have done, not a single one was even similar to any other one. I made a lot of progress on the laptop during the day. I was also able to fully convert the application to use the memory function.

February 4, 2003

     After spending the past couple of days working on the laptop, I was able to get the Logitech SDK installed. After the last reboot, the laptop ran into a couple of errors and ended up getting stuck in an infinite loop. Since I didn't have the time to work on the issue immediately, I enlisted the help of Matt in Computer Services. Since the laptop is an IBM, and Matt is IBM certified, I thought I'd let him work on it. After showing him what had occurred, he began to work on it and didn’t think the problem was that serious. He was confident in getting it repaired today. I asked him to communicate with Mr. Blahnik directly from here on. I also began refining the algorithm I am using since time is of the essence in the application. In lieu of getting the laptop fixed, I wrote a very simple application and posted it to the G: drive for Mr. Blahnik to use to make sure things were working. The program is called Verify_Cam and is necessary to ensure key features of Logitech's software are functional.

February 5, 2003

     There are still problems with the laptop. Windows 98 needs to be removed and Windows XP needs to be installed. There was a problem with the File Allocation Table and XP can correct the problem.

February 10, 2003

     The laptop now has XP installed on it and I am trying again to install the Logitech SDK. There are still many problems with XP. I can get the Logitech software working, but not the SDK DLL's. I brought the laptop back to Mr. Blahnik so he could try working on it. I am at a standstill until we figure out what is wrong with the laptop. If something needs to be changed with the program, we have some serious problems. We need to get to the bottom of the laptop problem. Since I am using Windows 2000, and that is all that we have been able to get the software working on, we are having Matt put Windows 2000 on the machine. The thinking is that we know this operating system works, so we should be able to get the laptop to work.

February 13, 2003

     Mr. Blahnik informed me today that he was able to get everything working with Windows XP and some additional software. He was able to download this software through a 3rd party site. The laptop is sent back to Matt to put Windows XP back onto it. Hopefully this will be the last time computer services needs to work on the laptop. We will be all set with Windows XP, so we can now focus on the actual software development. I will begin adding the turning logic into the software to prepare it for golf cart installation.

February 20, 2003

     Today, I met with Mr. Blahnik about our next step. He shared the client/server application he has been working on. The application is unique to anything that I have been developing thus far, but I will be able to use the code and technology I have developed. The application works by having 1 server. The server has the photo software as well as the camera installed on it. The key part to this design is that once the camera and photos are installed on the server, the client never has to worry about or even know how things are connected. The client simply sends commands like "Get*Picture". The client then receives the picture from the server (everything permitting) for processing. The beauty of this design is that many clients can be created and running at the same time. Each client can be written to perform a completely different task, yet interact with the same server. I am looking forward to being able to create different clients.

February 22, 2003

     I began creating a new client that uses my code. Things are going very well with new system. The client/server application setup will work very well.

March 1st, 2003

This week was spent doing a bunch of little things. I am working on the algorithms to speed up the recognition software. I also have been speaking to Mr. Blahnik about our next steps. I hope we will be able to test on the cart soon since we have many different interactions that have to occur.

March 12th, 2003

We tested on the cart for the first time today. For a first test, it was success! While testing in the garage with a single object, everything went pretty well. We jacked up the front end of the cart, allowing for easier motion of the wheels. Taking a uniquely colored object (A bright yellow Pennzoil bottle), we moved the bottle back and forth. The wheels followed relatively quickly. The garage had controlled lighting, so we tried the same thing outside. We had problems when we went outside with lighting and other issues. We didn't test outside very much, since we had a couple of issues with the lighting. The RGB values of the target pixels were changing so much, even though they didn't seem to by the naked eye.

March 25th - 27th, 2003

Based on results from the first cart test, I began making changes to the code. I added easier controls for exposure and gain. I also began experimenting with different solutions to the lighting problem. One possible solution is to recalibrate the image each time, but that would need to be done automatically. Mr. Blahnik suggested that I search for shapes and colors, not just colors. I thought about this solution, but it seems that that brings us back to the exact same issue is we lose the color. Another possible solution would be to light the object we are searching for. Through testing, we found that this also only works in the shade or low light. During periods of heavy sunlight, the same issue remains.

April 16-18nd, 2003

I began experimenting with some form of auto-calibration. It is a solution that will need to be worked on more, but I am trying to compensate for different lighting issues. Sunlight is affected my RGB pixel colors so much, that the object is lost if extra shade / sun hits it. I am beginning to test things like averaging pixel colors to help with color calibration.

April 16th-18th, 2003

This week, I went through my entire software to make sure there weren't any 'dumb' errors. Trying to debug the error regarding the overcompensation is proving very difficult. The only way I have been successful has been to setup the laptop on the cart, then test algorithms on the fly. I created a couple of different solutions, all but one I commented out. As we tried each one, I would switch the one that was commented out.

April 22nd, 2003

We tested some new algorithm changes on the cart today. We are still having the weaving issue. The software seems to be going much faster than the mechanics can operate. Commands then queue up so much that even if we stop the cart, the wheels still move back and forth to get everything out of the queue. I need to modify the algorithm to get rid of the command build up.

April 23rd, 2003

Again today we tested the cart. While testing, we tried many different things on the fly to try and get the commands out of the queue. A partial solution was to add a timer into the software so it doesn't operate quite so fast. It seems ironic that we spent most of our time early in the semester to speed things up, now we have to slow them down. I am learning a lot about actual projects by doing this. This is the first project I have worked on where natural factors were so important. (Lighting, electricity, mechanics, etc.) In a parallel motion, we worked on the lighting issue. A non to technical solution involved holding an umbrella over the object while moving so the sun could not affect the color as much. This solution seemed to work surprisingly well.

April 24th, 2003

Testing on the cart continued today. It seemed like there were about 5 small solutions that helped curb the problem with bouncing back and forth. Slowing down the software, waiting for an acknowledgement from the server, not making drastic / long turns, and waiting on the accelerator until the target is directly in front all helped to get a solution that worked well. We tried most of the solutions individually, but never tried all of them together. One thing we discussed was having some sort of speedometer on the cart. If we could tell how far we have moved, we could use math to figure out how far the cart has turned. If we know hoe far the cart has turned, we could anticipate corners. This would give up the ability to keep moving while the cart steered more proactively.