One of the techniques to find an object in a picture is by searching for its color. This sounds easy, as for a human we can easily see what is red or blue or any other color. But for a computer it is not that easy. What is red? Or blue?
Looking for a green object in a picture cannot be done by looking for the RGB value 00FF00. There are many shades of green that we as a human consider green. It may be lighter, darker, warmer, colder, etc.
So we need to find another approach. This is where HSV comes in. HSV stands for Hue, Saturation, and Value. If we can convert our color to those values, we can then build a range which we will consider as ‘the green’ ones.
Let’s dive into the code.
Create a class Blob with the following properties. This will simply hold our result pictures:
MyPic as picture
MyRealPic as picture
Now, we are going to find the blob of our desired color in the function GetBlobFromPic().
For this tutorial I’ll created the ranges for Red, Green, Blue and Yellow objects. We first convert the RGB pixel to it’s HSV values. To do this I used some math functions I found in a math article on the internet. I can’t recall which one, but I’ll try to find it back so I can post it here.
Once we have our HSV values, we’ll compare if they fall within the range we are looking for. If so, we mark it in our blob by setting the pixel to &C000001. You can also do this in a table, but for the purpose of this tuorial I also want to show the result of this search (shown in canvas2).
Here is the main part from this function:
' these values are precaulculated
select case FindColor
case &c00FF00 ' green
minH = 48
maxH = 120
minS = 40
maxS = 255
case &cFF0000 ' red
minH = 236
maxH = 14
minS = 40
maxS = 255
case &cFFFF00 ' yellow
minH = 25
maxH = 47
minS = 100
maxS = 255
case &c0000FF 'blue
minH = 121
maxH = 170
minS = 40
maxS = 255
end select
' search for the color within our range
for ax = 0 to SrcPic.Width - 1
for ay = 0 to SrcPic.Height - 1
C = SrcRGB.Pixel(ax,ay)
H = C.Hue * 255
V = C.Value* 255
S = C.Saturation * 200
minV = 50 - (30/215 * (S - 40))
maxV = 170 + (70/215 * (S - 40))
if minH - MaxH < 0 then
if H >= MinH and H <= maxH and S >= minS and S <= maxS and V >= minV and S <= maxV then
TgtRGB.Pixel(ax,ay) = &c000001
end if
else
if (h >= minH or H <= maxH) and S >= minS and S <= maxS and V >= minV and S <= maxV then
TgtRGB.Pixel(ax,ay) = &c000001
end if
end if
next
next
Then, we do some cleanup. If more than half the pixels surrounding our found pixel are not marked, unmark it (&cFFFFFF). We’ll search like this:
AAA
AXA
AAA
Where X is our marked pixel we want to check and A the ones around it. For speed, I also check X itself so I don’t have to use an IF in the loop.
Here is the code that does this:
for ax = 0 to tgtPic.Width - 1
for ay = 0 to tgtPic.Height - 1
Teller = 0
for zx = ax - 1 to ax + 1
for zy = ay - 1 to ay + 1
if TgtRGB.Pixel(zx,zy) <> &c000001 then
Teller = Teller + 1
end if
next
if Teller > 4 then
TgtRGB.Pixel(ax,ay) = &cFFFFFF
end if
next
next
next
Last, We’ll use our ‘map’ with marked pixels to get the original pixels that we can show in Canvas3:
for ax = 0 to tgtPic.Width - 1
for ay = 0 to tgtPic.Height - 1
if TgtRGB.Pixel (ax,ay) = &c000001 then
realRGB.Pixel(ax,ay) = OrigRGB.Pixel(ax,ay)
end if
next
next
That’s it!
Here are some other results we get from our program:
To make this even better, you can first do a Gaussian Blur on the picture so most of the holes in the found object are closed. The gaussian blur is not in this tutorial, but a quick seach on the internet will give you several sites where this is explained.
I hope this explained a little bit how we can do a fast search of an colored object in a photo.
I had some requests for the XBOX 360 Kinect source code so I decided to put it up on the blog. It contains an encrypted object KinectCanvas that can be freely used in your own projects and an example on how to use it.
The blob detection is done in pure RealBasic. Provided is a PC driver for the kinect. The only thing this gives back are pictures, just like a normal webcam. As it is a Kinect, we also get a depth picture and that is the one we’ll use to do the fun stuff. The depth picture is something like this:
For the attached demo project this is my setup at home:
This is how you install the included driver:
1. Unzip everything to a folder (let’s say c:\KinectCanvas).
2. Start CL-NUI-Platform-1.0.0.1210.exe to install the kinect driver
3. plug in your Kinect and power
4. start KinectCanvas.exe or open the source code
You may have to change the position of the kinect camera by using the motor functions:
ret = KinectCanvas1.startMotor
' value between -8000 and 8000
ret = Kinectcanvas1.SetMotorPosition(-4500)
KinectCanvas1.StopMotor
The most import function is the FindBlobs() function:
' for the moment use these params as they are for hand detection
aantalblobs= KinectCanvas1.FindBlobs(110, 170, 15, 150, false)
Here are the available functions, but not all of them are tested yet:
FindBlobs (MinRangeCM as integer, MaxRangeCM as integer, DepthOfBlobCM as integer, SizeOfBlobPixels as integer, Optional CheckGestures as Boolean = false) as integer
GetCameraColorFrameRAW(MakePicture as boolean, timeOut as integer) as boolean
GetCameraColorFrameRGB24(MakePicture as boolean, timeOut as integer) as boolean
GetCameraColorFrameRGB32(MakePicture as boolean, timeOut as integer) as boolean
GetCameraDepthFrameCorrected12(MakePicture as boolean, timeOut as integer) as boolean
GetCameraDepthFrameCorrected8(MakePicture as boolean, timeOut as integer) as boolean
GetCameraDepthFrameRAW(MakePicture as boolean, timeOut as integer) as boolean
GetCameraDepthFrameRGB32(MakePicture as boolean, timeOut as integer) as boolean
GetMotorAccelerometer(byref x as integer, byref y as integer, byref z as integer) as boolean
SetMotorLED(mode as integer) as boolean
SetMotorPosition(Position as integer) as boolean
StartCamera(tmpTimer as KinectTimer) as boolean
StartMotor() as boolean
StopCamera()
StopMotor()
The makepicture param is not used yet. It may be build in to speed up things. For the moment this is what you do if you want to get the picture for example from the depth camera in 32 bit colors:
' in the KinectTimer1.Action
Dim Ret as boolean
Dim a as integer
Dim b as integer
Dim tmpPic as picture
Dim surf as RGBSurface
Dim Tel as integer
Ret = KinectCanvas1.GetCameraDepthFrameRGB32(false, 1000)
tmpPic = newpicture(640, 480, 32)
surf = tmpPic.RGBSurface
for a =0 to 479
for b =0 to 639
surf.Pixel(b,a) = rgb(KinectCanvas1.CameraData.byte(Tel),KinectCanvas1.CameraData.byte(Tel+1), KinectCanvas1.CameraData.byte(Tel+2))
Tel = Tel + 4
next
next
canvas1.graphics.drawpicture tmpPic, 0,0
This code will result into the depth picture you see on top of this article.
The code is very experimental but you can use it to play with the Kinect in RealBasic!
Some people asked me what the output of 12Scan looks like. Well, it’s a plain TXT file with the matrix in it. A zero if it is not checked, A one if the user checked it. Any other program can then use this data how they want.
At OneTwo, I’m working at a new program with the work title 12Scan. It is a tool written in RealBasic to scan a page and analyze its contents.
It needs to be able to ‘read’ a page where people color in preprinted empty circles and return a txt file with a grid that represents the users selections. The picture can be taken with any device like flat scanners, webcams or mobile phone cameras.
Here is an example of such an empty page that could be used to check the attendance in a school:
It turns out it was not that easy to build as it looks. Especially the light and the environment were quite a challenge. With the knowledge I learned by building the augmented reality library for android, I could use some of the same techniques in this program.
A couple of things needed to be done. I wanted to avoid any rotation or deformation of the picture to keep the maximum quality of the picture. That meant manipulating the picture in other ways to find the hand colored points.
After some grey scaling, smoothing, blurring, sharpening and the other usual picture manipulations I’ve got a workable picture. The next function searched the blobs. I’ve used some functions I made in the augmented reality lib and that seems to work very well. Next was finding the square shape of the found blobs so I could create the matrix for all hand colored blobs. This looks very simple for a human, but for a computer it is quite a challenge.
This could be solved using a technique called PointsCloud. It searched for the corners within a ‘cloud of points’. The last part was aligning the hand colored points into the found matrix.
We are very satisfied with our preliminary results. It works very well with different light intensities and even on wrinkled or with coffee smothered pages. Here is a little video that shows our results:
This is done by dirty experimental code so my next task is rewriting it in a nice framework and optimizing the code for speed.
I had some questions about glowing and reflection. This tutorial will go a little more in depth on this subject.
I also show an alternative way to do the glowing with an inverted mask. This way we can put the glow and reflection mask into one picture.
This is what we going to build:
First let’s analyse our mask picture and explain the different parts:
1. The glow:
As you can see the glow is inverted compared with the glow we used in our previous lessons. The more black a pixel is, the less transparent it will be and thus the white layer will glow more.
2. The CD
Again, here the more black, the less transparent. We want to mimic a plastic case and we want to see our CD cover so we’ll make it completely white (transparent)
3. The reflection
And also so for the reflection. The more black a pixel is, the less we want to see our background through.
This is a complete schematic of all the layers we are going to use: Click picture to see full size
As you can see, we’ll use some temporary pictures to build the glow part and the reflection part.
Let’s go to the code:
In the LoadReflectPicture() function we’ll build all our layers. I commented the code to show where each part goes.
Sub LoadReflectPicture(tmpBackground as String, tmpImage as string, tmpMask as String, StartReflectMask as integer, Left as integer, Top as integer)
dim f as FolderItem
dim bgPic as Picture
dim imgPic as Picture
dim mskPic as Picture
' this picture will hold our flipped an cropped image
dim flpcropPic as Picture
' load the pictures
f = GetFolderItem(tmpBackground)
bgPic = f.OpenAsPicture
f = GetFolderItem(tmpImage)
imgPic = f.OpenAsPicture
f = GetFolderItem(tmpMask)
mskPic = f.OpenAsPicture
' calculate the Height of the reflection mask
dim HeightReflectMask as integer
HeightReflectMask =mskPic.Height - StartReflectMask
' draw the background
gBuffer.drawPicture bgPic,0,0
' Make a temporary picture that will hold our image with the glow and reflection
dim tmpPic as Picture
tmpPic = NewPicture(mskPic.Width, mskPic.Height, 32)
' first the glow, because we want to reflect this also
' draw our image
tmpPic.Graphics.DrawPicture imgPic,0,0
' to draw our glow, we need another temporary picture
dim glowPic as Picture
glowPic = NewPicture(mskPic.Width, StartReflectMask, 32)
' a white layer to do the glow
glowPic.Graphics.ForeColor = &cFFFFFF
glowPic.Graphics.FillRect 0,0, glowPic.Width, glowPic.Height
' the mask
glowPic.Mask.Graphics.DrawPicture mskPic, 0,0, glowPic.Width, glowPic.Height, 0,0, glowPic.Width, glowPic.Height
' and write it to our temporary picture
tmpPic.Graphics.DrawPicture glowPic,0,0
' now let's do the cropping and flipping
flpcropPic = FlipAndCrop(tmpPic, HeightReflectMask)
' draw it to our temporary picture
tmpPic.Graphics.DrawPicture flpcropPic, 0, StartReflectMask
' and last our reflection mask
tmpPic.Mask.Graphics.DrawPicture mskPic, 0, StartReflectMask, flpcropPic.Width, flpcropPic.Height, 0,StartReflectMask,flpcropPic.Width, flpcropPic.Height
' and write it to our buffer
gBuffer.DrawPicture tmpPic,Left, Top
End Sub
In the FlipAndCrop() function we do the actual flip and crop. We crop the CD cover to half the height of the original to mimic some perspective.
Function FlipAndCrop(srcPic as picture, HeightReflectMask as integer) As picture
dim tgtPic as Picture
dim srcRGB as RGBSurface
dim tgtRGB as RGBSurface
dim x,y as Integer
dim tmpc as Color
dim y2 as integer
Dim picWidth as integer
Dim picHeight as integer
picWidth = srcPic.Width
picHeight = srcPic.Height - HeightReflectMask ' only the CD, not the part ready for the reflection
tgtPic = NewPicture(picWidth, HeightReflectMask, 32)
srcRGB = srcPic.RGBSurface
tgtRGB = tgtPic.RGBSurface
' the part of the original picture we want to reflect and crop
' the height = 2 x HeightReflectMask
y2 = HeightReflectMask
for y = picHeight - (HeightReflectMask * 2) to picHeight step 2 ' step 2 so we'll crop
for x = 0 to picWidth
tgtRGB.Pixel(x,y2) = srcRGB.Pixel(x,y)
next
y2 = y2 - 1 ' we go from bottom to top, so flipping it
next
Return tgtPic
End Function
I hope this makes things a little bit clearer, but feel free to ask questions.
In this next tutorial we’ll going to add a menu to our elements so the users can really interact with them.
The idea would be that when we click on an element, this element becomes active and gets a menu with a play and stop button. When we click on another element, the menu is removed from the previous element and opened on the new element. When we click on something else than an element, it disappears.
The picture below shows what we want to do:
Let’s get started with the code:
We create a new class ABMenu that will hold our play and stop buttons. We give it a type and a picture.
X as integer
Y as integer
W as integer
H as integer
Image as picture
Sub Constructor(tmpType as integer, tmpImage as picture)
me.Type = tmpType
me.Image = tmpImage
End Sub
Sub Destructor()
CleanUp
End Sub
Sub CleanUp()
Image = nil
End Sub
In the ABelement, add a new property HasMenu. This way we can choose if an element has a menu or not. Also add a table that holds our MenuButtons.
Hasmenu as Boolean
MenuButtons(-1) as ABMenu
Change the Init() function to set our HasMenu variable:
Sub Init(tmpID as integer, tmpType as integer, tmpX as integer, tmpY as integer, tmpInnerX as integer, tmpInnerY as integer, tmpImage as picture, tmpMask as picture, tmpGlow as picture, tmpVisible as boolean, tmpHasMenu as boolean )
me.ID = tmpID
me.x = tmpX
me.y = tmpY
me.Visible = tmpVisible
me.w = tmpImage.Width
me.h = tmpImage.Height
me.Type = tmpType
me.MyGlow = tmpGlow
me.MyMask = tmpMask
me.InnerX = tmpInnerX
me.innerY = tmpInnerY
me.HasMenu = tmpHasMenu ' new
Image = NewPicture(w,h, 32)
image.Graphics.DrawPicture tmpImage,0,0
if MyMask <> nil then
me.w = MyMask.Width
me.h = MyMask.Height
end if
pBuffer = NewPicture(w,h, 32)
gBuffer = pBuffer.Graphics
DrawMe
End Sub
A new function AddMenuButton() will allow us to add Menu buttons to each element:
Sub AddMenuButton(tmpType as integer, tmpPicture as picture)
Dim tmpMenu as new ABMenu(tmpType, tmpPicture)
MenuButtons.Append tmpMenu
End Sub
Now for the changes in the ABCanvas class. We’ll add a new property CurrentMenuElement. This is similar to our DragElement from a previous lesson: it holds a temporary link to the element that currently has the menu.
CurrentMenuElement as ABElement
We make a change to our RefreshElement fuction because we want also to refresh the space that our menu is taking:
Sub RefreshElement(tmpElem as ABElement, Extra as integer) ' new
Dim iLeft, iTop as Integer
iLeft = tmpElem.x - tmpElem.w \ 2 - Extra ' changed
iTop = tmpElem.y - tmpElem.h \ 2 - Extra ' changed
' redraw only me and all other ABElements we are covering
DrawMe iLeft,iTop,tmpElem.w + Extra * 2,tmpElem.h + Extra * 2 ' changed
End Sub
Two new functions will handle a click on a menu button and if we hover over a menu button. We go through our current menu buttons and see if the mouse is over one of them. In that case we can start or stop playing the CD, DVD or comic. The code to play things is not included here as it would take us to far from our Canvas tutorials. A google search or the RealStudio forum can help you with that.
' to handle a click on one of our menu buttons
Function HandleMenuClick(X as integer, Y as integer) As boolean
Dim a as integer
dim tmpMenu as ABMenu
if CurrentMenuElement <> nil then
for a = 0 to Ubound(CurrentMenuElement.MenuButtons)
tmpMenu = CurrentMenuElement.MenuButtons(a)
if X >= tmpMenu.X and X <= tmpMenu.X + tmpMenu.W and Y >= tmpMenu.Y and Y <= tmpMenu.Y + tmpMenu.H then
' a button was clicked, handle what needs to happen (like start or stop playing)
MsgBox "You pressed button " + Str(tmpMenu.Type) + " on element type " + str(CurrentMenuElement.Type) + " with ID " + str(CurrentMenuElement.ID)
Return true
end if
next
end if
Return false
End Function
' to change our mouse cursor if we are hovering above a button
Function HandleMenuMove(X as integer, Y as integer) As boolean
Dim a as integer
dim tmpMenu as ABMenu
if CurrentMenuElement <> nil then
for a = 0 to Ubound(CurrentMenuElement.MenuButtons)
tmpMenu = CurrentMenuElement.MenuButtons(a)
if X >= tmpMenu.X and X <= tmpMenu.X + tmpMenu.W and Y >= tmpMenu.Y and Y <= tmpMenu.Y + tmpMenu.H then
Return true
end if
next
end if
Return false
End Function
Now for the main event: the function to show our menu around our active element. Once we’ve clicked a Element, the menu should be drawn around it. We’ll start with an orange frame around the Element and then we’ll draw our menu buttons. As all our elements have their own buttons, we iterate over the MenuButtons() table and draw them from right to left. We do this because we do not know how many buttons will be drawn and we want them right aligned.
Only when the menu is completely drawn we’ll push the changed part of the buffer to the canvas.
Sub ShowMenu(tmpElem as ABelement)
Dim iLeft, iTop as Integer
Dim a as integer
Dim posX as integer
' get the real left and right from our center x and y
iLeft = tmpElem.x - tmpElem.w \ 2
iTop = tmpElem.y - tmpElem.h \ 2
' draw our menu
gBuffer.ForeColor = &cFF5E0F
gBuffer.PenWidth = 3
gBuffer.PenHeight = 3
' we do minus 5 because our shadow = 5
gBuffer.DrawRect iLeft - 2, iTop - 23, tmpElem.w + 3 - 5, tmpElem.h + 24 - 5
gBuffer.PenWidth = 1
gBuffer.PenHeight = 1
gBuffer.FillRect iLeft - 2, iTop - 23, tmpElem.w + 3 - 5, 24
' and our buttons
PosX = iLeft + tmpElem.w - 23
for a = 0 to Ubound(tmpElem.MenuButtons)
gBuffer.DrawPicture tmpElem.MenuButtons(a).Image, PosX, iTop - 21
tmpElem.MenuButtons(a).X = posX
tmpElem.MenuButtons(a).Y = iTop - 21
tmpElem.MenuButtons(a).W = tmpElem.MenuButtons(a).Image.width
tmpElem.MenuButtons(a).H = tmpElem.MenuButtons(a).Image.height
posX = posX - 20
next
' and draw the result to the canvas
self.Graphics.DrawPicture pBuffer, iLeft - 3, iTop - 24, tmpElem.w + 6, tmpElem.h + 27, iLeft - 3, iTop - 24, tmpElem.w + 6, tmpElem.h + 27
End Sub
Let’s also write a function to hide our menu:
Sub HideMenu()
if CurrentMenuElement <> nil then
' let's redraw
' we add some extra pixels around the object when we redraw because we want the menu removed
RefreshElement CurrentMenuElement, 25
' and we reset our CurrentMenuElement
CurrentMenuElement = nil
end if
End Sub
Almost there! All we now have to do is handle the mouse down and mouse up event so that they can show or hide the menu, and if a menu button is clicked handle the button action.
First our MouseDown(). A very import part is that if we click on our canvas, we want to check if we are clicking a menu button. This has to be done first and before we’ll handle any other actions. If a menu button is handled, we can exit the function with return false to stop any other mouse events. If we click on the desktop, the existing menu is hidden.
Function MouseDown(X As Integer, Y As Integer) As Boolean
' the first thing we do is checking if we clicked on a menu
if HandleMenuClick(X,Y) then
' it is handled
Return false
end if
Dim tmpElem as ABElement
tmpElem = ElementHit(X,Y)
if tmpElem nil then
' does someone have a menu that is now visible?
HideMenu ' new
if IsContextualClick then
' right mouse button
' bring the found ABelement to the front
BringToFront tmpElem
' group them together
GroupMyType tmpElem.type, X, Y, 500
return false
else
' left mouse button
' remember our current position
mLastX = X
mLastY = Y
' bring the found ABelement to the front
BringToFront tmpElem
' refresh the element so it is redrawn
RefreshElement tmpElem, 0
' remember this ABElement so we can drag it around
DragElem = tmpElem
Return true
end if
else
' does someone else have a menu that is now visible?
HideMenu ' new
end if
' continue with the default MouseDown event
return MouseDown(X,Y)
End Function
In our MouseUp() event we’ll show our menu.
Sub MouseUp(X As Integer, Y As Integer)
if DragElem <> nil then
' refresh one last time so we have the very last position
RefreshElement DragElem,0
' set our DragElem to nothing
DragElem = Nil
end if
Dim tmpElem as ABElement
tmpElem = ElementHit(X,Y)
if tmpElem <> nil then
if IsContextualClick = false then
' left mouse button
' show our menu, if any
if tmpElem.HasMenu then
ShowMenu tmpElem
' remember this ABelement having the menu
CurrentMenuElement = tmpElem
end if
end if
end if
' continue with the default MouseUp event
MouseUp X,Y
End Sub
And in the MouseMove() event we’ll handle the mouse cursor if we are hovering above a menu button:
Sub MouseMove(X As Integer, Y As Integer)
' first check if we are over a menu
if HandleMenuMove(X,Y) then
self.MouseCursor = System.Cursors.FingerPointer
Return
end if
' if we are above one of our Elements, we change the mouse cursor to a little hand
if ElementHit(X,Y) <> nil then
self.MouseCursor = System.Cursors.FingerPointer
else
self.MouseCursor = System.Cursors.StandardPointer
end if
' continue with the default MouseMove event
MouseMove X,Y
End Sub
Finally we’ll change the LoadType() function in the Main form to add our menu buttons to our elements. The CD and DVD we’ll give a play and stop button, the Comic only a play button. Here is the changed code:
tmpElem.Init(Counter,tmpType, rnd * (me.Width - tmpPic.width) + tmpPic.Width / 2, rnd * (me.Height - tmpPic.Height) + tmpPic.Height / 2,InnerX,InnerY, tmpPic, tmpPicMask, tmpPicGlow, true, true) ' new we use true to add a menu
' and add the menu items
' load our play button for our menu
select case tmpType
case 1 ' a comic has only a play button
Menuf = GetFolderItem("PlayButton.png")
tmpElem.AddMenuButton(1, menuf.OpenAsPicture)
case 2,3 ' a CD and DVD has a play and stop button
Menuf = GetFolderItem("PlayButton.png")
tmpElem.AddMenuButton(1, menuf.OpenAsPicture)
Menuf = GetFolderItem("StopButton.png")
tmpElem.AddMenuButton(2, menuf.OpenAsPicture)
end select
Phew, Done! Let’s run our application with the added menu. It should work like this:
As some of you correctly pointed out the LoadType() function does not work in RB version 2011+. It seems from 2011 on, the items list starts from 1 instead of 0. So change the line
for a = 0 to all.Count - 1
into:
for a = 1 to all.Count
Thanks to everyone for bringing this to my attention.
if you like my work
