Raspberry Pi Photo-tank-robot

I always liked to construct small car kits. Later I liked to assemble robot kits with Arduino or Raspberry Pi. Finally, I felt stuck in the guidelines and wanted to make something beyond that.

One day I explored robotic parts and encountered a tank chassis made from glossy aluminum. I was inspired immediately. That chassis is going to be a base of a great robot!

I took off some parts from a previous car: Raspberry, power converter, motor driver, battery — and put them on the chassis. The tank became alive. I developed a simple REST interface with Python for Raspberry and even more simple app for Android. Then I was able to control the tank from my phone.

A tank has to shoot and I decided to set up a camera. There was a problem with a cheap camera case — it was pretty bad — so tight when closed that camera lens fell off and when opened the whole camera fell off. I finished with the opened case with a yellow electrical tape. Now the tank gained an ability to take photos.

I was surprised to see how smooth the tank moves around a room. Wheeled vehicles had some difficulties with turns on a soft carpet due to their skid-steering approach.

I considered a few ways of the tank further development. The most obvious aim was to make something like Roomba vacuum cleaner in terms of indoor navigation. But the big difference would be the tank should rely on its camera vision.

Then I opened for myself the world of Computer Vision. I downloaded OpenCV and decided to start with color markers attached to big furniture pieces. I printed a red circle, put it on TV and started to teach the robot to find the circle.

OpenCV applied a color mask to filter out wrong colors. Then I was searching for a round contour. If the contour found — the goal reached. But not so fast. There were some problems. The color depends a lot on a light in the room. So I had to search not for the red but for a range of brown colors. Also, the shape of the circle could be affected by an angle of the camera. Moreover, it can happen that some occasional circle would intrude into the camera shot.

So the idea of vision-based navigation failed. Anyway, the initial experience with OpenCV was greate. There is a snippet of code I used to find a red circle:

import cv2
import numpy as np
import sys
def mask_color(img, c1, c2):
    img = cv2.medianBlur(img, 5)
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, c1, c2)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)
    return mask
def find_contours(img):
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    blurred = cv2.GaussianBlur(gray, (5, 5), 0)
    thresh = cv2.threshold(blurred, 30, 255, cv2.THRESH_BINARY)[1]
    thresh = cv2.bitwise_not(thresh)
    im2, cnts, hierarchy = cv2.findContours(thresh, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
    cp_img = img.copy()
    cv2.drawContours(cp_img, cnts, -1, (0,255,0), 3)
    return cp_img
def find_circles(img):
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    blurred = cv2.medianBlur(gray,5)
    circles = cv2.HoughCircles(blurred,cv2.HOUGH_GRADIENT,1,20,param1=50,param2=30,minRadius=0,maxRadius=0)
    cimg = img
    if circles is not None:
        circles = np.uint16(np.around(circles))
        for i in circles[0,:]:
            cv2.circle(img,(i[0],i[1]),i[2],(255,0,0),2)
            cv2.circle(img,(i[0],i[1]),2,(0,0,255),3)
            print "C", i[0],i[1],i[2]
    return cimg
def find_circle(img, rgb):
    tolerance = 4
    hsv = cv2.cvtColor(rgb, cv2.COLOR_BGR2HSV)
    H = hsv[0][0][0]
    c1 = (H - tolerance, 100, 100)
    c2 = (H + tolerance, 255, 255)
    c_mask = mask_color(img, c1, c2)
    rgb = cv2.cvtColor(c_mask,cv2.COLOR_GRAY2RGB)
    cont_img = find_contours(rgb)
    circ_img = find_circles(cont_img)
    cv2.imshow("Image", circ_img)
    cv2.waitKey(0)
if __name__ == '__main__':
    img_name = sys.argv[1]
    img = cv2.imread(img_name)
    rgb = np.uint8([[[0, 0, 255 ]]])
    find_circle(img, rgb)

Next month I spent in technological frustration. The only retreat was cat hunting — I used the tank to take a picture of my cat and then recognize it with a Haar cascade (a cascade for cat faces is a part of OpenCV). The success rate was not high — the cascades are pretty basic approach and the cat can look many different ways.

Anyway, this side activity had a positive impact on the tank — as far as it was not always possible to catch the cat into a static frame, I put a camera stand with two servo motors and made them controlled via Raspberry.

Started to recognize images very soon I came to neural networks. Tensorflow turned out to have great abilities in images detection. I fed to TF many pictures taken by the tank — and most of the furniture and the cat were recognized correctly. This approach looked much more promising than the colored markers.

The initial research was done on my laptop. Then it was time to move to Raspberry. Many thanks to Sam Abrahams who had enough passion and patience to compile TF on Raspberry. I can imagine it was not an easy task, just read his instruction on GitHub.

During the further search, I discovered OpenCV made a big step toward neural networks. Its new module DNN (Deep Neural Networks) provides an ability to use already prepared neural network without even having Tensorflow installed. There were some tricks — at first, the latest version of TF-network was not compatible with OpenCV and I had to find the right one.

Second, I did not find OpenCV 3.4.1 (with stable DNN) for Raspberry Pi and I had to do it on my own. It is much easier to build OpenCV than Tensorflow. One more issue — the build failed on Raspbian Stretch (latest), but gone smoothly on Raspbian Jessie. That was a working solution to detect objects right on the tank, lightweight (no whole Tensorflow required) and local (no external calculations or data exchange).

I put the built OpenCV 3.4.1 to GitHub.

There is a code to detect objects with OpenCV-DNN:

import cv2 as cv
import tf_labels
import sys
DNN_PATH = "---path-to:ssd_mobilenet_v1_coco_11_06_2017/frozen_inference_graph.pb"
DNN_TXT_PATH = "--path-to:ssd_mobilenet_v1_coco.pbtxt"
LABELS_PATH = "--path-to:mscoco_label_map.pbtxt"
tf_labels.initLabels(PATH_TO_LABELS)
cvNet = cv.dnn.readNetFromTensorflow(pb_path, pb_txt)
img = cv.imread(sys.argv[1])
rows = img.shape[0]
cols = img.shape[1]
cvNet.setInput(cv.dnn.blobFromImage(img, 1.0/127.5, (300, 300), (127.5, 127.5, 127.5), swapRB=True, crop=False))
cvOut = cvNet.forward()
for detection in cvOut[0,0,:,:]:
    score = float(detection[2])
    if score > 0.25:
        left = int(detection[3] * cols)
        top = int(detection[4] * rows)
        right = int(detection[5] * cols)
        bottom = int(detection[6] * rows)
        label = tf_labels.getLabel(int(detection[1]))
        print(label, score, left, top, right, bottom)
        text_color = (23, 230, 210)
        cv.rectangle(img, (left, top), (right, bottom), text_color, thickness=2)
        cv.putText(img, label, (left, top), cv.FONT_HERSHEY_SIMPLEX, 1, text_color, 2)
cv.imshow('img', img)
cv.waitKey()

By that moment I already knew the navigation based on only images is not easy at all. Measuring distance is the key feature and it hard to perform from a plain picture. So I go the traditional way and added an ultrasonic distance meter to the tank. Small technical difficulties (the sensor responds by 5V, Raspberry GPIO able to get only 3.3) were resolved by level shifter chip.

Also, I paid attention to the tank appearance. The first version used cardboard paper to attach the camera and the chips. I invested some time and money into a laser course. After that, I was able to design and cut acrylic plates for my needs. There were many attempts until I got the perfect main plate and the camera holder!

Finally, I got a good robot to start with autonomous navigation. This is a big topic and I was a little bit exhausted by bringing all the thighs together.

This is a REST interface running on the robot:

GET /ping 
GET /version 
GET /name 
GET /dist  
POST /fwd/on 
POST /fwd/off 
POST /back/on 
POST /back/off 
POST /left/on 
POST /left/off 
POST /right/on 
POST /right/off  
POST /photo/make 
GET /photo/:phid 
GET /photo/list  
POST /cam/up 
POST /cam/down 
POST /cam/right 
POST /cam/left  
POST /detect/haar/:phid 
POST /detect/dnn/:phid

The server code is available on GitHub.

Then started to pick the tank name. There were considered: PiTank (powered by Raspberry Pi), Selfie-tank (for its initial intention), Catfinder (reflecting the mission), Tensorflower (because of DNN), Never-Stop-Explorer (inspired by a box of shoes from the North Face).

I decided to go simple and use PiTank. Unfortunately it is turned out to be a registered name. So I changed one letter to make it sound like a bad french: PiTanq.