CV2: Finding Patterns On Images

Author(s): Krikor Postalian-Yrausquin

Originally published on Towards AI.

In this article, I used computer vision and neural networks to find a word in a text written in cursive more than a hundred years ago.

In this short example, I resort to the CV2 package, which is focused on computer vision to digest an image with text in cursive and extract one specific word, following a model trained with Tensorflow / Keras.

import cv2
import numpy as np
import pandas as pd
from google.colab.patches import cv2_imshow
from statistics import mean
import tensorflow as tf

The image is read and transformed from color to a two-color scale.

image = cv2.imread('test.jpg')
#convert to grayscale
image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
cv2_imshow(image)

The image is interpreted in the backend as a tensor, formed of three points for each pixel. These three points represent the saturation of color.

image

Now, I use the inRange function to set each pixel in a binary classification form, white or black. Then, I have to reverse the values since the standard in machine learning is white-over-black.

lower = np.array([0, 0, 120])
upper = np.array([0, 0, 255])
msk = cv2.inRange(image, lower, upper)
msk = cv2.bitwise_not(msk)
cv2_imshow(msk)

The next two functions are erosion and dilation. The first one is done to remove white points that might be noise (imagine sand eroding a rock). The second one is to inflate the white areas to create a fuzzy schema; with this supposedly, one continuous block has to be equivalent to one word, or at least close to it.

In [5]:

#it is necessary to define a kernel, in this case it is the shape of the figure we want to extract (rectangle, elipse, circle, etc). In this case rectangle
kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(2,2))
#then I erode the whites to remove the noise points, or at least reduce them
rrmsk = cv2.erode(msk,kernel,iterations = 1)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(10,5))
rrmsk = cv2.dilate(msk, kernel, iterations=1)
cv2_imshow(rrmsk)

This section is just as an illustration performed to identify the words in the initial image. I also created a word object that stores rectangles of pictures of each word mined this way.

contours, hierarchy = cv2.findContours(rrmsk, mode=cv2.RETR_EXTERNAL, method=cv2.CHAIN_APPROX_SIMPLE)

minw = 50
minh = 10
image = cv2.imread('test.jpg')
image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
cleancontours = []
words = []
for contour in contours:
 x,y,w,h = cv2.boundingRect(contour)
 if ((w>=minw) & (h>=minh)):
 cleancontours.append(contour)
 word = image[y-5:y+h+5,x-5:x+w+5]
 words.append(word)
cleancontours = tuple(cleancontours)
imageC = cv2.imread('test.jpg')
imageC = cv2.cvtColor(imageC, cv2.COLOR_BGR2HSV)
cv2.drawContours(imageC, cleancontours, -1, (0,255,0), 3)
cv2_imshow(imageC)

I take those words and handle them the same way I did in the previous steps.

image = cv2.imread('test.jpg')
image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

lower = np.array([0, 0, 150])
upper = np.array([0, 0, 255])
masks = []
for word in words:
 if len(word) > 0:
 mask = cv2.inRange(word, lower, upper)
 mask = cv2.bitwise_not(mask)
 masks.append(mask)

Here, I will deal with the positive training samples. I will be looking for the word “Garcia” in the image. I have collected samples of other images for the same word, written by the same person. I cleaned these pictures the same way as described before, then transformed them to the same size (the average of all samples ingested). The third image displayed here is an average of all 16 images.

lower = np.array([0, 0, 150])
upper = np.array([0, 0, 255])

samples = ['sample1.jpg','sample2.jpg','sample3.jpg','sample4.jpg','sample5.jpg','sample6.jpg','sample7.jpg','sample8.jpg','sample9.jpg','sample10.jpg','sample11.jpg','sample12.jpg','sample13.jpg','sample14.jpg','sample15.jpg','sample16.jpg']
train = []
hs = []
ws = []
for sample in samples:
 im = cv2.imread(sample)
 im = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
 im = cv2.inRange(im, lower, upper)
 im = cv2.bitwise_not(im)
 height, width = im.shape
 hs.append(height)
 ws.append(width)
 train.append(im)
from statistics import mean
hh = int(mean(hs))+1
ww = int(mean(ws))+1

trainr = []
for im in train:
 im = cv2.resize(im, (ww, hh), interpolation = cv2.INTER_CUBIC)
 trainr.append(im)
cv2_imshow(trainr[2])
cv2_imshow(trainr[1])
meanimg = np.mean(trainr, axis=0)
cv2_imshow(meanimg)

I take the average image and transform into binary values.

lower = 80
upper = 255
meanimgT = cv2.inRange(meanimg, lower, upper)
cv2_imshow(meanimgT)

In this section, I take the average and extract an Euclidean distance from each of the words in the text. Let’s see if this simple method gives good results.

minh = min(hs)-20
minw = min(ws)-20
maxh = max(hs)+20
maxw = max(ws)+20

testr = []
for im in masks:
 height, width = im.shape
 if ((height >= minh) & (width >= minw) & (height <= maxh) & (width <= maxw)):
 im = cv2.resize(im, (ww, hh), interpolation = cv2.INTER_CUBIC)
 testr.append(im)
a,b=meanimg.shape
leng = a*b
distc = []
i = 0
for im in testr:
 distance = np.sqrt(np.sum(np.square(meanimg - im)))/leng
 dist = pd.DataFrame({'position':[i], 'distance':[distance]})
 i = i + 1
 distc.append(dist)
distc = pd.concat(distc, axis=0, ignore_index=True)
distc.sort_values('distance')

And…. the closest word is effectively Garcia.

cv2_imshow(testr[23])

In this next section, I take seven images of bodies of text, and I process them the same way as with the test image. The idea is to extract a number of dummy pictures of words that I can use as a negative training set.

lower = np.array([0, 0, 120])
upper = np.array([0, 0, 255])

samples2 = ['dummy1.jpg','dummy2.jpg','dummy3.jpg','dummy4.jpg','dummy5.jpg','dummy6.jpg','dummy7.jpg']
minw = 50
minh = 10
train2r = []
for sample in samples2:
 im = cv2.imread(sample)
 im = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
 im = cv2.inRange(im, lower, upper)
 im = cv2.bitwise_not(im)
 kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(2,2))
 im = cv2.erode(im,kernel,iterations = 1)
 kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(10,5))
 im = cv2.dilate(im, kernel, iterations=1)
 image = cv2.imread(sample)
 image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
 for contour in contours:
 x,y,w,h = cv2.boundingRect(contour)
 if ((w>=minw) & (h>=minh)):
 word = image[y-5:y+h+5,x-5:x+w+5]
 if (len(word) > 0):
 try:
 mask = cv2.inRange(word, lower, upper)
 mask = cv2.bitwise_not(mask)
 height, width = mask.shape
 if ((height >= minh) & (width >= minw) & (height <= maxh) & (width <= maxw)):
 mask = cv2.resize(mask, (ww, hh), interpolation = cv2.INTER_CUBIC)
 train2r.append(mask)
 except Exception as error:
 print(error)

This is the core of the neural network model in Keras. In this case, layers are used that support two dimensional inputs and one dimensional binary output. It required a good number of attempts since the training set is very small and the model jumped fast into an overfitting region. I dealt with that with a low batch size, which allowed “wiggle” in the selection of data for each epoch, and by running it several times on different samples.

from random import sample
train2r_s = sample(train2r,30)

training = trainr + train2r_s
ys = ([1] * len(trainr)) + ([0] * len(train2r_s))
training = np.array(training)
ys = np.array(ys)
#need shape of the input
nn, xx, yy = np.array(training).shape
#initialize
neur = tf.keras.models.Sequential()
#layers
neur.add(tf.keras.layers.Conv2D(5,3, activation='relu', input_shape=(xx,yy,1)))
neur.add(tf.keras.layers.Conv2D(15,3, activation='tanh'))
neur.add(tf.keras.layers.Conv2D(15,3, activation='tanh'))
neur.add(tf.keras.layers.Flatten())
neur.add(tf.keras.layers.Dense(10, activation='tanh'))
#output layer
neur.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))

neur.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
neur.fit(np.array(training), np.array(ys), batch_size=15, epochs=3)

To reduce the overfitting, I made a loop to train the model in different random training sets each time. I was able to do this because I had a large sample of negatives. I also needed to account for the high imbalance between positives and negatives.

for i in list(range(0,30)):
 train2r_s = sample(train2r,30)
 training = trainr + train2r_s
 ys = ([1] * len(trainr)) + ([0] * len(train2r_s))
 training = np.array(training)
 ys = np.array(ys)
 print("training part: " + str(i+1))
 neur.fit(np.array(training), np.array(ys), batch_size=30, epochs=3)

Run on test, unseen data.

test_out = neur.predict(np.array(testr))
pd.DataFrame(test_out).sort_values(0, ascending=False)

Extract the top testing image (value closest to 1), which is effectively the word Garcia.

check = 23
print(test_out[check])
cv2_imshow(testr[check])

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI