Unlock the full potential of AI with Building LLMs for Production—our 470+ page guide to mastering LLMs with practical projects and expert insights!

# CV2: Finding Patterns On Images

Last Updated on November 18, 2023 by Editorial Team

#### Author(s): Krikor Postalian-Yrausquin

Originally published on Towards AI.

In this article, I used computer vision and neural networks to find a word in a text written in cursive more than a hundred years ago.

In this short example, I resort to the CV2 package, which is focused on computer vision to digest an image with text in cursive and extract one specific word, following a model trained with Tensorflow / Keras.

`import cv2import numpy as npimport pandas as pdfrom google.colab.patches import cv2_imshowfrom statistics import meanimport tensorflow as tf`

The image is read and transformed from color to a two-color scale.

`image = cv2.imread('test.jpg')#convert to grayscaleimage = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)cv2_imshow(image)`

The image is interpreted in the backend as a tensor, formed of three points for each pixel. These three points represent the saturation of color.

`image`

Now, I use the inRange function to set each pixel in a binary classification form, white or black. Then, I have to reverse the values since the standard in machine learning is white-over-black.

`lower = np.array([0, 0, 120])upper = np.array([0, 0, 255])msk = cv2.inRange(image, lower, upper)msk = cv2.bitwise_not(msk)cv2_imshow(msk)`

The next two functions are erosion and dilation. The first one is done to remove white points that might be noise (imagine sand eroding a rock). The second one is to inflate the white areas to create a fuzzy schema; with this supposedly, one continuous block has to be equivalent to one word, or at least close to it.

In [5]:

`#it is necessary to define a kernel, in this case it is the shape of the figure we want to extract (rectangle, elipse, circle, etc). In this case rectanglekernel = cv2.getStructuringElement(cv2.MORPH_RECT,(2,2))#then I erode the whites to remove the noise points, or at least reduce themrrmsk = cv2.erode(msk,kernel,iterations = 1)kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(10,5))rrmsk = cv2.dilate(msk, kernel, iterations=1)cv2_imshow(rrmsk)`

This section is just as an illustration performed to identify the words in the initial image. I also created a word object that stores rectangles of pictures of each word mined this way.

`contours, hierarchy = cv2.findContours(rrmsk, mode=cv2.RETR_EXTERNAL, method=cv2.CHAIN_APPROX_SIMPLE)minw = 50minh = 10image = cv2.imread('test.jpg')image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)cleancontours = []words = []for contour in contours: x,y,w,h = cv2.boundingRect(contour) if ((w>=minw) & (h>=minh)): cleancontours.append(contour) word = image[y-5:y+h+5,x-5:x+w+5] words.append(word)cleancontours = tuple(cleancontours)imageC = cv2.imread('test.jpg')imageC = cv2.cvtColor(imageC, cv2.COLOR_BGR2HSV)cv2.drawContours(imageC, cleancontours, -1, (0,255,0), 3)cv2_imshow(imageC)`

I take those words and handle them the same way I did in the previous steps.

`image = cv2.imread('test.jpg')image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)lower = np.array([0, 0, 150])upper = np.array([0, 0, 255])masks = []for word in words: if len(word) > 0: mask = cv2.inRange(word, lower, upper) mask = cv2.bitwise_not(mask) masks.append(mask)`

Here, I will deal with the positive training samples. I will be looking for the word “Garcia” in the image. I have collected samples of other images for the same word, written by the same person. I cleaned these pictures the same way as described before, then transformed them to the same size (the average of all samples ingested). The third image displayed here is an average of all 16 images.

`lower = np.array([0, 0, 150])upper = np.array([0, 0, 255])samples = ['sample1.jpg','sample2.jpg','sample3.jpg','sample4.jpg','sample5.jpg','sample6.jpg','sample7.jpg','sample8.jpg','sample9.jpg','sample10.jpg','sample11.jpg','sample12.jpg','sample13.jpg','sample14.jpg','sample15.jpg','sample16.jpg']train = []hs = []ws = []for sample in samples: im = cv2.imread(sample) im = cv2.cvtColor(im, cv2.COLOR_BGR2HSV) im = cv2.inRange(im, lower, upper) im = cv2.bitwise_not(im) height, width = im.shape hs.append(height) ws.append(width) train.append(im)from statistics import meanhh = int(mean(hs))+1ww = int(mean(ws))+1trainr = []for im in train: im = cv2.resize(im, (ww, hh), interpolation = cv2.INTER_CUBIC) trainr.append(im)cv2_imshow(trainr[2])cv2_imshow(trainr[1])meanimg = np.mean(trainr, axis=0)cv2_imshow(meanimg)`

I take the average image and transform into binary values.

`lower = 80upper = 255meanimgT = cv2.inRange(meanimg, lower, upper)cv2_imshow(meanimgT)`

In this section, I take the average and extract an Euclidean distance from each of the words in the text. Let’s see if this simple method gives good results.

`minh = min(hs)-20minw = min(ws)-20maxh = max(hs)+20maxw = max(ws)+20testr = []for im in masks: height, width = im.shape if ((height >= minh) & (width >= minw) & (height <= maxh) & (width <= maxw)): im = cv2.resize(im, (ww, hh), interpolation = cv2.INTER_CUBIC) testr.append(im)a,b=meanimg.shapeleng = a*bdistc = []i = 0for im in testr: distance = np.sqrt(np.sum(np.square(meanimg - im)))/leng dist = pd.DataFrame({'position':[i], 'distance':[distance]}) i = i + 1 distc.append(dist)distc = pd.concat(distc, axis=0, ignore_index=True)distc.sort_values('distance')`

And…. the closest word is effectively Garcia.

`cv2_imshow(testr[23])`

In this next section, I take seven images of bodies of text, and I process them the same way as with the test image. The idea is to extract a number of dummy pictures of words that I can use as a negative training set.

`lower = np.array([0, 0, 120])upper = np.array([0, 0, 255])samples2 = ['dummy1.jpg','dummy2.jpg','dummy3.jpg','dummy4.jpg','dummy5.jpg','dummy6.jpg','dummy7.jpg']minw = 50minh = 10train2r = []for sample in samples2: im = cv2.imread(sample) im = cv2.cvtColor(im, cv2.COLOR_BGR2HSV) im = cv2.inRange(im, lower, upper) im = cv2.bitwise_not(im) kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(2,2)) im = cv2.erode(im,kernel,iterations = 1) kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(10,5)) im = cv2.dilate(im, kernel, iterations=1) image = cv2.imread(sample) image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) for contour in contours: x,y,w,h = cv2.boundingRect(contour) if ((w>=minw) & (h>=minh)): word = image[y-5:y+h+5,x-5:x+w+5] if (len(word) > 0): try: mask = cv2.inRange(word, lower, upper) mask = cv2.bitwise_not(mask) height, width = mask.shape if ((height >= minh) & (width >= minw) & (height <= maxh) & (width <= maxw)): mask = cv2.resize(mask, (ww, hh), interpolation = cv2.INTER_CUBIC) train2r.append(mask) except Exception as error: print(error)`

This is the core of the neural network model in Keras. In this case, layers are used that support two dimensional inputs and one dimensional binary output. It required a good number of attempts since the training set is very small and the model jumped fast into an overfitting region. I dealt with that with a low batch size, which allowed “wiggle” in the selection of data for each epoch, and by running it several times on different samples.

`from random import sampletrain2r_s = sample(train2r,30)training = trainr + train2r_sys = ([1] * len(trainr)) + ([0] * len(train2r_s))training = np.array(training)ys = np.array(ys)#need shape of the inputnn, xx, yy = np.array(training).shape#initializeneur = tf.keras.models.Sequential()#layersneur.add(tf.keras.layers.Conv2D(5,3, activation='relu', input_shape=(xx,yy,1)))neur.add(tf.keras.layers.Conv2D(15,3, activation='tanh'))neur.add(tf.keras.layers.Conv2D(15,3, activation='tanh'))neur.add(tf.keras.layers.Flatten())neur.add(tf.keras.layers.Dense(10, activation='tanh'))#output layerneur.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))neur.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])neur.fit(np.array(training), np.array(ys), batch_size=15, epochs=3)`

To reduce the overfitting, I made a loop to train the model in different random training sets each time. I was able to do this because I had a large sample of negatives. I also needed to account for the high imbalance between positives and negatives.

`for i in list(range(0,30)): train2r_s = sample(train2r,30) training = trainr + train2r_s ys = ([1] * len(trainr)) + ([0] * len(train2r_s)) training = np.array(training) ys = np.array(ys) print("training part: " + str(i+1)) neur.fit(np.array(training), np.array(ys), batch_size=30, epochs=3)`

Run on test, unseen data.

`test_out = neur.predict(np.array(testr))pd.DataFrame(test_out).sort_values(0, ascending=False)`

Extract the top testing image (value closest to 1), which is effectively the word Garcia.

`check = 23print(test_out[check])cv2_imshow(testr[check])`

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI