README.md 15.3 KB
Newer Older
Fábio Prestes's avatar
Fábio Prestes committed
1 2
## __Title: Pynovisao__
#### Authors (email):
3 4 5 6 7 8 9 10 11 12 13 14
- Adair da Silva Oliveira Junior
- Alessandro dos Santos Ferreira
- Diego André Sant'Ana (diegoandresantana@gmail.com)
- Diogo Nunes Gonçalves (dnunesgoncalves@gmail.com)
- Everton Castelão Tetila (evertontetila@gmail.com)
- Fabio Prestes Cesar Rezende (fpcrezende@gmail.com)
- Felipe Silveira (eng.fe.silveira@gmail.com)
- Gabriel Kirsten Menezes (gabriel.kirsten@hotmail.com)
- Gilberto Astolfi (gilbertoastolfi@gmail.com)
- Hemerson Pistori (pistori@ucdb.br)
- Joao Vitor de Andrade Porto (jvaporto@gmail.com)
- Nícolas Alessandro de Souza Belete (nicolas.belete@gmail.com)
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

## Resume:

Computer Vision Tool Collection for Inovisão. This collection of tools allows the user to select an image (or folder) and realize numerous actions such as:
- Generate new Datasets and classes
- Segmentation of images
- Extract features from an image
- Extract frames from videos
- Train Machine Learning algorithms
- Classify using CNNs
- Experiment with data using Keras
- Create XML files from segments previously created.

## Open Software License: 

NPOSL-30 https://opensource.org/licenses/NPOSL-3.0 - Free for non-profit use (E.g.: Education, scientific research, etc.). Contact Inovisão's Prof. Hemerson Pistori (pistori@ucdb.br), should any interest in commercial exploration of this software arise.

## How to cite:
33 34 35

[1] dos Santos Ferreira, A., Freitas, D. M., da Silva, G. G., Pistori, H., & Folhes, M. T. (2017). Weed detection in soybean crops using ConvNets. Computers and Electronics in Agriculture, 143, 314-324.

Fábio Prestes's avatar
Fábio Prestes committed
36 37
## How to Install

Fábio Prestes's avatar
Fábio Prestes committed
38
- **Option 1, Linux-only Script**
39

40
You can easily install Pynovisão utilizing the automated installation script given with it, as seen by the following steps:
Gabriel Kirsten's avatar
Gabriel Kirsten committed
41

42
- From inside of this directory:
Gabriel Kirsten's avatar
Gabriel Kirsten committed
43
```
44
 [...]/pynovisao
Gabriel Kirsten's avatar
Gabriel Kirsten committed
45
```
46

47 48 49 50 51
- Execute the following command:
```
$ sudo bash INSTALL.sh
```
**NOTE**: This script has been tested for Ubuntu versions 19.04 and 18.04
Gabriel Kirsten's avatar
Gabriel Kirsten committed
52

Fábio Prestes's avatar
Fábio Prestes committed
53
- **Option 2, without INSTALL.sh**
Gabriel Kirsten's avatar
Gabriel Kirsten committed
54

55
Besides it's dependencies, Python 2.7.6 or Python 3.6 is needed. (Latest tested versions for this software)
Gabriel Kirsten's avatar
Gabriel Kirsten committed
56

57
- Installing the necessary dependencies on Python 3.6:
Gabriel Kirsten's avatar
Gabriel Kirsten committed
58
```
59 60 61 62 63 64 65
$ sudo apt-get update
$ sudo apt-get install libfreetype6-dev tk tk-dev python3-pip openjdk-8-jre openjdk-8-jdk weka weka-doc python3-tk python3-matplotlib
$ source ~/.bashrc
$ sudo pip3 install numpy
$ sudo pip3 install -r requirements_pip3.txt
$ sudo pip3 install tensorflow 
$ sudo pip3 install keras
Gabriel Kirsten's avatar
Gabriel Kirsten committed
66
```
67 68

- Installing the necessary dependencies on Python 2.7:
Gabriel Kirsten's avatar
Gabriel Kirsten committed
69
```
70 71 72
$ sudo apt-get update
$ sudo apt-get install libfreetype6-dev tk tk-dev python-pip openjdk-8-jre openjdk-8-jdk weka weka-doc python-tk python-matplotlib
$ source ~/.bashrc
Gabriel Kirsten's avatar
Gabriel Kirsten committed
73
$ sudo pip install numpy
74 75 76
$ sudo pip install -r requirements_pip3.txt
$ sudo pip install tensorflow 
$ sudo pip install keras
Gabriel Kirsten's avatar
Gabriel Kirsten committed
77
```
78

Fábio Prestes's avatar
Fábio Prestes committed
79 80
## How to install Caffe ( Optional )

Fábio Prestes's avatar
Fábio Prestes committed
81
#### Ubuntu / Windows
Fábio Prestes's avatar
Fábio Prestes committed
82 83 84 85 86 87 88 89 90
In order to use the CNNCaffe classifier, a ConvNet based on the AlexNet topology, it is necessary to install Caffe.

It's installation is more complex than the ones previously mentioned, and more detailed instructions can be found below:
-  http://caffe.berkeleyvision.org/installation.html

After installing Caffe, in order to realize classification with it you will need to train it with Pynovisão using the command line, since there currently is no interface for ConvNet Training.

The tutorial for training can be found below:
- http://caffe.berkeleyvision.org/gathered/examples/imagenet.html
91

Fábio Prestes's avatar
Fábio Prestes committed
92 93 94 95
Finally, it is necessary to configure your CNNCaffe.
- For the fields *ModelDef, ModelWeights* and *MeanImage*, you must supply the relative paths to the traning done previously.
- For the field *LabelsFile* you must supply the path to a file that describes all the classes in order (0, 1, 2, ..., n-1; where n is the number of classes trained).
- A example file can be found in **[...]/pynovisao/examples/labels.txt**.
96

Fábio Prestes's avatar
Fábio Prestes committed
97
## How to use:
98 99

#### Opening the software
Fábio Prestes's avatar
Fábio Prestes committed
100
- In order to download Pynovisao, click the download button in the top right of the screen (Compressed folder), or type the following command in a terminal:
101
```
Fábio Prestes's avatar
Fábio Prestes committed
102
 $ git clone http://git.inovisao.ucdb.br/inovisao/pynovisao
103
```
104

Fábio Prestes's avatar
Fábio Prestes committed
105 106 107 108
- From inside of this directory:
```
 [...]/pynovisao
```
109

Fábio Prestes's avatar
Fábio Prestes committed
110 111 112 113
- Enter the folder named **[...]/pynovisao/src** or type the following command in the terminal to do so:
```
 $ cd src
```
114

Fábio Prestes's avatar
Fábio Prestes committed
115 116 117 118 119 120 121 122
- Next, type the following command if you desire to run it using Python 2.7:
```
 $ python main.py
```
- Or, should you want to run it using Python 3.6:
```
 $ python3 main.py
```
123

Fábio Prestes's avatar
Fábio Prestes committed
124
Now you are able to run Pynovisão!
Fábio Prestes's avatar
Fábio Prestes committed
125
    
Fábio Prestes's avatar
Fábio Prestes committed
126
#### Other options:
127

Fábio Prestes's avatar
Fábio Prestes committed
128
- Show All options available
129

Fábio Prestes's avatar
Fábio Prestes committed
130 131 132 133 134 135 136 137 138 139
```
 $ python main.py --help
```

- Executes the program, defining the wanted classes and it's respective colours (X11 colour names)

```
 $ python main.py --classes "Soil Soy Grass LargeLeaves" --colors "Orange SpringGreen RebeccaPurple Snow"
```

Fábio Prestes's avatar
Fábio Prestes committed
140
- A Linux script exists in *[...]/pynovisao/src/util* to help divide images into training, validation and testing datasets. It has not been implemented to the main GUI. In order to use it, use the folowwing commands:
Fábio Prestes's avatar
Fábio Prestes committed
141 142 143 144 145 146

```
 $ cd src/util
 $ chmod 755 split_data.sh
 $ ./split_data -h
```
147
# Features of this software:
Fábio Prestes's avatar
Fábio Prestes committed
148

149
## File
150
### Open Image (Shortcut: Ctrl + O)
Fábio Prestes's avatar
Fábio Prestes committed
151
Opens a file selection windows and allows the user to choose a desired image to work upon.
152
### Restore Image (Shortcut: Ctrl + R)
Fábio Prestes's avatar
Fábio Prestes committed
153
Restores the selected image to it's original state.
154
### Close Image (Shortcut: Ctrl + W)
Fábio Prestes's avatar
Fábio Prestes committed
155
Closes the currently selected image.
156
### Quit (Shortcut: Ctrl + Q)
Fábio Prestes's avatar
Fábio Prestes committed
157 158
Closes Pynovisão.

159
## View
160
### Show Image Axis (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
161
Shows a X/Y axis on the Image.
162
### Show Image Toolbar (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
163
Shows a list of all the images in the selected folder.
164
### Show Log (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
165 166
Shows a log with information about the current processes and Traceback errors should they happen.

167
## Dataset
168
### Add new class (Shortcut: Ctrl + A)
Fábio Prestes's avatar
Fábio Prestes committed
169
Create a new class. This will create a new folder in the /data folder.
170
### Set Dataset Path (Shortcut: Ctrl + D)
Fábio Prestes's avatar
Fábio Prestes committed
171
Choose the folder with the desired images.
172
### Dataset Generator (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
173 174
Creates a new dataset utilizing the selected folder.

175
## Segmentation
176
### Choose Segmenter (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
177
Choose the desired segmentation method. Please research the desired method before segmenting. The Default option is SLIC.
178
### Configure (Shortcut: Ctrl + G)
Fábio Prestes's avatar
Fábio Prestes committed
179 180 181 182 183 184
Configure the parameters for the segmentation.
- Segments: Number of total segments the image should be split into.
- Sigma: How "square" the segment is.
- Compactness: How spread out across the image one segment will be. A higher compactness will result in more clearly separated borders.
- Border Color: The color of the created segments' borders. This is only visual, it will not affect the resulting segment.
- Border Outline: Will create a border for the segment borders.
185
### Execute (Shortcut: Ctrl + S)
Fábio Prestes's avatar
Fábio Prestes committed
186 187
Execute the chosen segmentation method with the desired parameters.
Once Segmented, the user can manually click on the desired segments and they will be saved in data/demo/**name-of-the-class**/**name-of-the-image**_**number-of-the-segment**.tif.
188
### Assign using labeled image (Shortcut: Ctrl + L)
Fábio Prestes's avatar
Fábio Prestes committed
189
Use a mask/bicolor image created using a labelling software (LabelMe/LabelImg) and applies it to the original/selected image, and generates all the correct segments inside such mask.
190
### Execute folder (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
191
Same as the Execute command, however it realizes the segmentation on an entire folder at once.
192
### Create .XML File (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
193 194
Will create a .xml file using the chosen segments. The .xml will be saved in data/XML/**name-of-the-image**.xml

195
## Feature Extraction
196
### Select Extractors (Shortcut: Ctrl + E)
Fábio Prestes's avatar
Fábio Prestes committed
197 198 199 200 201 202 203 204 205 206
Select the desired extractors to use. The currently available extractors are:
- Color Statistics;
- Gray-Level Co-Ocurrence Matrix;
- Histogram of Oriented Gradients;
- Hu Image Moments;
- Image Moments (Raw/Central);
- Local Binary Patterns;
- Gabor Filter Bank;
- K-Curvature Angles.
Please research what each extractor does, and choose accordingly. By default all extractors are chosen.
207
### Execute (Shortcut: Ctrl + F)
Fábio Prestes's avatar
Fábio Prestes committed
208
Execute the chosen Extractors. It will create a training.arff file in the data/demo folder.
209
### Extract Frames (Shortcut: Ctrl + V)
Fábio Prestes's avatar
Fábio Prestes committed
210 211
Will extract frames from a video. The user must choose the folder where the desired videos are, and the destination folder where the consequent frames will be extracted to.

212
## Training
213
### Choose Classifier (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
214 215 216 217 218
Choose the desired classifier to use. Only one can be chosen at a time.
- CNNKeras
- CNNPseudoLabel
- SEGNETKeras
If the user is interested in implementing it's own classifiers into Pynovisão, please go to **Implementing a new classifier in Pynovisão**
219
### Configure (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
220 221
Choose the desired parameters for the currently selected classifier.
Each classifier has it's own parameters and configurations, and therefore must be extensibly research should the desired result be achieved.
222
### Execute (Shortcut: Ctrl + T)
Fábio Prestes's avatar
Fábio Prestes committed
223 224
Train the selected classifier utilizing al the chosen parameters and the training.arff file created previously.

225
## Classification
226
### Load h5 weights (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
227
*Only used for CNN classifiers* Take a previously created weight .h5 file and use it for this classification.
228
### Execute (Shortcut: Ctrl + C)
Fábio Prestes's avatar
Fábio Prestes committed
229
Execute the current classifier over the currently selected image.
230
### Execute folder (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
231 232
Same as the previous command, however executes all the image files inside a selected folder at once.

233
## Experimenter
234
### Ground Truth (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
235
Utilizes the currently selected image as the ground truth for the experimentations.
236
### Execute Graphical Confusion Matrix (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
237
For each classifier, creates a graphic with it's confusion matrix for the choen dataset.
238
### Cross Validation (Shortcut: Ctrl + X)
Fábio Prestes's avatar
Fábio Prestes committed
239
Performs cross validation utilizing the previously experimented classifiers.
240
### Experimenter All (Shortcut: Ctrl + P)
Fábio Prestes's avatar
Fábio Prestes committed
241
Runs all Weka classifiers and experiments with them.
Fábio Prestes's avatar
Fábio Prestes committed
242

243
## XML
244
### Configure folders (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
245
Choose the target folder for the original images and the other target folder for the segments to be searched and conevrted into a .xml file.
246
### Execute Conversion (Shortcut: Not Defined)
Fábio Prestes's avatar
Fábio Prestes committed
247
Executes the conversion using the two given folders. The file with the annotations will be saved in *[...]/pynovisao/data/XML*, with the name ***image** + .xml*. 
248

249
# Implementing a new classifier in Pynovisão
250

251
In this section we shall show the steps needed to implement a new classifier into Pynovisão. As an example, we are using **Syntactic**, of type **KTESTABLE** and vocabulary size as an hyperparameter.
252

253
Inicially, you need to create a class where all the types of your classifier are in a dictionary (Key, Value). The class must be created inside *[...]/pynovisao/src/classification/*. As an example, look for the *SyntacticAlias* in *[...]/pynovisao/src/classification/syntactic_alias.py*.
254

255 256 257
The next step is creating the .py file for your classifier in your directory *[...]/pynovisao/src/classification/*, for example, *syntactic.py*.
In this newly-created file you must implement your classifier class extending the class **Classifier**, which is implemented in the file *[...]/pynovisao/src/classification/classifier.py*.
See the example below:
258 259 260

```python
#syntactic.py
261
#minimal required imports
262 263 264 265 266 267 268 269 270
from collections import OrderedDict
from util.config import Config
from util.utils import TimeUtils
from classifier import Classifier

class Syntactic(Classifier):
    """Class for syntactic classifier"""
```

271
In the contructor class you must inform default values for the parameters. In the case fo the example below, **classname** is the type of classifier and **options** is the size of the alphabet. Besides, some attributes must be inicialized: **self.classname** and **self.options**. The attribute **self.dataset** (optional) is the path to the training and testing dataset which tells the user in the GUI. Having this attribute in the class is important to get access to the dataset in any of the methods and is initialized in the method **train** discussed later.
272 273 274 275 276 277 278 279 280 281

```python
def __init__(self, classname="KTESTABLE", options='32'):

        self.classname = Config("ClassName", classname, str)
        self.options = Config("Options", options, str)
        self.dataset = None
        self.reset()
```

282
The methods **get_name**, **get_config**, **set_config**, **get_summary_config** and **must_train** have default implementations, as seen in example in *[..]/pynovisao/src/classification/classifier.py*.
283

284
The **train** method must be implemented in order to train your classifier. The **dataset** parameter is given the path to the training images. Within the method, the value of the attribute self.dataset, declared as optional in the constructor, is altered to the current training directory.
285 286 287 288 289

```python
def train(self, dataset, training_data, force = False):              
        
        dataset += '/'
290
        # Attribute which retains the dataset path.
291
        self.dataset = dataset 
292
  	    # The two tests below are default.
293 294 295 296 297 298 299
        
        if self.data is not None and not force:
            return 
        
        if self.data is not None:
            self.reset()
		
300
	   # Implement here your training.
301 302
```

303
The **classify** method must be implemented should you want your classifier to be able to predict classes for images. The **dataset** parameter is given the training images, and **test_dir** is given the temporary folder path created by Pynovisão, where the testing images are located. This folder is created within the **dataset** directory and, to acesss it, just concatenate **dataset** and **test_dir** as show in the example below. The parameter test_data is a .arff file with data for the testing images.
304
 
305
 This method must return a list containing all the predicted classes by the classifier. E.g.: [‘weed’,’weed’,’target_stain’, ‘weed’]
306 307 308 309

```python
def classify(self, dataset, test_dir, test_data):
      
310
	   # Directory retaining the testing images.
311 312
       path_test = dataset + '/' + test_dir + '/'        
        
313
       # Implement heere the prediction algorithm for your classifier.
314
 
315
       return # A list with the predicted classes
316 317
```

318 319
The **cross_validate** must be implemented and return a string (info) with the metrics.
Obs.: The attribute **self.dataset**, updated in **train**, can be used in **cross_validate** to access the training images folder.
320 321 322 323 324 325

```python
def cross_validate(self, detail = True):
        start_time = TimeUtils.get_time()        
        info =  "Scheme:\t%s %s\n" % (str(self.classifier.classname) , "".join([str(option) for option in self.classifier.options]))
	  
326
	   # Implement here the cross validation.
327 328
	   return info
```
329
The **reset** method must also be implemented in default form, as seen below.
330 331 332 333 334 335 336

```python
def reset(self):
        self.data = None
        self.classifier = None
```

337 338 339
After implementing your classifier, you must configure it in Pynovisão by modifying **[...]/pynovisao/src/classification/__init__.py**.

Should utility classes be necessary, they must be created in **[...]/pynovisao/src/util/**. They must also be registered as modules in **[...]/pynovisao/src/util/__init__.py**.
340 341


342
Should any problem related to the number of processes arise, add these two variables in your terminal:
343

344 345
```
export OMP_NUM_THREADS=**number of threads your cpu has**
346
export KMP_AFFINITY="verbose,explicit,proclist=[0,3,5,9,12,15,18,21],granularity=core"
347
```
348