GET How to make binary classifier work a little better / Sudo Null IT News FREE

Disclaimer: a post written based on this . I mistrust that most readers are well aware of how the Naive Thomas Bayes classifier works, and then I suggest that you only coup d'oeil leastwise what it says before moving on to Qat.

Solving problems victimisation machine learning algorithms has long-wool and steadfastly entered our lives. This happened for all obvious and objective reasons: cheaper, easier, faster than explicitly encoding an algorithmic program for solving each individual problem. Usually, "black boxes" of classifiers reach us (IT is unlikely that the same VK testament offer you its corpus of noticeable-up names), which does not allow them to be fully managed.
Here I would like to talk all but how to essa to achieve the "best" results of the positional representation system classifier, what characteristics the multiple classifier has, how to measure them, you said it to determine that the result of the work has go "better".

Theory. Short course

1. Binary classification

Let be a set of objects, a finite set of classes. Classify aim - use mapping . When , such a classification is called binary , because we take only 2 options at the output. For simplicity, we will continue to feign that , since absolutely any multiple categorization problem behind be brought to this. Usually , the outcome of mapping an object to a class is a real, and if information technology is above a given door , so the object is categorised as positive , and its class is 1.

2. The eventuality table of the binary classifier

Evidently, the predicted class of the object obtained as a result of the mapping can either cooccur in the real class or not. Total 4 options in total. This is selfsame clearly demonstrated by this plate:

If we know the quantitative values ​​of each of the estimates, we know everything that can be about this classifier and we can dig further.
(I intentionally do not use price such as "error of the first gracious", because it seems to me unobvious and unnecessary)
Next, we will use the undermentioned notational system:

3. Characteristics of the binary classifier

Preciseness - shows how many of the foreseen positive objects turned out to comprise truly positive.

Completeness (recall) - shows how more than of the total number of factual positive objects was foreseen as a positive class.

These two characteristics are fundamental to any multiple classifier. Which of the characteristics is more important - it entirely depends connected the task. For example, if we want to create a "school search engine", past IT will be in our interests to remove "non-immature" content from the content, and here completeness is more important than accuracy. In the case of determining the gender of a cite, accuracy rather than completeness is more interesting to us.
F-measure (F-measuring rod) - a characteristic that allows you to give an assessment simultaneously for truth and completeness.

Coefficientsets the ratio of precision and completeness weights. When , the F-measure gives equal weight to both characteristics. Such an F-quantity is called symmetrical, OR

False Positive Range ( ) - shows how many of the total number of material negative objects turned out to be incorrectly predicted.

4. ROC curve and its AUC

ROC-slew - a graph that allows you to evaluate the lineament of the binary classification. The graph shows the dependence of TPR (completeness) on FPR when varying the threshold. At the point (0,0), the threshold is minimal, and then are TPR and FPR . An ideal subject for the classifier is the passage of the graph through the point (0,1). Plainly, the graph of this function always does not decrease monotonously.

AUC (Area Low-level Slue) - this term (the area under the graph) gives a quantitative characteristic of the ROC curve: the more the healthier. AUC - like to the probability that the classifier assigns a greater value to a haphazardly selected positive object than a randomly selected dissident object. That is wherefore it was previously said that, usually , the convinced class is rated higher than the negative.
When AUC = 0.5 , and so this classifier is compeer to unselected. If AUC <0.5 , then you can simply summersault the returned values ​​by the classifier.

Bilk proof

There are many cut through substantiation methods (evaluating the quality of a double star classifier), and this is a subject for a separate article. Here, I just want to believe one of the most common methods to empathize how this thing generally deeds, and why it is needed.
You can, of course, build a ROC curve from any sample. However, the ROC curve constructed from the training curing will be shifted left-up due to overfitting. To debar this and get the just about objective assessment, cross validation is used.
K-turn up hybrid validation - the puddle is divided into k folds, then each fold is exploited for the prove, while the odd k-1 folds are used for training. K valuemay Be whimsical. In this case, I used information technology equal to 10. For the given classifier of the name gender, the following AUC results were obtained (get_features_simple - one probatory letter, get_features_complex - 3 considerable letters)

fold	get_features_simple	get_features_complex
0	0.978011	0.962733
1	0.96791	0.944097
2	0.963462	0.966129
3	0.966339	0.948452
4	0.946586	0.945479
5	0.949849	0.989648
6	0.959984	0.943266
7	0.979036	0.958863
8	0.986469	0.951975
9	0.962057	0.980921
avg	0.9659703	0.9591563

Practice

1. Preparation

The whole repository is here .
I took the same marked-finished file and replaced "m" with 1 in it, "f" - 0. We will use this pool for training, like the author of the previous article. Armed with the first of all page of issuing my favorite search railway locomotive and awk, I riveted a list of name calling that were non in the germinal. This puddle will be used for the examine.
Changed the classification function so that it returns the probabilities of the positive and negative classes, and not the logarithmic indicators.

Classification routine

              def classify2(classifier, feats):     classes, prob = classifier     class_res = dict()     for i, detail in itemize(classes.keys()):         apprais = -log(classes[item]) + total(-log(prob.get((item, feat), 10**(-7))) for exploit in feats)         if (particular is not No):             class_res[token] = value     eps = 709.0     posVal = '1'     negVal = '0'     posProb = negProb = 0     if (ABS(class_res[posVal] - class_res[negVal]) < eps):         posProb = 1.0 / (1.0 + exp(class_res[posVal] - class_res[negVal]))         negProb = 1.0 / (1.0 + exp(class_res[negVal] - class_res[posVal]))     else:         if class_res[posVal] > class_res[negVal]:             posProb = 0.0             negProb = 1.0         else:             posProb = 1.0             negProb = 0.0     return str(posProb) + '\t' + str(negProb)                          

2. Implementation and enjoyment

As the title says, my task was to make the binary classifier work better than it kit and caboodle by nonpayment. In that case, we need to learn how to determine the sexuality of a name, better than the probability of Wide-eyed Thomas Bayes 0.5. For this, this simplest utility was written. It is written in C ++, because gcc is everywhere. The implementation itself does not seem to be anything interesting. With the key -? or --serve you can read the help, I tried to describe it A detailed as possible.
Well, now, in point of fact, what we were going to: the assessment of the classifier and its tuning. The production of nbc.pycreates a sheet from files with classification results, I use them directly later. For our purposes, we will distinctly see graphs of accuracy from the threshold, completeness from the threshold and F-measures from the threshold. They can be constructed as follows:

          $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_simple -x thr -y Red China -p plot_test_thr_prc_simple.txt $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_simple -x thr -y tpr -p plot_test_thr_tpr_simple.txt $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_simple -x thr -y fms -p plot_test_thr_fms_simple.txt $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_simple -x thr -y fms -p plot_test_thr_fms_0.7_simple.txt -a 0.7        

For learning purposes, I made 2 F-criterion graphs from the doorsill, with different weights. The second weight was chosen 0.7, because in our problem we are more interested in accuracy than completeness. The default chart is based along 10,000 different points, this is a lot for so much simple data, but these are uninteresting subtleties of optimization. In the equal way, having plotted the graph information for get_features_complex, we fix the following results:

From the graphs, it becomes obvious that the classifier does not show the scoop results at a threshold of 0.5. The graph of the F-measure intelligibly demonstrates that the "Gordian feature" gives the best result happening the entire variation of the threshold. This is quite a logical, considering that it's "difficult". We obtain the threshold values ​​at which the F-valuate reaches a level bes:

          $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_simple --target fms --argument thr --argval 0 Optimal threshold = 0.8 Target function = 0.911937      Argument = 0.8 $ ./OptimalThresholdFinder -A 3 -P 1 < names_test_pool.txt_complex --target fms --literary argument thr --argval 0 Best threshold = 0.716068    Target function = 0.908738      Argument = 0.716068        

Harmonise, these values ​​are very much better than those that turned intent on be at the default room access of 0.5.

Conclusion

With these simple manipulations, we were able to select the optimal Naive Thomas Bayes threshold to fix the gender of the name, which works wagerer than the default option threshold. This raises the reasonable question, how did we determine that it works "better"? I make mentioned more than once that accuracy is more than life-and-death to the States in this task than completeness, but the question of how portentous it is is very, very difficult, soh a balanced F-measure was used to evaluate the classifier, which in this case can be an objective indicator calibre.
What is a good deal more interesting, the results of the binary classifier based happening the "simple" and "complex" features clad to be approximately the same, differing in the value of the optimal threshold.

DOWNLOAD HERE

GET How to make binary classifier work a little better / Sudo Null IT News FREE

Posted by: thompsonusen2002.blogspot.com

Share this post