LITMIR.BIZ

      where

      (2.14)

      Gini index: This is an impurity‐based criterion that measures the divergence between the probability distributions of the target attribute’s values. The Gini (G) index is defined as

      (2.15)

      Consequently, the evaluation criterion for selecting the attribute a_i is defined as the Gini gain (GG):

      (2.16)

      Likelihood ratio chi‐squared statistics: The likelihood ratio (lr) is defined as

      (2.17)

      This ratio is useful for measuring the statistical significance of the information gain criteria. The zero hypothesis (H₀) is that the input attribute and the target attribute are conditionally independent. If H₀ holds, the test statistic is distributed as χ² with degrees of freedom equal to (dom(a_i) − 1) · (dom(y) − 1).

Normalized impurity‐based criterion: The impurity‐based criterion described above is biased toward attributes with larger domain values. That is, it prefers input attributes with many values over attributes with less values. For instance, an input attribute that represents the national security number will probably get the highest information gain. However, adding this attribute to a decision tree will result in a poor generalized accuracy. For that reason, it is useful to “normalize” the impurity‐based measures, as described in the subsequent paragraphs.

      Gain ratio ( gr): This ratio “normalizes” the information gain (ig) as follows: gr(a_i, S) = ig(a_i, S)/e(a_i, S). Note that this ratio is not defined when the denominator is zero. Also, the ratio may tend to favor attributes for which the denominator is very small. Consequently, it is suggested in two stages. First, the information gain is calculated for all attributes. Then, taking into consideration only attributes that have performed at least as well as the average information gain, the attribute that has obtained the best ratio gain is selected. It has been shown that the gain ratio tends to outperform simple information gain criteria both from the accuracy aspect as well as from classifier complexity aspect.

      Distance measure: Similar to the gain ratio, this measure also normalizes the impurity measure. However, the method used is different: