The following node is available in the Open Source KNIME predictive analytics and data mining platform version 2.7.1. Discover over 1000 other nodes, as well as enterprise functionality at
http://knime.com.
Tree Ensemble Learner
Learns an ensemble of decision trees (such as random forest variants). Each of the decision tree models is
learned on a different set of rows (records) and/or a different set of columns (describing attributes), whereby
the latter can also be a bit-vector descriptor (e.g. molecular fingerprint). The output model describes an
ensemble of decision tree models and is applied in the corresponding predictor node using a simply majority
vote.
The following configuration settings learn a model that is similar to the
Random Forest™ classifier described by
Leo Breiman and Adele Cutler:
- Tree Options - Split Criterion: Gini Index
- Tree Options - Limit number of levels (tree depth): unlimited
- Tree Options - Minimum node size: unlimited
- Ensemble Configuration - Number of models: Arbitrary (Random Forest arguably does not overfit)
- Ensemble Configuration - Data Sampling: Use all rows (fraction = 1) but choose sampling
with replacement (bootstrapping)
- Ensemble Configuration - Attribute Sampling: Sample using a different set of attributes for each tree
node split; usually square root of number of attributes but can vary
Experiments have shown the results on different data sets are very similar to the
Random Forest implementation available in R.
Known differences are in the missing value handling (currently not available in this node) and the split
creation for nominal attributes (the original Random Forest classifier uses binary nominal splits; this
implementation creates child nodes for each possible split attribute). Note also, this implementation keeps
additional statistics for each split (class distribution), which can be seen in the view but which can also
increase the model size (in terms of MB on disc) quite a lot.
The decision tree construction takes place in main memory (all data and all models are kept in memory).
Note: The node is currently under development; models used with the current version are
likely not backward compatible.
Dialog Options
Attribute Selection
- Target Column
-
Select the column containing the value to be learned. Rows with missing values in this column will be ignored
during the learning process.
- Attribute Selection
-
Select the attributes to use learn the model. Two variants are possible.
Fingerprint attribute uses the different bit positions in the selected bit vector as learning
attributes (for instance a bit vector of length 1024 is expanded to 1024 binary attributes). All bit vectors
in the selected column must have the same length.
Column attributes are nominal and numeric columns used as descriptors. Numeric columns are split in a
<= fashion; nominal columns are currently split by creating child nodes for each of the values.
- Ignore columns without domain information
-
If selected, nominal columns with no domain information are ignored (as they likely have too many possible
values anyway).
- Enable Hightlighting (#patterns to store)
-
If selected, the node stores the selected number of rows and allows highlighting them in the node view.
Tree Options
- Split Criterion
- Choose the
split criterion here. Gini is usually a good choice and is used in CART (as described by Breiman et al);
information gain is used in C4.5; information gain ratio normalizes the standard information gain by the split
entropy to overcome some unfair preference for nominal splits with many child nodes.
- Use mid points splits (only for numeric attributes)
-
Uses for numerical splits the middle point between two class boundaries. If unselected the split attribute value
is the smaller value with "<=" relationship.
- Limit number of levels (tree depth)
-
Number of tree levels to be learned. For instance, a value of 1 would only split the (single) root node
(decision stump).
- Minimum split node size
- Minimum number of records in a decision tree node so that another split
is attempted. Note, this option does not make any implications on the minimum number of records in a
terminal node. If enabled, this number needs to be at least twice as large as the minimum child node size
(as otherwise for binary splits one of the two children would have less records than specified).
- Minimum child node size
- Minimum number of records in child nodes. It can be at most half of
the minimum split node size (see above). Note, this parameter is currently ignored for nominal splits.
- Use fixed root attribute
-
If selected the chosen column will be used as root split attribute in all decision trees -- even if the column
is not in the attribute sample (see below).
Ensemble Configuration
- Number of models
-
The number of decision trees to learn. A "reasonable" value can range from very few (say 10) to many thousands
for small data sets with few target category values.
- Data Sampling (Rows)
-
The sampling of the data rows for each individual tree: If disabled each tree learner gets the full data set,
otherwise each tree is learned with a different data sample. A data fraction of 1 (=100%) chosen
"with replacement" is called bootstrapping. For sufficiently large data sets this bootstrap sample contains
about 2/3 different data rows from the input, some of which replicated multiple times. Rows which are not used
in the training of a tree are called out-of-bag (see below).
- Attribute Sampling (Columns)
-
Defines the sampling of attributes to learn an individual tree. This can either be a function based on the
number of attributes (linear fraction or square root) or some absolute value. The latter can be used in
conjunction with flow variables to inject some other value derived from the number of attributes (e.g. Breiman
suggests to start with the square root of number of attributes but also try to double/half that number).
- Attribute Selection
-
Use same set of attributes for each tree describes that the attributes are sampled once for each tree
and this sample is then used to construct the tree.
Use different set of attributes for each tree node samples a different set of candidate attributes in
each of the tree nodes from which the optimal one is chosen to perform the split.
- Use static random seed
-
Choose a seed to get reproducible results.
Ports
Input Ports
0 |
The data to learn from. It must contain at least one nominal target column and
either a fingerprint (bitvector) column or another numeric or nominal column.
|
Output Ports
0 |
The input data with the out-of-bag error estimates, i.e. for each input row the majority vote of all models that
did not use the row in the training. If the entire data was used to train the individual models then this
output will contain the input data with missing prediction and confidence values. The appended columns are
equivalent to the columns appended by the corresponding predictor node. There is one additional column
model count, which contains the number of models used for the voting (number of models not using the row
throughout the learning.)
|
1 |
A statistics table on the attributes used in the different tree learners. Each row represents one training
attribute with these statistics: #splits (level x) as the number of models, which use the attribute as
split on level x (with level 0 as root split); #candidates (level x) is the number of times an
attribute was in the attribute sample for level x (in a random forest setup these samples differ from
node to node). If no attribute sampling is used #candidates is the number of models. Note, these numbers
are uncorrected, i.e. if an attribute is selected on level 0 but is also in the candidate set of level 1 (but
will not be split on level 1 because it has been split one level up), the #candidate number will still count
the attribute as candidate.
|
2 |
The trained model. |
Views
- Tree Views
- An decision tree viewer for all the trained models. Use the spinner to iterate
through the different models.
This node is contained in Decisions Tree Ensembles for KNIME
provided by KNIME GmbH, Konstanz, Germany.