Dgdrxrt

I had a discussion with a friend and we were talking about the advantages of random forest over linear regression.

At some point, my friend said that one of the advantages of the random forest over the linear regression is that it takes automatically into account the combination of features.

By this he meant that if I have a model with

• Y as a target


• X, W, Z as the predictors

then the random forests tests also the combinations of the features (e.g. X+W) whereas in linear regression you have to build these manually and insert them at the model.

I am quite confused, is this true?

Also if it true then is it about any kind of combination of features (e.g. X*W, X+W+Z etc) or only for some specific ones (e.g. X+W)?

I think it is true. Tree based algorithms especially the ones with multiple trees has the capability of capturing different feature interactions. Please see this article from xgboost official documentation and this discussion. You can say it's a perk of being a non parametric model (trees are non parametric and linear regression is not). I hope this will shed some light on this thought.

I would say it is not true as Random forests which are made up of decision trees does perform feature selection but they do not perform feature engineering (feature selection is different from feature engineering). Decision trees use a metric called Information gain (which is total entropy minus the weighted entropy) as per which useful features are separated from bad features. Simply to say whichever feature exhibit the highest information gain on this iteration is chosen as the node on which the tree on this iteration is split or you can say which feature reduces the entropy(aka randomness) the most in this iteration is chosen as the node upon which the tree is split on this iteration. So if you data is text, trees are split upon words. If your data is real valued numbers, tree is split upon that. Hope it helps

For more details check this

The statement "it tests combination of features" is not true. It tests individual features. However, a tree can approximate any continuous function $f$ over training points, since it is a universal approximator just like neural networks.

In Random Forest (or Decision Tree, or Regression Tree), individual features are compared to each other, not a combination of them, then the most informative individual is peaked to split a leaf. Therefore, there is no notion of "better combination" in the whole process.

Furthermore, Random Forest is a bagging algorithm which does not favor the randomly-built trees over each other, they all have the same weight in the aggregated output.

It is worth noting that "Rotation forest" first applies PCA to features, which means each new feature is a linear combination of original features. However, this does not count since the same pre-processing can be used for any other method too.

EDIT:

@tam provided a counter-example for XGBoost, which is not the same as Random Forest. However, the issue is the same for XGBoost. Its learning process comes down to splitting each leaf greadily based on a single feature instead of selecting the best combination of features among a set of combinations, or the best tree among a set of trees.

From this explanation, you can see that The Structure Score is defined for a tree (which is a function) based on the first- and second-order derivatives of loss function in each leaf $j$ ($G_j$ and $H_j$ respectively) summed over all $T$ leaves, i.e.
$$textobj^*=-frac12 sum_j=1^TfracG_jH_j + lambda + gamma T$$
However, the optimization process greedily splits a leaf using the best individual feature that gives the highest gain in $textobj^*$.

A tree $t$ is built by greedily minimizing the loss, i.e. branching on the best individual feature, and when the tree is built, process goes to create the next tree $t+1$ in the same way, and so on.

Here is the key quote from XGBoost paper:

This score is like the impurity score for evaluating decision trees,
except that it is derived for a wider range of objective functions [..] Normally it is impossible to enumerate all the possible tree
structures q. A greedy algorithm that starts from a single leaf and
iteratively adds branches to the tree is used instead.

In summary:

Although a tree represents a combination of features (a function), but
none of XGBoost and Random Forest are selecting between functions.
They build and aggregate multiple functions by greedily favoring individual
features.