Random forests and boosting

Why it matters

For tabular business data (customers, orders, campaigns) tree ensembles are usually the most accurate models you can train, and they need little preparation. Gradient-boosted trees (XGBoost, LightGBM, CatBoost) win most competitions on spreadsheet-style data and run inside countless churn, fraud and pricing systems. The idea behind them, many imperfect models combined, also shows up in LLM work as voting and self-consistency.

What you'll be able to do

• Explain why averaging many models beats one
• Describe bagging and random forests: trees on resampled data with random features
• Describe boosting: trees added one by one to fix the remaining errors
• Compare a single tree, a forest and a boosted model on held-out data, and read feature importances

Before this lesson

• Decision trees
• Bias and variance

Next in this phase: Nearest neighbors