{"id":3510,"date":"2025-04-07T02:48:10","date_gmt":"2025-04-07T02:48:10","guid":{"rendered":"http:\/\/optimumsportsperformance.com\/blog\/?p=3510"},"modified":"2025-04-07T12:13:21","modified_gmt":"2025-04-07T12:13:21","slug":"xgboost-tuning-tutorial-in-xgboost-and-tidymodels","status":"publish","type":"post","link":"https:\/\/optimumsportsperformance.com\/blog\/xgboost-tuning-tutorial-in-xgboost-and-tidymodels\/","title":{"rendered":"XGBOOST Tuning – Tutorial in {xgboost} and {tidymodels}"},"content":{"rendered":"

A colleague recently asked me about XGBOOST (Extreme Gradient Boosting) models so I figured I’d put together a short tutorial of using XGBOOST both with the {xgboost<\/strong>} package and within the {tidymodels<\/strong>} environment.<\/p>\n

Like all machine learning models, XGBOOST, has a number of different knobs and levers we can twist, push, and pull in order to tune the model and identify the best parameters for learning from the data. One can get a look at these parameters by loading the {xgboost<\/strong>} package and then typing ?xgboost<\/strong><\/em> into the console.<\/p>\n

XGBOOST falls in the class of tree-based models. It is an effective machine learning algorithm for identifying patterns in data and it can be used for regression or classification problems.<\/p>\n

When fitting the model in {tidymodels<\/strong>} it is pretty straight forward as far as the {tidymodels<\/strong>} syntax goes. However, if fitting models with the {xgboost<\/strong>} package, there are a number of considerations we need to make with the data. For example. XGBOOST can only use numeric data, so any categorical features will need to be one-hot-encoded. Additionally, the data cannot be in a data frame form, but rather needs to be turned into a matrix.<\/p>\n

Let’s jump in!<\/p>\n

(NOTE:<\/strong><\/em> All code is available on my Github page<\/a><\/span><\/strong>.)<\/em><\/p>\n

Load Data & Packages<\/strong><\/span><\/p>\n

We will keep the data simple and use the mtcars<\/strong> data from base R. XGBOOST is well suited for complex data with a lot of interactions and non-linearity, but I’d like to use a simple data set so that the models can run fast for you and you get the gist of fitting them. Tuning all of the parameters can take a lot of time if you are dealing with a big data set.<\/p>\n

\"\"<\/a><\/p>\n

Our goal is to predict mpg<\/em><\/strong> using the other features in the data.<\/p>\n

\"\"<\/a><\/p>\n

{xgboost} for a regression problem<\/strong><\/span><\/p>\n

First we split our data into a training and testing data set.<\/p>\n

\r\nset.seed(2025)\r\nN <- nrow(mtcars)\r\nids <- sample(x = 1:N, size = floor(N * 0.7), replace = FALSE)\r\ntrain <- mtcars[ids, ]\r\ntest <- mtcars[-ids, ]\r\n<\/pre>\n
Prepare the data<\/em><\/p>\n
For the {xgboost<\/strong>} package to run we need the predictor variables to be in a data matrix and the outcome variables to be a vector. We then place the prepared data into a dense matrix so we can run our XGBOOST model.<\/p>\n
\r\ntrain_x <- data.matrix(train[, -1])\r\ntrain_y <- train$mpg\r\n\r\ntest_x <- data.matrix(test[, -1])\r\ntest_y <- test$mpg\r\n\r\nxgb_train <- xgb.DMatrix(data = train_x, label = train_y)\r\nxgb_test <- xgb.DMatrix(data = test_x, label = test_y)\r\n<\/pre>\n
Tune the model<\/em><\/p>\n
First, we walk through an example of of how the xgb.cv<\/strong><\/em> function works and what it outputs. Then we can write a loop to extract the relevant information for the purposes of the tuning the hyperparameters.<\/p>\n
Notice that we specified a variety of hyperparameters (e.g., max.depth<\/strong><\/em>, eta<\/strong><\/em>, subsample<\/strong><\/em>, and colsample_bytree<\/strong><\/em>). These aren’t all of the possible hyperparameters that can be tuned, though. Rather than hash out them all out here, you can find a list of their definitions by either using the help function in the R console, ?xgb.cv()<\/strong><\/em> or checking out THIS<\/a><\/strong><\/span> webpage.<\/p>\n
\r\nfit_example <- xgb.cv(data = xgb_train,\r\n                      params = list(booster = "gbtree",\r\n                        objective = "reg:squarederror",\r\n                        max.depth = 15,\r\n                        eta = 0.1,\r\n                        subsample = 0.1,\r\n                        colsample_bytree = 0.5),\r\n                      eval_metric = "rmse",\r\n                      nfold = 5,\r\n                      watchlist = list(train = xgb_train, test = xgb_test),\r\n                      early_stopping_rounds = TRUE,\r\n                      nrounds = 1000)\r\n<\/pre>\n
There are a number of features within the model that we can extract:<\/p>\n