The way we can circumvent this issue is by fitting our model in {tidymodels} using cross-validation so that we can tune the mtry <\/em><\/strong>and trees <\/strong><\/em>values. Once we have the optimum values for these hyper-parameters we will use the {randomForest}<\/strong> package and build a new model using those values. We will then make our predictions with this model on new data.<\/p>\n NOTE: <\/strong>I’m not 100% certain this is the best way to approach this problem inside or outside of {tidymodels}. If someone has a better solution, please drop it into the comments section or shoot me an email!<\/em><\/p>\n Load Packages & Data<\/strong><\/span><\/p>\n We will use the mtcars<\/strong> data set and try and predict the car’s mpg from the disp, hp, wt, qsec, drat, gear, and carb columns.<\/p>\n Split data into cross-validation sets Specify the model type & build a tuning grid<\/strong><\/span><\/p>\n The model type will be a random forest regression using the randomForest <\/strong>engine.<\/p>\n The tuning grid will be a vector of values for both mtry <\/strong>and trees<\/strong> to provide the model with options to try as it tunes the hyper-parameters.<\/p>\n Create a model recipe & workflow<\/strong><\/span><\/p>\n Fit and tun the model<\/strong><\/span><\/p>\n View the model performance and identify the best model<\/strong><\/span><\/p>\n Extract the optimal mtry and trees values for minimizing<\/strong><\/span> rmse<\/strong><\/p>\n Re-fit the model using the optimal mtry and trees values<\/strong><\/span><\/p>\n Now that we’ve identified the hyper-parameters that minimize rmse, we will re-fit the model using the {randomForest} <\/strong>package, so that we can get predictions for all of the trees, and specify the mtry <\/strong> and ntree <\/strong>values that were extracted from the {tidymodels} model within the function.<\/p>\n Create new data and make predictions<\/strong><\/span><\/p>\n When making the predictions we have to make sure to pass the argument predict.all = TRUE<\/strong>.<\/p>\n ## New data ## Make Predictions What do predictions look like?<\/strong><\/span><\/p>\n Because we requested predict all, we have the ability to see a prediction for each of the 725 trees that were fit. Below we will look at the first and last 6 predictions of the 725 individual trees for the first observation in our new data set.<\/p>\n What do predictions look like<\/strong><\/span>?<\/p>\n Taking the mean of the 725 predictions will produce the predicted value for the new observation, using the wisdom of the crowds. Similarly, the standard deviation of these 725 predictions will give us a sense for the variability of the weak learners. We can use this information to produce our confidence intervals. We calculate our confidence intervals as the standard deviation of predictions multiplied by the t-critical value, which we calculate from a t-distribution with the degrees of freedom equal to 725 – 1.<\/p>\n Make a prediction with confidence intervals for all of the observations in our new data<\/strong><\/span><\/p>\n First we will make a single point prediction (the average, wisdom of the crowds, prediction) and then we will write a for()<\/strong> loop to create the lower and upper 95% Confidence Intervals using the same approach as above.<\/p>\n View the new observations with their predictions and create a plot of the predictions versus the actual data<\/strong><\/span><\/p>\n The three columns on the right show us the predicted miles per gallon and the 95% confidence interval for each of the five new observations. The plot shows us the point prediction and confidence interval along with the actual mpg (in red), which we can see falls within each of the ranges.<\/p>\n\r\n## Load packages\r\nlibrary(tidymodels)\r\nlibrary(tidyverse)\r\nlibrary(randomForest)\r\n\r\n## load data\r\ndf <- mtcars %>%\r\n select(mpg, disp, hp, wt, qsec, drat, gear, carb)\r\n\r\nhead(df)\r\n<\/pre>\n
\n<\/strong><\/span>
\nThis is a small data set, so rather than spending data by splitting it into training and testing sets, I’m going to use cross-validation on all of the available data to fit the model and tune the hyper-parameters<\/p>\n\r\n## Split data into cross-validation sets\r\nset.seed(5)\r\ndf_cv <- vfold_cv(df, v = 5)\r\n<\/pre>\n
\r\n## specify the random forest regression model\r\nrf_spec <- rand_forest(mtry = tune(), trees = tune()) %>%\r\n set_engine("randomForest") %>%\r\n set_mode("regression")\r\n\r\n## build a tuning grid\r\nrf_tune_grid <- grid_regular(\r\n mtry(range = c(1, 7)),\r\n trees(range = c(500, 800)),\r\n levels = 5\r\n)\r\n<\/pre>\n\r\n## Model recipe\r\nrf_rec <- recipe(mpg ~ ., data = df)\r\n\r\n## workflow\r\nrf_workflow <- workflow() %>%\r\n add_recipe(rf_rec) %>%\r\n add_model(rf_spec)\r\n<\/pre>\n
\r\n## set a control function to save the predictions from the model fit to the CV-folds\r\nctrl <- control_resamples(save_pred = TRUE)\r\n\r\n## fit model\r\nrf_tune <- tune_grid(\r\n rf_workflow,\r\n resamples = df_cv,\r\n grid = rf_tune_grid,\r\n control = ctrl\r\n)\r\n\r\nrf_tune\r\n<\/pre>\n
![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2022\/03\/Screen-Shot-2022-03-22-at-9.10.25-PM.png\")
<\/a><\/p>\n\r\n## view model metrics\r\ncollect_metrics(rf_tune)\r\n\r\n## Which is the best model?\r\nselect_best(rf_tune, "rmse")\r\n\r\n## Look at that models performance\r\ncollect_metrics(rf_tune) %>%\r\n filter(mtry == 4, trees == 725)\r\n<\/pre>\n
<\/a>
\nHere we see that the model with the lowest root mean squared error (rmse) has an mtry = 4<\/strong> and trees = 725<\/strong>.<\/p>\n\r\n## Extract the best mtry and trees values to optimize rmse\r\nm <- select_best(rf_tune, "rmse") %>% pull(mtry)\r\nt <- select_best(rf_tune, "rmse") %>% pull(trees)\r\n\r\nm\r\nt\r\n<\/pre>\n
<\/a><\/p>\n\r\n## Re-fit the model outside of tidymodels with the optimized values\r\nrf_refit <- randomForest(mpg ~ ., data = df, mtry = m, ntree = t)\r\nrf_refit\r\n<\/pre>\n
\nset.seed(859)
\nrow_id <- sample(1:nrow(df), size = 5, replace = TRUE)
\nnewdat <- df[row_id, ]
\nnewdat<\/p>\n
\npred.rf <- predict(rf_refit, newdat, predict.all = TRUE)
\npred.rf<\/p>\n\r\n## Look at all 725 predictions for the first row of the data\r\nhead(pred.rf$individual[1, ])\r\ntail(pred.rf$individual[1, ])\r\n<\/pre>\n
<\/a><\/p>\n\r\n# Average prediction -- what the prediction function returns\r\nmean(pred.rf$individual[1, ])\r\n\r\n# SD of predictions\r\nsd(pred.rf$individual[1, ])\r\n\r\n# get t-critical value for df = 725 - 1\r\nt_crit <- qt(p = 0.975, df = t - 1)\r\n\r\n# 95% CI\r\nmean(pred.rf$individual[1, ]) - t_crit * sd(pred.rf$individual[1, ])\r\nmean(pred.rf$individual[1, ]) + t_crit * sd(pred.rf$individual[1, ])\r\n<\/pre>\n
<\/a><\/p>\n\r\n## Now for all of the predictions\r\nnewdat$pred_mpg <- predict(rf_refit, newdat)\r\n\r\n## add confidence intervals\r\nlower <- rep(NA, nrow(newdat))\r\nupper <- rep(NA, nrow(newdat))\r\n\r\nfor(i in 1:nrow(newdat)){\r\n lower[i] <- mean(pred.rf$individual[i, ]) - t_crit * sd(pred.rf$individual[i, ])\r\n upper[i] <- mean(pred.rf$individual[i, ]) + t_crit * sd(pred.rf$individual[i, ])\r\n}\r\n\r\nnewdat$lwr <- lower\r\nnewdat$upr <- upper\r\n<\/pre>\n
\n<\/strong><\/span><\/p>\n\r\n## Look at the new observations, predctions and confidence intervals and plot the data\r\n## new data\r\nnewdat\r\n\r\n## plot\r\nnewdat %>%\r\n mutate(car_type = rownames(.)) %>%\r\n ggplot(aes(x = pred_mpg, y = reorder(car_type, pred_mpg))) +\r\n geom_point(size = 5) +\r\n geom_errorbar(aes(xmin = lwr, xmax = upr),\r\n width = 0.1,\r\n size = 1.3) +\r\n geom_point(aes(x = mpg),\r\n size = 5,\r\n color = "red") +\r\n theme_minimal() +\r\n labs(x = "Predicted vs Actual MPG",\r\n y = NULL,\r\n title = "Predicted vs Actual (red) MPFG from Random Forest",\r\n subtitle = "mpg ~ disp + hp + wt + qsec + draft + gear + carb")\r\n\r\n<\/pre>\n
<\/a><\/p>\n