{"id":2608,"date":"2022-07-25T01:13:01","date_gmt":"2022-07-25T01:13:01","guid":{"rendered":"http:\/\/optimumsportsperformance.com\/blog\/?p=2608"},"modified":"2022-11-08T03:34:05","modified_gmt":"2022-11-08T03:34:05","slug":"tidymodels-extract-model-coefficients-for-all-cross-validated-folds","status":"publish","type":"post","link":"https:\/\/optimumsportsperformance.com\/blog\/tidymodels-extract-model-coefficients-for-all-cross-validated-folds\/","title":{"rendered":"tidymodels – Extract model coefficients for all cross validated folds"},"content":{"rendered":"

As I’ve discussed previously, we sometimes don’t have enough data where doing a train\/test split makes sense. As such, we are better off building our model using cross-validation. In previous blog articles, I’ve talked about how to build models using cross-validation within the {tidymodels<\/strong>} framework (see HERE<\/a><\/strong><\/span> and HERE<\/a><\/span><\/strong>). In my prior examples, we fit the model over the cross-validation folds and then constructed the final model that we could then use to make predictions with, later on.<\/p>\n

Recently, I ran into a situation where I wanted to see what the model coefficients look like across all of the cross-validation folds. So, I decided to make a quick blog post on how to do this, in case it is useful to others.<\/p>\n

Load Packages & Data<\/strong><\/span><\/p>\n

We will use the {mtcars<\/strong>} package from R and build a regression model, using several independent variables, to predict miles per gallon (mpg).<\/p>\n

\r\n### Packages -------------------------------------------------------\r\n\r\nlibrary(tidyverse)\r\nlibrary(tidymodels)\r\n\r\n### Data -------------------------------------------------------\r\n\r\ndat <- mtcars dat %>%\r\n  head()\r\n<\/pre>\n
 <\/p>\n
Create Cross-Validation Folds of the Data<\/strong><\/span><\/p>\n
I’ll use 10-fold cross validation.<\/p>\n
\r\n### Modelling -------------------------------------------------------\r\n## Create 10 Cross Validation Folds\r\n\r\nset.seed(1)\r\ncv_folds <- vfold_cv(dat, v = 10)\r\ncv_folds\r\n<\/pre>\n
\"\"<\/a><\/p>\n
Specify a linear model and set up the model formula<\/strong><\/span><\/p>\n
\r\n## Specify the linear regression engine\r\n## model specs\r\nlm_spec <- linear_reg() %>%\r\n  set_engine("lm") \r\n\r\n\r\n## Model formula\r\nmpg_formula <- mpg ~ cyl + disp + wt + drat\r\n<\/pre>\n
Set up the model workflow\u00a0 and fit the model to the cross-validated folds<\/strong><\/span><\/p>\n
\r\n## Set up workflow\r\nlm_wf <- workflow() %>%\r\n  add_formula(mpg_formula) %>%\r\n  add_model(lm_spec) \r\n\r\n## Fit the model to the cross validation folds\r\nlm_fit <- lm_wf %>%\r\n  fit_resamples(\r\n    resamples = cv_folds,\r\n    control = control_resamples(extract = extract_model, save_pred = TRUE)\r\n  )\r\n<\/pre>\n
\"\"<\/a><\/p>\n
Extract the model coefficients for each of the 10 folds (this is the fun part!)<\/strong><\/span><\/p>\n
Looking at the lm_fit<\/strong> output above, we see that it is a tibble consisting of various nested lists. The id<\/strong> column indicates which cross-validation fold the lists in each row pertain to. The model coefficients for each fold are stored in the .extracts<\/strong> column of lists. Instead of printing out all 10, let’s just have a look at the first 3 folds to see what they look like.<\/p>\n
\r\nlm_fit$.extracts %>% \r\n  .[1:3]\r\n<\/pre>\n
\"\"<\/a><\/p>\n
There we see in the .extracts<\/strong> column, <lm<\/strong>> <\/strong>indicating the linear model for each fold. With a series of unnesting we can snag the model coefficients and then put them into a tidy format using the {broom<\/strong>} package. I’ve commented out each line of code below so that you know exactly what is happening.<\/p>\n
\r\n# Let's unnest this and get the coefficients out\r\nmodel_coefs <- lm_fit %>% \r\n  select(id, .extracts) %>%                    # get the id and .extracts columns\r\n  unnest(cols = .extracts) %>%                 # unnest .extracts, which produces the model in a list\r\n  mutate(coefs = map(.extracts, tidy)) %>%     # use map() to apply the tidy function and get the coefficients in their own column\r\n  unnest(coefs)                                # unnest the coefs column you just made to get the coefficients for each fold\r\n\r\nmodel_coefs\r\n<\/pre>\n
\"\"<\/a><\/p>\n
Now that we have a table of estimates, we can plot the coefficient estimates and their 95% confidence intervals. The term<\/strong> column indicates each variable. We will remove the (Intercept)<\/strong> for plotting purposes.<\/p>\n
Plot the Coefficients<\/strong><\/span><\/p>\n
\r\n## Plot the model coefficients and 2*SE across all folds\r\nmodel_coefs %>%\r\n  filter(term != "(Intercept)") %>%\r\n  select(id, term, estimate, std.error) %>%\r\n  group_by(term) %>%\r\n  mutate(avg_estimate = mean(estimate)) %>%\r\n  ggplot(aes(x = id, y = estimate)) +\r\n  geom_hline(aes(yintercept = avg_estimate),\r\n             size = 1.2,\r\n             linetype = "dashed") +\r\n  geom_point(size = 4) +\r\n  geom_errorbar(aes(ymin = estimate - 2*std.error, ymax = estimate + 2*std.error),\r\n                width = 0.1,\r\n                size = 1.2) +\r\n  facet_wrap(~term, scales = "free_y") +\r\n  labs(x = "CV Folds",\r\n       y = "Estimate \u00b1 95% CI",\r\n       title = "Regression Coefficients \u00b1 95% CI for 10-fold CV",\r\n       subtitle = "Dashed Line = Average Coefficient Estimate over 10 CV Folds per Independent Variable") +\r\n  theme_classic() +\r\n  theme(strip.background = element_rect(fill = "black"),\r\n        strip.text = element_text(face = "bold", size = 12, color = "white"),\r\n        axis.title = element_text(size = 14, face = "bold"),\r\n        axis.text.x = element_text(angle = 60, hjust = 1, face = "bold", size = 12),\r\n        axis.text.y = element_text(face = "bold", size = 12),\r\n        plot.title = element_text(size = 18),\r\n        plot.subtitle = element_text(size = 16))\r\n<\/pre>\n
\"\"<\/a><\/p>\n
Now we can clearly see the model coefficients and confidence intervals for each of the 10 cross validated folds.<\/p>\n
Wrapping Up<\/strong><\/span><\/p>\n
This was just a quick and easy way of fitting a model using cross-validation to extract out the model coefficients for each fold. Often, this is probably not necessary as you will fit your model, evaluate your model, and be off and running. However, there may be times where more specific interrogation of the model is required or, you might want to dig a little deeper into the various outputs of the cross-validated folds.<\/p>\n
All of the code is available on my GitHub page<\/a><\/span><\/strong>.<\/p>\n
If you notice any errors in code, please reach out!<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
As I’ve discussed previously, we sometimes don’t have enough data where doing a train\/test split makes sense. As such, we are better off building our model using cross-validation. In previous blog articles, I’ve talked about how to build models using cross-validation within the {tidymodels} framework (see HERE and HERE). In my prior examples, we fit […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47,45,43],"tags":[],"class_list":["post-2608","post","type-post","status-publish","format-standard","hentry","category-model-building-in-r","category-r-tips-tricks","category-sports-analytics"],"_links":{"self":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2608","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/comments?post=2608"}],"version-history":[{"count":3,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2608\/revisions"}],"predecessor-version":[{"id":2616,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2608\/revisions\/2616"}],"wp:attachment":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/media?parent=2608"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/categories?post=2608"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/tags?post=2608"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}