{"id":2982,"date":"2023-04-05T02:31:59","date_gmt":"2023-04-05T02:31:59","guid":{"rendered":"http:\/\/optimumsportsperformance.com\/blog\/?p=2982"},"modified":"2023-04-10T00:43:41","modified_gmt":"2023-04-10T00:43:41","slug":"rolling-mean-and-sd-not-including-the-most-recent-observation","status":"publish","type":"post","link":"https:\/\/optimumsportsperformance.com\/blog\/rolling-mean-and-sd-not-including-the-most-recent-observation\/","title":{"rendered":"Rolling Mean and SD not including the most recent observation"},"content":{"rendered":"<p>A colleague recently asked me a good question regarding some feature engineering for some data he was working with. He was collecting training load data and wanted to create a z-score for each observation, BUT, he didn&#8217;t want the most recent observation to be included into the calculation of the mean and standard deviation. Basically, he wanted to represent the z-score for the most recent observation normalized to the observations that came before it.<\/p>\n<p>This is an interesting issue because it makes me think of sports science research that uses z-scores to calculate the relationship between training load and injury. If the z-score is calculated retrospectively on the season then the observed z-scores and their relationship to the outcome of interest (injury) is a bit misleading as the mean and standard deviation of the full season data is not information one would have had on the day in which the injury occurred. All the practitioner would know, as the season progresses along, is the mean and standard deviation of the data up to the most recent observation.<\/p>\n<p>So, let&#8217;s calculate some lagged mean and standard deviation values! The full code is available on my <strong><span style=\"color: #0000ff;\"><a style=\"color: #0000ff;\" href=\"https:\/\/github.com\/pw2\/R-Tips-Tricks\/blob\/master\/Rolling%20Mean%20and%20SD%20not%20including%20most%20recent%20observation.R\">GITHUB page<\/a><\/span><\/strong>.<\/p>\n<p><span style=\"text-decoration: underline;\"><strong>Loading Packages and Simulating Data<\/strong><\/span><\/p>\n<p>Aside from loading {<strong>tidyverse<\/strong>} we will also load the {<strong>zoo<\/strong>} package, which is a common package used for constructing rolling window functions (this is useful as it prevents us from having to write our own custom function).<\/p>\n<p>We will start with a simple data set of just 10 training load observations.<\/p>\n<pre class=\"brush: r; title: ; notranslate\" title=\"\">\r\n## Load Libraries\r\nlibrary(tidyverse)\r\nlibrary(zoo)\r\n\r\n## Simulate Data\r\nset.seed(45)\r\nd &lt;- tibble(\r\n  day = 1:10,\r\n  training_load = round(rnorm(n = 10, min = 250, max = 350), 0)\r\n)\r\n\r\nd\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.09.30-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2983\" src=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.09.30-PM.png\" alt=\"\" width=\"248\" height=\"332\" srcset=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.09.30-PM.png 382w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.09.30-PM-224x300.png 224w\" sizes=\"auto, (max-width: 248px) 100vw, 248px\" \/><\/a><\/p>\n<p><span style=\"text-decoration: underline;\"><strong>Calculate the z-score with the mean and standard deviation of all data that came before it<\/strong><\/span><\/p>\n<p>To do this we use the <strong>rollapplyr()<\/strong> function from the {<strong>zoo<\/strong>} package. If we want to include the most recent observation in the mean and standard deviation we can run the function as is. However, because we want the mean and standard deviation of all data previous to, but not including, the most recent observation we wrap this entire function in the <strong>lag<\/strong><strong>()<\/strong> function, which will take the data in the row directly above the recent observation. The <strong>width<\/strong> argument indicates the width of the window we want to calculate the function over. In this case, since we have a <strong>day<\/strong> variable we can use that number as our window width to ensure we are getting all observed data prior to the most recent observation.<\/p>\n<pre class=\"brush: r; title: ; notranslate\" title=\"\">\r\nd &lt;- d %&gt;%\r\n  mutate(lag_mean = lag(rollapplyr(data = training_load, width = day, FUN = mean, fill = NA)),\r\n         lag_sd = lag(rollapplyr(data = training_load, width = day, FUN = sd, fill = NA)),\r\n         z_score = (training_load - lag_mean) \/ lag_sd)\r\n\r\nd\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.14.20-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2984\" src=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.14.20-PM.png\" alt=\"\" width=\"466\" height=\"307\" srcset=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.14.20-PM.png 762w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.14.20-PM-300x198.png 300w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.14.20-PM-624x411.png 624w\" sizes=\"auto, (max-width: 466px) 100vw, 466px\" \/><\/a><\/p>\n<p>This looks correct. As a sanity check, let&#8217;s calculate the mean and standard deviation of the first 4 rows of training load observations and see if those values correspond to what is in the <strong>lag_mean<\/strong> and <strong>lag_sd<\/strong> columns at row 5.<\/p>\n<pre class=\"brush: r; title: ; notranslate\" title=\"\">\r\nfirst_four &lt;- d %&gt;%\r\n  slice(1:4) %&gt;%\r\n  pull(training_load)\r\n\r\nmean(first_four)\r\nsd(first_four)\r\n<\/pre>\n<p><a href=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.15.01-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2985\" src=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.15.01-PM.png\" alt=\"\" width=\"214\" height=\"115\" \/><\/a><\/p>\n<p>It worked!<\/p>\n<p><span style=\"text-decoration: underline;\"><strong>A more complicated example<\/strong><\/span><\/p>\n<p>Okay, that was an easy one. We had one athlete and we had a training day column, which we could use for the window with, already built for us. What if we have a data set with multiple athletes and only training dates, representing the day that training happened?<\/p>\n<p>To make this work we will <strong>group_by()<\/strong> the athlete, and use the <strong>row_number()<\/strong> function to calculate a training day variable that represents our window width. Then, we simply use the same code above.<\/p>\n<p>Let&#8217;s simulate some data.<\/p>\n<pre class=\"brush: r; title: ; notranslate\" title=\"\">\r\nset.seed(67)\r\nd &lt;- tibble(\r\n  training_date = rep(seq(as.Date(&quot;2023-01-01&quot;),as.Date(&quot;2023-01-05&quot;), by = 1), times = 3),\r\n  athlete = rep(c(&quot;Karl&quot;, &quot;Bonnie&quot;, &quot;Thomas&quot;), each = 5),\r\n  training_load = round(runif(n = 15, min = 250, max = 350), 0)\r\n)\r\n\r\nd\r\n<\/pre>\n<p><a href=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.25.47-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2986\" src=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.25.47-PM.png\" alt=\"\" width=\"439\" height=\"483\" srcset=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.25.47-PM.png 634w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.25.47-PM-272x300.png 272w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.25.47-PM-624x687.png 624w\" sizes=\"auto, (max-width: 439px) 100vw, 439px\" \/><\/a><\/p>\n<p>Now we run all of our functions for each athlete.<\/p>\n<pre class=\"brush: r; title: ; notranslate\" title=\"\">\r\nd &lt;- d %&gt;%\r\n  group_by(athlete) %&gt;%\r\n  mutate(training_day = row_number(), \r\n         lag_mean = lag(rollapplyr(data = training_load, width = training_day, FUN = mean, fill = NA)),\r\n         lag_sd = lag(rollapplyr(data = training_load, width = training_day, FUN = sd, fill = NA)),\r\n         z_score = (training_load - lag_mean) \/ lag_sd)\r\n\r\n\r\nd\r\n<\/pre>\n<p><a href=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-2987\" src=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM-1024x610.png\" alt=\"\" width=\"625\" height=\"372\" srcset=\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM-1024x610.png 1024w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM-300x179.png 300w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM-768x458.png 768w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM-624x372.png 624w, https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2023\/04\/Screenshot-2023-04-04-at-7.27.50-PM.png 1228w\" sizes=\"auto, (max-width: 625px) 100vw, 625px\" \/><\/a><\/p>\n<p><span style=\"text-decoration: underline;\"><strong>Wrapping Up<\/strong><\/span><\/p>\n<p>There we have it, a simple way of calculating rolling z-scores while using the mean and standard deviation of the observations that came before the most recent observation!<\/p>\n<p>If you&#8217;d like the entire code, check out my <strong><span style=\"color: #0000ff;\"><a style=\"color: #0000ff;\" href=\"https:\/\/github.com\/pw2\/R-Tips-Tricks\/blob\/master\/Rolling%20Mean%20and%20SD%20not%20including%20most%20recent%20observation.R\">GITHUB page<\/a><\/span><\/strong>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A colleague recently asked me a good question regarding some feature engineering for some data he was working with. He was collecting training load data and wanted to create a z-score for each observation, BUT, he didn&#8217;t want the most recent observation to be included into the calculation of the mean and standard deviation. Basically, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45,43,42],"tags":[],"class_list":["post-2982","post","type-post","status-publish","format-standard","hentry","category-r-tips-tricks","category-sports-analytics","category-sports-science"],"_links":{"self":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2982","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/comments?post=2982"}],"version-history":[{"count":4,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2982\/revisions"}],"predecessor-version":[{"id":3016,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/posts\/2982\/revisions\/3016"}],"wp:attachment":[{"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/media?parent=2982"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/categories?post=2982"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/optimumsportsperformance.com\/blog\/wp-json\/wp\/v2\/tags?post=2982"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}