Learning Objectives

  • Describe the purpose of the dplyr and tidyr packages.
  • Select certain columns in a data frame with the dplyr function select.
  • Select certain rows in a data frame according to filtering conditions with the dplyr function filter.
  • Link the output of one dplyr function to the input of another function with the ‘pipe’ operator %>%.
  • Add new columns to a data frame that are functions of existing columns with mutate.
  • Use the split-apply-combine concept for data analysis.
  • Use summarize, group_by, and tally to split a data frame into groups of observations, apply a summary statistics for each group, and then combine the results.

Data Manipulation using dplyr and tidyr

Bracket subsetting is handy, but it can be cumbersome and difficult to read, especially for complicated operations. Enter dplyr. dplyr is a package for making tabular data manipulation easier. It pairs nicely with tidyr which enables you to swiftly convert between different data formats for plotting and analysis.

Packages in R are basically sets of additional functions that let you do more stuff. The functions we’ve been using so far, like str() or data.frame(), come built into R; packages give you access to more of them. Before you use a package for the first time you need to install it on your machine, and then you should import it in every subsequent R session when you need it. You should already have installed the tidyverse package. This is an “umbrella-package” that installs several packages useful for data analysis which work together well such as tidyr, dplyr, ggplot2, tibble, etc.

The tidyverse package tries to address 3 major problems with some of base R functions: 1. The results from a base R function sometimes depend on the type of data. 2. Using R expressions in a non standard way, which can be confusing for new learners. 3. Hidden arguments, having default operations that new learners are not aware of.

To load the package type:

library("tidyverse")    ## load the tidyverse packages, incl. dplyr

What are dplyr and tidyr?

The package dplyr provides easy tools for the most common data manipulation tasks. It is built to work directly with data frames, with many common tasks optimized by being written in a compiled language (C++). An additional feature is the ability to work directly with data stored in an external database. The benefits of doing this are that the data can be managed natively in a relational database, queries can be conducted on that database, and only the results of the query are returned.

This addresses a common problem with R in that all operations are conducted in-memory and thus the amount of data you can work with is limited by available memory. The database connections essentially remove that limitation in that you can connect to a database of many hundreds of GB, conduct queries on it directly, and pull back into R only what you need for analysis.

The package tidyr addresses the common problem of wanting to reshape your data for plotting and use by different R functions. Sometimes we want data sets where we have one row per measurement. Sometimes we want a data frame where each measurement type has its own column, and rows are instead more aggregated groups - like plots or aquaria. Moving back and forth between these formats is nontrivial, and tidyr gives you tools for this and more sophisticated data manipulation.

To learn more about dplyr and tidyr after the workshop, you may want to check out this handy data transformation with dplyr cheatsheet and this one about tidyr.

We’ll read in our data using the read_csv() function, from the tidyverse package readr, instead of read.csv(), the base function for reading in data. The data we are going to be using today should already be in your R_DAVIS_2020 project in the folder data.

surveys <- read_csv("data/portal_data_joined.csv")
## Rows: 34786 Columns: 13
## ── Column specification ─────────────────────────────────────────────────────────────────────────────────────
## Delimiter: ","
## chr (6): species_id, sex, genus, species, taxa, plot_type
## dbl (7): record_id, month, day, year, plot_id, hindfoot_length, weight
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## inspect the data
str(surveys)

Notice that the class of the data is now tbl_df This is referred to as a “tibble”. Tibbles are almost identical to R’s standard data frames, but they tweak some of the old behaviors of data frames. For our purposes the only differences between data frames and tibbles are that:

  1. When you print a tibble, R displays the data type of each column under its name; it prints only the first few rows of data and only as many columns as fit on one screen.
  2. Columns of class character are never automatically converted into factors.

Selecting columns and filtering rows

We’re going to learn some of the most common dplyr functions: select(), filter(), mutate(), group_by(), summarize(), and join. To select columns of a data frame, use select(). The first argument to this function is the data frame (surveys), and the subsequent arguments are the columns to keep.

select(surveys, plot_id, species_id, weight)

To choose rows based on a specific criteria, use filter():

filter(surveys, year == 1995)
## # A tibble: 1,180 × 13
##    record_id month   day  year plot_id species_id sex   hindfoot_length weight genus     species taxa  plot_type
##        <dbl> <dbl> <dbl> <dbl>   <dbl> <chr>      <chr>           <dbl>  <dbl> <chr>     <chr>   <chr> <chr>    
##  1     22314     6     7  1995       2 NL         M                  34     NA Neotoma   albigu… Rode… Control  
##  2     22728     9    23  1995       2 NL         F                  32    165 Neotoma   albigu… Rode… Control  
##  3     22899    10    28  1995       2 NL         F                  32    171 Neotoma   albigu… Rode… Control  
##  4     23032    12     2  1995       2 NL         F                  33     NA Neotoma   albigu… Rode… Control  
##  5     22003     1    11  1995       2 DM         M                  37     41 Dipodomys merria… Rode… Control  
##  6     22042     2     4  1995       2 DM         F                  36     45 Dipodomys merria… Rode… Control  
##  7     22044     2     4  1995       2 DM         M                  37     46 Dipodomys merria… Rode… Control  
##  8     22105     3     4  1995       2 DM         F                  37     49 Dipodomys merria… Rode… Control  
##  9     22109     3     4  1995       2 DM         M                  37     46 Dipodomys merria… Rode… Control  
## 10     22168     4     1  1995       2 DM         M                  36     48 Dipodomys merria… Rode… Control  
## # … with 1,170 more rows

select is used for rows and filter is used for columns.

Pipes

What if you want to select and filter at the same time? There are three ways to do this: use intermediate steps, nested functions, or pipes.

With intermediate steps, you create a temporary data frame and use that as input to the next function, like this:

surveys2 <- filter(surveys, weight < 5)
surveys_sml <- select(surveys2, species_id, sex, weight)

This is readable, but can clutter up your workspace with lots of objects that you have to name individually. With multiple steps, that can be hard to keep track of.

You can also nest functions (i.e. one function inside of another), like this:

surveys_sml <- select(filter(surveys, weight < 5), species_id, sex, weight)

This is handy, but can be difficult to read if too many functions are nested, as R evaluates the expression from the inside out (in this case, filtering, then selecting).

The last option is pipes. Pipes let you take the output of one function and send it directly to the next, which is useful when you need to do many things to the same dataset. Pipes in R look like %>% and are made available via the magrittr package, installed automatically with dplyr. If you use RStudio, you can type the pipe with Ctrl + Shift + M if you have a PC or Cmd + Shift + M if you have a Mac.

surveys %>%
  filter(weight < 5) %>%
  select(species_id, sex, weight)
## # A tibble: 17 × 3
##    species_id sex   weight
##    <chr>      <chr>  <dbl>
##  1 PF         F          4
##  2 PF         F          4
##  3 PF         M          4
##  4 RM         F          4
##  5 RM         M          4
##  6 PF         <NA>       4
##  7 PP         M          4
##  8 RM         M          4
##  9 RM         M          4
## 10 RM         M          4
## 11 PF         M          4
## 12 PF         F          4
## 13 RM         M          4
## 14 RM         M          4
## 15 RM         F          4
## 16 RM         M          4
## 17 RM         M          4

In the above code, we use the pipe to send the surveys dataset first through filter() to keep rows where weight is less than 5, then through select() to keep only the species_id, sex, and weight columns. Since %>% takes the object on its left and passes it as the first argument to the function on its right, we don’t need to explicitly include the data frame as an argument to the filter() and select() functions any more.

Some may find it helpful to read the pipe like the word “then”. For instance, in the above example, we took the data frame surveys, then we filtered for rows with weight < 5, then we selected columns species_id, sex, and weight. The dplyr functions by themselves are somewhat simple, but by combining them into linear workflows with the pipe, we can accomplish more complex manipulations of data frames.

If we want to create a new object with this smaller version of the data, we can assign it a new name:

surveys_sml <- surveys %>%
  filter(weight < 5) %>%
  select(species_id, sex, weight)

surveys_sml
## # A tibble: 17 × 3
##    species_id sex   weight
##    <chr>      <chr>  <dbl>
##  1 PF         F          4
##  2 PF         F          4
##  3 PF         M          4
##  4 RM         F          4
##  5 RM         M          4
##  6 PF         <NA>       4
##  7 PP         M          4
##  8 RM         M          4
##  9 RM         M          4
## 10 RM         M          4
## 11 PF         M          4
## 12 PF         F          4
## 13 RM         M          4
## 14 RM         M          4
## 15 RM         F          4
## 16 RM         M          4
## 17 RM         M          4

Note that the final data frame is the leftmost part of this expression.

Challenge

Using pipes, subset the surveys data to include individuals collected before 1995 and retain only the columns year, sex, and weight. Name this dataframe surveys_challenge

ANSWER
surveys_challenge <- surveys %>%
    filter(year < 1995) %>%
    select(year, sex, weight)


Mutate

Frequently you’ll want to create new columns based on the values in existing columns, for example to do unit conversions, or to find the ratio of values in two columns. For this we’ll use mutate().

To create a new column of weight in kg:

surveys %>%
  mutate(weight_kg = weight / 1000)
## # A tibble: 34,786 × 14
##    record_id month   day  year plot_id species_id sex   hindfoot_length weight genus   species  taxa   plot_type
##        <dbl> <dbl> <dbl> <dbl>   <dbl> <chr>      <chr>           <dbl>  <dbl> <chr>   <chr>    <chr>  <chr>    
##  1         1     7    16  1977       2 NL         M                  32     NA Neotoma albigula Rodent Control  
##  2        72     8    19  1977       2 NL         M                  31     NA Neotoma albigula Rodent Control  
##  3       224     9    13  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  4       266    10    16  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  5       349    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  6       363    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  7       435    12    10  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  8       506     1     8  1978       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  9       588     2    18  1978       2 NL         M                  NA    218 Neotoma albigula Rodent Control  
## 10       661     3    11  1978       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## # … with 34,776 more rows, and 1 more variable: weight_kg <dbl>

You can also create a second new column based on the first new column within the same call of mutate():

surveys %>%
  mutate(weight_kg = weight / 1000,
         weight_kg2 = weight_kg * 2)
## # A tibble: 34,786 × 15
##    record_id month   day  year plot_id species_id sex   hindfoot_length weight genus   species  taxa   plot_type
##        <dbl> <dbl> <dbl> <dbl>   <dbl> <chr>      <chr>           <dbl>  <dbl> <chr>   <chr>    <chr>  <chr>    
##  1         1     7    16  1977       2 NL         M                  32     NA Neotoma albigula Rodent Control  
##  2        72     8    19  1977       2 NL         M                  31     NA Neotoma albigula Rodent Control  
##  3       224     9    13  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  4       266    10    16  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  5       349    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  6       363    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  7       435    12    10  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  8       506     1     8  1978       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
##  9       588     2    18  1978       2 NL         M                  NA    218 Neotoma albigula Rodent Control  
## 10       661     3    11  1978       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## # … with 34,776 more rows, and 2 more variables: weight_kg <dbl>, weight_kg2 <dbl>

If this runs off your screen and you just want to see the first few rows, you can use a pipe to view the head() of the data. (Pipes work with non-dplyr functions, too, as long as the dplyr or magrittr package is loaded).

surveys %>%
  mutate(weight_kg = weight / 1000) %>%
  head()
## # A tibble: 6 × 14
##   record_id month   day  year plot_id species_id sex   hindfoot_length weight genus   species  taxa   plot_type
##       <dbl> <dbl> <dbl> <dbl>   <dbl> <chr>      <chr>           <dbl>  <dbl> <chr>   <chr>    <chr>  <chr>    
## 1         1     7    16  1977       2 NL         M                  32     NA Neotoma albigula Rodent Control  
## 2        72     8    19  1977       2 NL         M                  31     NA Neotoma albigula Rodent Control  
## 3       224     9    13  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## 4       266    10    16  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## 5       349    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## 6       363    11    12  1977       2 NL         <NA>               NA     NA Neotoma albigula Rodent Control  
## # … with 1 more variable: weight_kg <dbl>

The first few rows of the output are full of NAs, so if we wanted to remove those we could insert a filter() in the chain:

surveys %>%
  filter(!is.na(weight)) %>%
  mutate(weight_kg = weight / 1000) %>%
  head()
## # A tibble: 6 × 14
##   record_id month   day  year plot_id species_id sex   hindfoot_length weight genus   species  taxa   plot_type
##       <dbl> <dbl> <dbl> <dbl>   <dbl> <chr>      <chr>           <dbl>  <dbl> <chr>   <chr>    <chr>  <chr>    
## 1       588     2    18  1978       2 NL         M                  NA    218 Neotoma albigula Rodent Control  
## 2       845     5     6  1978       2 NL         M                  32    204 Neotoma albigula Rodent Control  
## 3       990     6     9  1978       2 NL         M                  NA    200 Neotoma albigula Rodent Control  
## 4      1164     8     5  1978       2 NL         M                  34    199 Neotoma albigula Rodent Control  
## 5      1261     9     4  1978       2 NL         M                  32    197 Neotoma albigula Rodent Control  
## 6      1453    11     5  1978       2 NL         M                  NA    218 Neotoma albigula Rodent Control  
## # … with 1 more variable: weight_kg <dbl>

is.na() is a function that determines whether something is an NA. The ! symbol negates the result, so we’re asking for every row where weight is not an NA.

Challenge

Create a new data frame from the surveys data that meets the following criteria: contains only the species_id column and a new column called hindfoot_half containing values that are half the hindfoot_length values. In this hindfoot_half column, there are no NAs and all values are less than 30. Name this data frame surveys_hindfoot_half.

Hint: think about how the commands should be ordered to produce this data frame!

ANSWER
surveys_hindfoot_half <- surveys %>%
    filter(!is.na(hindfoot_length)) %>%
    mutate(hindfoot_half = hindfoot_length / 2) %>%
    filter(hindfoot_half < 30) %>%
    select(species_id, hindfoot_half)


Group by and summarize

Many data analysis tasks can be approached using the split-apply-combine paradigm: split the data into groups, apply some analysis to each group, and then combine the results. dplyr makes this very easy through the use of the group_by() function.

group_by() is often used together with summarize(), which collapses each group into a single-row summary of that group. group_by() takes as arguments the column names that contain the categorical variables for which you want to calculate the summary statistics. So to compute the mean weight by sex:

surveys %>%
  group_by(sex) %>%
  summarize(mean_weight = mean(weight, na.rm = TRUE))
## # A tibble: 3 × 2
##   sex   mean_weight
##   <chr>       <dbl>
## 1 F            42.2
## 2 M            43.0
## 3 <NA>         64.7

You may also have noticed that the output from these calls doesn’t run off the screen anymore. It’s one of the advantages of tbl_df over data frame.

You can also group by multiple columns:

surveys %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight, na.rm = TRUE))
## `summarise()` has grouped output by 'sex'. You can override using the `.groups` argument.
## # A tibble: 92 × 3
## # Groups:   sex [3]
##    sex   species_id mean_weight
##    <chr> <chr>            <dbl>
##  1 F     BA                9.16
##  2 F     DM               41.6 
##  3 F     DO               48.5 
##  4 F     DS              118.  
##  5 F     NL              154.  
##  6 F     OL               31.1 
##  7 F     OT               24.8 
##  8 F     OX               21   
##  9 F     PB               30.2 
## 10 F     PE               22.8 
## # … with 82 more rows

When grouping both by sex and species_id, the first rows are for individuals that escaped before their sex could be determined and weighted. You may notice that the last column does not contain NA but NaN (which refers to “Not a Number”). To avoid this, we can remove the missing values for weight before we attempt to calculate the summary statistics on weight. Because the missing values are removed first, we can omit na.rm = TRUE when computing the mean:

surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight))
## `summarise()` has grouped output by 'sex'. You can override using the `.groups` argument.
## # A tibble: 64 × 3
## # Groups:   sex [3]
##    sex   species_id mean_weight
##    <chr> <chr>            <dbl>
##  1 F     BA                9.16
##  2 F     DM               41.6 
##  3 F     DO               48.5 
##  4 F     DS              118.  
##  5 F     NL              154.  
##  6 F     OL               31.1 
##  7 F     OT               24.8 
##  8 F     OX               21   
##  9 F     PB               30.2 
## 10 F     PE               22.8 
## # … with 54 more rows

Here, again, the output from these calls doesn’t run off the screen anymore. If you want to display more data, you can use the print() function at the end of your chain with the argument n specifying the number of rows to display:

surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight)) %>%
  print(n = 15)
## `summarise()` has grouped output by 'sex'. You can override using the `.groups` argument.
## # A tibble: 64 × 3
## # Groups:   sex [3]
##    sex   species_id mean_weight
##    <chr> <chr>            <dbl>
##  1 F     BA                9.16
##  2 F     DM               41.6 
##  3 F     DO               48.5 
##  4 F     DS              118.  
##  5 F     NL              154.  
##  6 F     OL               31.1 
##  7 F     OT               24.8 
##  8 F     OX               21   
##  9 F     PB               30.2 
## 10 F     PE               22.8 
## 11 F     PF                7.97
## 12 F     PH               30.8 
## 13 F     PL               19.3 
## 14 F     PM               22.1 
## 15 F     PP               17.2 
## # … with 49 more rows

Once the data are grouped, you can also summarize multiple variables at the same time (and not necessarily on the same variable). For instance, we could add a column indicating the minimum weight for each species for each sex:

surveys %>%
  filter(!is.na(weight)) %>%
  group_by(sex, species_id) %>%
  summarize(mean_weight = mean(weight),
            min_weight = min(weight))
## `summarise()` has grouped output by 'sex'. You can override using the `.groups` argument.
## # A tibble: 64 × 4
## # Groups:   sex [3]
##    sex   species_id mean_weight min_weight
##    <chr> <chr>            <dbl>      <dbl>
##  1 F     BA                9.16          6
##  2 F     DM               41.6          10
##  3 F     DO               48.5          12
##  4 F     DS              118.           45
##  5 F     NL              154.           32
##  6 F     OL               31.1          10
##  7 F     OT               24.8           5
##  8 F     OX               21            20
##  9 F     PB               30.2          12
## 10 F     PE               22.8          11
## # … with 54 more rows

Challenge

  1. Use group_by() and summarize() to find the mean, min, and max hindfoot length for each species (using species_id).
  2. What was the heaviest animal measured in each year? Return the columns year, genus, species_id, and weight.
  3. You saw above how to count the number of individuals of each sex using a combination of group_by() and tally(). How could you get the same result using group_by() and summarize()? Hint: see ?n.
ANSWER
## Answer 1
surveys %>%
    filter(!is.na(hindfoot_length)) %>%
    group_by(species_id) %>%
    summarize(
        mean_hindfoot_length = mean(hindfoot_length),
        min_hindfoot_length = min(hindfoot_length),
        max_hindfoot_length = max(hindfoot_length)
    )
## # A tibble: 25 × 4
##    species_id mean_hindfoot_length min_hindfoot_length max_hindfoot_length
##    <chr>                     <dbl>               <dbl>               <dbl>
##  1 AH                         33                    31                  35
##  2 BA                         13                     6                  16
##  3 DM                         36.0                  16                  50
##  4 DO                         35.6                  26                  64
##  5 DS                         49.9                  39                  58
##  6 NL                         32.3                  21                  70
##  7 OL                         20.5                  12                  39
##  8 OT                         20.3                  13                  50
##  9 OX                         19.1                  13                  21
## 10 PB                         26.1                   2                  47
## # … with 15 more rows
## Answer 2
surveys %>%
    filter(!is.na(weight)) %>%
    group_by(year) %>%
    filter(weight == max(weight)) %>%
    select(year, genus, species, weight) %>%
    arrange(year)
## # A tibble: 27 × 4
## # Groups:   year [26]
##     year genus     species     weight
##    <dbl> <chr>     <chr>        <dbl>
##  1  1977 Dipodomys spectabilis    149
##  2  1978 Neotoma   albigula       232
##  3  1978 Neotoma   albigula       232
##  4  1979 Neotoma   albigula       274
##  5  1980 Neotoma   albigula       243
##  6  1981 Neotoma   albigula       264
##  7  1982 Neotoma   albigula       252
##  8  1983 Neotoma   albigula       256
##  9  1984 Neotoma   albigula       259
## 10  1985 Neotoma   albigula       225
## # … with 17 more rows
## Answer 3
surveys %>%
  group_by(sex) %>%
  summarize(n = n())
## # A tibble: 3 × 2
##   sex       n
##   <chr> <int>
## 1 F     15690
## 2 M     17348
## 3 <NA>   1748