install.packages("tidyverse")
library(tidyverse)4 - Data wrangling and tidying
Introduction
We are now in the fourth week of the Fall 2024 semester. One quarter of the way through. Congratulations, y’all!
Today we’re going to start moving a little bit from abstract “how to” do things in R (but of course we’ll still have plenty of that) and into specific tasks that are relevant to food and ag scientists who need to do data science work. The focus today is going to be on
- Reading data into
R(as well as a little discussion of how we store data) - Learning to use
tidyverse, as set ofRpackages that makesRcode easier and more intuitive (most of the time) - (If we have time), returning to functions, including how to make them (and your scripts) readable and useful
We’re going to talk about file extensions and review how your computer stores data. Just like my discussion of data types in Week 1 (or, let’s be real, anything I do), this will be an informal and probably not 100% technically correct discussion. But learning to see these things properly will help you become a more capable user of scientific data.
Storing data on computers
One of the main challenges my students and others face in using R is getting data into the damn thing. I personally don’t think that most stats classes, or most classes on R, spend enough time on how to store your data, how to get it into a shape that R won’t struggle with too much, and how to actally get it into R (and what happens when you do).
I think one of the reasons for this is that typical software like SPSS or Minitab (or JMP) gives you a spreadsheet (Excel-like) interface to deal with your data, which you can either copy-paste in from Excel or input directly. Something like this:
In R, we have to use tools that are similar to the command line to deal with files. With some exceptions we will go over briefly, reading data into R follows the same rules we’ve learned about (in painful detail) so far: it is fast and powerful, but it also requires that we be very precise about how our data is stored already and how we want to bring the data into R.
These details seem arcane and pointless at first, and so we’re going to start by talking about how we store data in general.
Data storage outside R
Who here stores their lab notebook data in Excel or some other easy to use spreadsheet? Raise your hand.
Me too! That’s because there’s nothing wrong with doing this (except that Excel is kind of a monster of a program, but that’s another story). Where we start running into problems is the intersection between data storage and data manipulation. Here is an example:
The problems with this approach are several-fold. First off, the analyses run in here (the average log calculation, for example), rely on the spatial alignment of the data. If the data are copy-pasted incorrectly, or a line is deleted, the analysis will become incorrect without giving any warning. Second, the visual annotation (the colored fields, the bolding, the outlines around cells) are not directly machine-readable (technically they are metadata that can be accessed with enough coding, but they are not directly attached to the data in a way that R or other programs can easily process).
But the worst thing about this well-organized lab notebook is that the data are organized in multiple columns and rows within the same sheet, and that there are informational headers and other notes throughout the sheet. The reason this is bad has to do with how computers (and R in particular) process files. In general, R is going to be expecting a file with consistent structure on each line.
Computers like structured data. Humans… may not. When we look at the above Excel file, our brains (which are pattern-finding engines) quickly intuit the patterns. We look to see where the colors are defined in the top lines, then check those out in the file. We understand that the first couple lines are “metadata”. But computers can’t know this (unless we build complex workflows for them to follow that encode that knowledge).
So, instead, computers prefer data where the structure is predefined and there is no need for complicated rules.
The typical kind of data we will deal with in this class is delimited data. Non-rigorously defined, delimited data is data stored where each field is delimited by the same character. So, in the Excel file above, imagine that each cell value (A1, etc) is followed by the same character. This character could be anything, technically, but there are some conventional choices: comma-separated, tab-separated, or space-separated (least common). A comma-separated file would look like this:
# Comma-separated data
cat_acquisition_order,name,weight\n
1,Nick,9\n
2,Margot,7\n
3,Little Guy,11\n
4,Mr. B,8\nThis might look garbage-y, but let’s get in there. On each line of the file, we have three fields, each separated by a comma, and each line is ended (explicitly for the example) with the newline character: \n (the newline character is what your computer reads as “go to the next line”, like when you hit enter on your keyboard to go to a new line, you are explicitly entering a \n or a similar character into the document).
One of the lines in this file looks like column names, whereas the rest look like a consistent pattern of data. So we can think of this as a way to represent tabular data in a plain-text file. In fact, comma-separated value files have their own, common file extensions: .csv. And they are one of the most common ways to read data into R, but it is by no means the only way–we will go over a few others below in the Why Data Storage is Hard section.
So how do we actually get these data into R? When we use JMP or SPSS or Minitab we can just open our .csv (or word file or whatever) and copy-paste the data into something that looks a lot like Excel. What happens if we try to copy-paste the cat example above into R?
In general, our workflow for R is the following:
- Store data in delimited format (usually .csv, but we will be talking about how to deal with existing Excel (.xls, .xlsx) files today as well.)
- Organize your file directories SOMEHOW
- Name your files usefully–if you’re not going to use some kind of version-control like git, it is a good idea to name files with dates (I like YYYYMMDD format).
- Use a function like
read_csv()to read your data intoR.- It makes sense to store your files (or a copy of your files) in the same folder as your
Rscripts
- It makes sense to store your files (or a copy of your files) in the same folder as your
- If you want to use
Rto make changes to your files, usewrite_table()or other functions to output your data back into a sensibly named folder. -
THIS IS VERY IMPORTANT: Do not store data in
R’s workspace. Your scripts/Quarto files should always read in a data file, and make all changes to that file. This is why we clear changes on exit.- We can store data in an
R-specific format as.RDatafiles using thesave()function, but this makes the most sense only if we’re producing big or complex output we don’t want to rerun, and if we annotate the file well.
- We can store data in an
Example files
We are going to be working with several files in today’s example, which are all stored in the <class directory>/data/Week 4/ directory on Canvas. You should store them in a similar format, although on your file system you’ll want to replace <class directory> with your actual directory structure. We’ll talk more about directory structure very briefly later today.
The first file is polyphenols testing.csv, and it contains data from Sihui Ma’s dissertation (2019) research into polyphenols in apples. In this particular dataset, Dr. Ma was interested in the effect of readings of polyphenols content in response to different ratios of constituent polyphenols, because she suspected that tests were differentially responsive to different polyphenols (she was correct).
The second file is soup pilot data.csv. This is from research I conducted in 2016 at Drexel University on the effect of course order on acceptance for parts of a meal. This file, however, is just pilot data–we needed to develop different soups with different levels of base liking (outside of a meal), and so these are data from tasting of 4 soups, 2 minestrone and 2 hot and sour.
To start with, both of these files are well-structured. Here are some non-exhaustive attributes of that well-structuring:
- Each line is an observation.
- In the polyphenols file, that means each line is a single observation (this is long data).
- In the soup file, each line is a subject, with 4 observations (each soup) on the same line (this is wide data).
- In the polyphenols file, that means each line is a single observation (this is long data).
- There is no formatting or other non-readable information.
- The format of each file is 1 header line (names of variables) followed by each line as an observation that fits the pattern.
If your files are not in this format, it can sometimes make your life easier to get them into this format before you try to put them into R, although I will show at least one other option later.
Making your life in R easier with the tidyverse
Having talked about how we store data on our computers, we’re going to talk about how to get data into R. This is a good point to introduce a large set of tools that we will be leaning on heavily for the rest of the semester. I personally think that the tidyverse tools make your life easier, especially when dealing with these kinds of data.frame/tabular data objects.
The tidyverse is a shortcut package that you should already be familiar with from your reading in R for Data Science. It loads a number of individual packages, including two that make many of the R operations we’ve been learning about for the first month of class more natural and comfortable. You can learn about all of them at the tidyverse package page.
If you haven’t already, you will need to install and load the tidyverse package to make this work.
Also, remember that the tidyverse approach isn’t the only right way to do things. I personally like the style of coding that tidyverse enables, but sometimes it makes things harder. Use the tool that best suits you!
Importing data
We’re going to first start with the part of the tidyverse that imports data. Today, we’re really only going to focus on two kinds of data: the comma-separated type we discussed above, and Excel data (typically stored in .xls or .xlsx files). These are the most common types of data storage you’ll deal with on a daily basis.
read_csv()
The most common type of data storage for when data are simple and tabular (like a spreadsheet) is a .csv file. We already talked about these data above. To get them into R, we are going to use the tidyverse::read_csv() function.
read_csv("data/Week 4/polyphenol testing.csv")Rows: 81 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
dbl (4): Catechin, PC B2, Chlorogenic acid, equivalents in gallic acid mg/L
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.# A tibble: 81 × 4
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L`
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 1 200 57.7
5 0.2 1 200 58.2
6 0.2 1 200 69.2
7 0.2 1 500 150.
8 0.2 1 500 162.
9 0.2 1 500 161.
10 0.2 30 5 15.2
# ℹ 71 more rowsread_csv("data/Week 4/soup pilot data.csv")Rows: 24 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
dbl (5): subject, minestrone1, minestrone2, hotandsour1, hotandsour2
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.# A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0 50 -50 10
2 2 70 80 -20 20
3 3 80 97 10 37
4 4 20 60 -80 65
5 5 65 96 37 99
6 6 30 80 -50 20
7 7 75 90 40 80
8 8 70 80 50 -10
9 9 40 90 -60 20
10 10 15 65 25 20
# ℹ 14 more rowsIt’s been a little while since we did this, but what happens if we want that data?
polyphenols testing # We read the data and just threw it away! (and used a bad name)Error in parse(text = input): <text>:1:13: unexpected symbol
1: polyphenols testing
^Remember, we have to store R objects using <-.
Now we can access these objects and learn about them.
class(polyphenols) # the read_*() functions create `tibble` objects[1] "spec_tbl_df" "tbl_df" "tbl" "data.frame" head(polyphenols, n = 5) # display the first n lines of an object--defaults to n = 6# A tibble: 5 × 4
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L`
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 1 200 57.7
5 0.2 1 200 58.2 soups$minestrone1 # the defaults for read_csv() expects columns to be named [1] 0 70 80 20 65 30 75 70 40 15 -50 75 -25 20 70 20 15 40 55
[20] 20 20 20 10 10And we can start to do basic data analysis. As a treat for all that work, let’s look at a boxplot of soup rating:
boxplot(soups[, -1]) # We use the [, -1] syntax to remove the first column of soups--why?
Some notes on tidyverse import functions
A couple little notes here on importing using read_csv() and other functions. First, these functions are cleaner versions of base-R functions like read.csv(), read.table(), etc. The key differences are
- The
tidyversefunctions import data astibble(a type ofdata.framewith nice properties) - Functions like
read_csv()will by default import characters columns ascharactervariables - Column names with disallowed characters (like
,(), etc) will be imported “as is”, surrounded by back-ticks (```)
- Row names are not allowed
If you need to import other kinds of tabular data, you should look into functions like read_table() (for space-delimited data), read_tsv() (for tab-delimited data), or read_lines() (for flexible, line-by-line input).
Naming: long (“tidy”) vs wide data
The name of the “tidyverse” comes from the core concept of “tidy” data. According to Wickham and Grolemund (2017), tidy data has the following properties:
- Each variable must have its own column.
- Each observation must have its own row.
- Each value must have its own cell.
The result of this is the style of data we see in the polyphenols dataset, which is structured as a tibble, a kind of data.frame that has some special built in functionality to display and print nicely:
polyphenols# A tibble: 81 × 4
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L`
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 1 200 57.7
5 0.2 1 200 58.2
6 0.2 1 200 69.2
7 0.2 1 500 150.
8 0.2 1 500 162.
9 0.2 1 500 161.
10 0.2 30 5 15.2
# ℹ 71 more rows“Tidy” data is a synonym for “long” or “tall” data. This is in contrast to the “wide” data of the soups dataset:
soups# A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0 50 -50 10
2 2 70 80 -20 20
3 3 80 97 10 37
4 4 20 60 -80 65
5 5 65 96 37 99
6 6 30 80 -50 20
7 7 75 90 40 80
8 8 70 80 50 -10
9 9 40 90 -60 20
10 10 15 65 25 20
# ℹ 14 more rowsIn soups, each row represents a case–the subject–and then there are multiple columns that represent different observations on a particular soup. The type-of-soup variable, meanwhile, has no column of its own. This is a common format for data storage, because it feels natural. But it may not always be the easiest format to access and manipulate data. In later classes we’ll learn about some easy functions to switch between these data formats using ideas from database software, but for now we’re just going to note their existence.
Why data storage is hard
OK, so we’ve pretty quickly gotten to getting data into R. That didn’t seem so bad. But let’s try importing a third file: Jey Kessinger’s thesis data data (2019), in which she had subjects sort 18 ciders into groups according to their own perceptions. The file is called cider sorting.csv, but it has two problems.
- It is not stored in the standardized location I set up for this Quarto.
- It is stored as an Excel file (
.xlsx) instead of as a.csvfile. - It is well-formatted for human reading, but not for computers.
In week 2 we spent some time going over working directories, and we’re going to return to that concept (and file storage in general) here, because I find this is a real barrier for many students.
Where files live in your computer
The first problem above is one of file organization. Who here uses a set of structured folders to organize their files?
But it seems like this is becoming less standard practice, according to this article in the Verge from Fall 2021. Both Windows and macOS have introduced more powerful search functions, so that if you have an idea of what your file is called you can kind of get away with just typing it into the search engine in the OS and having it return you the file.
In addition, macOS (with iCloud) and Windows (with OneDrive) increasingly push users towards cloud-based (non-local) storage as a default, which increases confusion about where individual files live at any given time.
Unfortunately, this plays pretty poorly with coding for research. We have a lot of data files that are often versioned–with the same file name but different dates. We also tend to have similar file names: “Cider ANOVA”, “polyphenols ANOVA”, etc. It takes longer and longer file names to specify these, and it also requires more and better memory. On the other hand, folder structure, of the type I show above, is sort of like “preserved memory”–I don’t actually remember teaching FST 3024 in 2018 very well, but I know if I go into that folder I will find files in descriptively labeled folders. There’ll be a folder for lectures/slides, one for grades, one for homework, etc.
Therefore for this class and for analysis projects, I recommend a simple file structure that can be expanded upon, rather than descriptively naming files. You should create a directory (a “folder”) for the class, and have sub-directories within that directory for each week. You will find this much easier to manage.
You might also take a look at the suggested organizational structure for R projects from Data Carpentry..
It turns out this file is stored in our general <path>/data folder, instead of <path>/data/Week 4/, where <path> here stands in for all of the folder pathway to whatever your class project folder is.
Dealing with other ways of storing data
So once we find the file we can read it, right? Let’s try our trusty read_csv() function
read_csv("data/cider sorting.xlsx")Multiple files in zip: reading '[Content_Types].xml'
Rows: 1 Columns: 1
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.# A tibble: 1 × 1
`<?xml version="1.0" encoding="UTF-8" standalone="yes"?>`
<chr>
1 "<Types xmlns=\"http://schemas.openxmlformats.org/package/2006/content-types\…Hmm. That doesn’t seem right. If we look carefully, we can realize this is not a .csv file. Luckily, tidyverse comes with a package called readxl, which we can use on .xls and .xlsx files.
read_excel("data/cider sorting.xlsx")# A tibble: 1,248 × 31
`Test Name` Description `Test Status` `Test Owner` `Saved for`
<chr> <chr> <chr> <chr> <chr>
1 102918_VA Cider Sort New Test Ready To Run J'Nai Phill… Virginia T…
2 <NA> <NA> <NA> <NA> <NA>
3 Section Name 1. First Section… <NA> <NA> <NA>
4 <NA> <NA> <NA> <NA> <NA>
5 Sample Number Sample Name Blinding Code Sample Type <NA>
6 1 Big Fish Cider C… 188 Sample <NA>
7 2 Bold Rock 826 Sample <NA>
8 3 Blue Bee 544 Sample <NA>
9 4 Old Hill 597 Sample <NA>
10 5 Bryant's 343 Sample <NA>
# ℹ 1,238 more rows
# ℹ 26 more variables: `Test Language` <chr>, `Test Expiry` <chr>, ...8 <chr>,
# ...9 <chr>, ...10 <chr>, ...11 <chr>, ...12 <chr>, ...13 <chr>,
# ...14 <chr>, ...15 <chr>, ...16 <chr>, ...17 <chr>, ...18 <chr>,
# ...19 <lgl>, ...20 <lgl>, ...21 <lgl>, ...22 <lgl>, ...23 <lgl>,
# ...24 <lgl>, ...25 <lgl>, ...26 <lgl>, ...27 <lgl>, ...28 <lgl>,
# ...29 <lgl>, ...30 <lgl>, ...31 <lgl>Still, this doesn’t seem right. I actually turned off warnings in the chunk here (using warning = FALSE) because read_excel() screamed at us! If we open up the file cider sorting.xlsx using Excel, you’ll see that the second problem with this file is that it is, like Michael Wesolowski’s thesis data, really mixing data organization and data analysis.
Breaking up human research notebooks into data files
It looks like there are actually multiple data tables crammed into one file. This is a clue that this isn’t really just a data file–it’s a directory. What we should do is restructure this file into multiple files, all in one directory called “cider sorting”. For example, we’d want to create a cider samples.csv file with just the sample data, as well as .csv files for the sorting and the questionnaire data.
One nice thing about Excel files is they allow reference by rows and columns, using numerals (1, 2, …, 1000, …) for rows and letters (A, B, …, AZ, …) for the columns. So we can use these ranges in read_excel() to do this kind of “surgery” on the
# Here we just get the sample blinding codes and subject demographics for a demo
sample_codes <-
read_excel("data/cider sorting.xlsx", range = "A7:D25")
demographics <-
read_excel("data/cider sorting.xlsx", range = "A1185:AE1250")
sample_codes# A tibble: 18 × 4
`Sample Number` `Sample Name` `Blinding Code` `Sample Type`
<dbl> <chr> <dbl> <chr>
1 1 Big Fish Cider Co. 188 Sample
2 2 Bold Rock 826 Sample
3 3 Blue Bee 544 Sample
4 4 Old Hill 597 Sample
5 5 Bryant's 343 Sample
6 6 Rustico 750 Sample
7 7 Potter's 221 Sample
8 8 Mt. Defiance 490 Sample
9 9 Blue Toad 354 Sample
10 10 Kindred Point 145 Sample
11 11 Wild Hare 140 Sample
12 12 Corcoran 911 Sample
13 13 Coyote Hole 542 Sample
14 14 Cobbler Mountain 363 Sample
15 15 Abemarle 241 Sample
16 16 Buskey 335 Sample
17 17 Castle Hill 810 Sample
18 18 Foggy Ridge 821 Sample demographics# A tibble: 65 × 31
Test_Name Section_Name Section_Number Sample_Set_Number Session_Number
<chr> <chr> <dbl> <dbl> <dbl>
1 102918_VA Cider… Second Sect… 2 1 1
2 102918_VA Cider… Second Sect… 2 2 1
3 102918_VA Cider… Second Sect… 2 3 1
4 102918_VA Cider… Second Sect… 2 4 1
5 102918_VA Cider… Second Sect… 2 5 1
6 102918_VA Cider… Second Sect… 2 6 1
7 102918_VA Cider… Second Sect… 2 7 1
8 102918_VA Cider… Second Sect… 2 8 1
9 102918_VA Cider… Second Sect… 2 9 1
10 102918_VA Cider… Second Sect… 2 10 1
# ℹ 55 more rows
# ℹ 26 more variables: Session_Name <chr>, Panelist_Code <chr>,
# Panelist_Name <chr>, Panelist_Display_Name <chr>, Panelist_Email <chr>,
# Q1__Gender <dbl>, Q1__Gender_COMMENTS <chr>, Q1__Gender_Time_Stamp <dttm>,
# Q2__Age <dbl>, Q2__Age_Time_Stamp <dttm>, Q3__Purchase_cider <dbl>,
# Q3__Purchase_cider_Time_Stamp <dttm>, Q4__Cider_choice <chr>,
# Q4__Cider_choice_Time_Stamp <dttm>, Q5__Drink_cider <dbl>, …Saving your work
I said that we should consider breaking up this file. How can we make new files (.csv, etc?) from R data? The write_csv() function (and various other write_*() functions) let you do this. It’s generally good practice, as we discussed above, to organize your project folders, so perhaps you want to save your raw data and your altered/modified/cleaned data in different folders. Here, I’ve created an output folder to work with.
We can take a data.frame like our sample_codes table and use write_csv() to save a .csv with its contents:
write_csv(x = sample_codes,
file = "output/cider-sample-codes.csv")Note that I set eval=FALSE in this chunk, since it’s kind of rude for me to write files on your computer! Try this out to see if it works.
If you have multiple data tables you want to save, or a bunch of objects that aren’t rectangular (tabular) data, you might want to instead save (part of) your workspace using the save() function. This will create a file of type .RData that, when opened in R, will load a set of arbitrary objects back into your workspace.
# first, let's make a non-tabular object
example_list <- list(FALSE, 1:20, letters)
save(example_list, polyphenols, soups, file = "output/week-4-worskpace.RData")
# Now we can delete these objects, since we saved them
rm(example_list, polyphenols, soups)
# And finally, reload them into our session using `load()`
load(file = "output/week-4-worskpace.RData")A couple words of warning:
- Any kind of
Routput function (likewrite_csv()) will typically overwrite a file that already exists. It’s like if youFile > Savean existing file in RStudio; it won’t ask if you want to make the changes! So be careful with what you’re doing. - Again, let me emphasize that it’s a good idea to save your edited/wrangled/cleaned data files separately (and obviously so) from your raw data. You don’t want to overwrite your raw data and be unable to remember how to get back to it.
Other data types
This is just one example of how the same file type can contain multiple (and sometimes pathological) ways of storing data.
There are many other ways of Structuring data. There is no need for the data to be tabular, for example. A common form of structured data is a JSON file–JavaScript Object Notation. These files look like this (example from Wikipedia):
# This is a JSON object
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 27,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
}
],
"children": [],
"spouse": null
}This is a very different kind of format than the .csv files we’ve been looking at, but you can see there are a number of “key-value” pairs here that give it structure, as well as being able to support a nested (recursive) structure, like that in the “address” field. This can be parsed by a computer easily because of its consistent structure–in fact, the jsonlite::fromJSON() function will happily do this. Can you run this and see what R thinks this is?
Note two advantages of this kind of data storage over tabular (.csv) files:
- It allows for nested/structured subfields, like the “address” field (which is itself a tiny JSON)
- It allows for fields with multiple/arbitrary entries
Neither of these are possible in tabular data or in R data frames–but these are both features of R’s list structure. And you’ll notice the JSON imports as a list object. As we will see in the rest of the course, we often encounter lists when we need flexibility.
Syntax conventions
Finally, before we start learning to wrangle the data we’ve just imported, a couple of notes about syntax. Last week, learning about subsetting, indexing, and writing formal control-flow structures, we focused a lot on typing things correctly so computers can understand them. There are a couple of new things that tidyverse functions expect from us that we haven’t seen before.
Unquoted arguments
In many of the “core” tidyverse functions like those we are going to learn about here, arguments to functions can (and often must) be left unquoted. That means that, for example, if we want to use the select() function to pick the minestrone1 column from soups (see below), we would merely need to write select(soups, minestrone1). Contrast this with base R, in which (using the [] syntax), we’d need to write soups[, "minestrone1"].
This is a programming choice based on how R uses names and symbols in the environment (see ?as.name for some confusing terminology). It is not critical to know the why of this, only to know that, for many functions in tidyverse, you supply plain variable names, or even vectors (using c() of plain variable names). If you are really curious about how all this works, the Advanced R book may be of interest (but then I would wonder why you were taking this class).
One pipe to rule them all: |> %>%
%>%One of the main tools that makes the tidyverse compelling is the pipe: |> (or, alternatively, %>% known as the magrittr pipe). This garbage-looking set of symbols is actually your best friend, you just don’t know it yet. I use this tool constantly in my R programming, but I’ve been avoiding it up to this point because it’s not part of base R (in fact that’s no longer strictly true, but it is kind of complicated at the moment).
OK, enough background, what the heck is a pipe? The term “pipe” comes from what it does: like a pipe, |> let’s whatever is on it’s left side flow through to the right hand side. Let’s try this out in some pseudocode
load soups data |> # Load the soups data into `R` AND THEN
select minestrone columns |> # Pick only the columns to do with minestrone AND THEN
filter first 10 subjects |> # Get only the first 10 subjects to participate AND THEN
run a t-test to compare soups # Run the statistical test we wantedIn this example, each place there is a |> I’ve added a comment saying “AND THEN”. This is because that’s exactly what the pipe does: it passes whatever happened in the previous step to the next function. Without the pipe, we’d end up doing something like this:
soups <- load soups data # Create an object with our data
soups_2 <- select soups minestrone columns # Create a second object for our selected columns
soups_3 <- filter soups_2 first 10 subjects # Create a third object for our filtered data
t-test(soups_3) # Run our t_test on the third data setThis is messy, harder to read, and means that we have to run every line again if we mess up, because if we go back and, for example, change soups (say we forgot to use read.table() with header = TRUE), the other objects won’t be automatically updated. But we don’t care about those intermediate objects–they’re just by-products of our workflow. The advantage of the |> pipe is that it gets rid of those, and makes code that is much more readable in the process.
There are a couple of points to know about using the pipe, and much more detail is available from the Wickham and Grolemund (2017) chapter on the subject. In the following examples (and throughout) we’re going to get a lot of experience in using this tool.
Pipes require that the lefthand side be a single functional command
This means that we can’t directly do something like rewrite sqrt(1 + 2) with |>:
# this gives us an error
1 + 2 |> sqrtError in sqrt: The pipe operator requires a function call as RHS (<input>:2:10)Instead, if we want to pass binary operationse in a pipe, we need to enclose them in () on the line they are in:
# Now this computes sqrt(1 + 2) = sqrt(3)
(1 + 2) |> sqrt() [1] 1.732051More complex piping is possible using the curly braces ({}), which create new R environments, but this is more advanced than you will generally need to be.
Pipes always pass the result of the lefthand side to the first argument of the righthand side
This sounds like a weird logic puzzle, but it’s not, as we can see if we look at some simple math. Let’s define a function for use in a pipe that computes the difference between two numbers:
# What does this simple function do?
subtract <- function(a, b) a - b
# Now let's test it out on something we know the answer to. This is an
# important test.
subtract(5, 4)[1] 1If we want to rewrite that as a pipe, we can write:
5 |> subtract(4)[1] 1But we can’t write
# this is actually making subtract(4, 5)
4 |> subtract(5) [1] -1We can explicitly force the pipe to work the way we want it to by using _ as the placeholder for the result of the lefthand side. In the vernacular of the tidyverse, the _ is a “pronoun”. It is a placeholder for a specific thing (the result of the lefthand side of the |>):
# now this properly computes subtract(5, 4)
4 |> subtract(5, b = _) [1] 1So, when you’re using pipes, make sure that the output of the lefthand side should be going into the first argument of the righthand side–this is often but not always the case, especially with non-tidyverse functions.
Pipes are a pain to type
Typing |> is no fun. But, happily, RStudio builds in a shortcut for you: cmd + shift + M (cmd = ctrl on Windows).
.
..
…
cmd + shift + M = |>
Pipes pass tibble names to autocomplete
One other quality-of-life improvement that pipes offer is that, for tibbles (and named objects in general), the pipe will allow you to use tab-based autocomplete with column names (variable names) from the previous step in the pipe. This can save you a ton of piping and make “human readable” column names more useful.
In our polyphenols data frame, we’ve got some pretty funky column names:
names(polyphenols)[1] "Catechin" "PC B2"
[3] "Chlorogenic acid" "equivalents in gallic acid mg/L"Do you really want to type out equivalents in gallic acid mg/L every time you want to refer to that column? While you could (and maybe should!) rename that column, you can also use tab-complete to just grab it in pipes.
polyphenols |>
# follow along here and hit `tab` to get the names!
select()Cool stuff!
NB: This only works in (newish) versions of RStudio–if you try to do this in base R.app or in the command line, you will not have this functionality.
Data wrangling basics
OK, with those bits of syntax out of the way, let’s talk about what tidyverse can get us. We’ll start with something simple, but nice. The augmented version of data.frame that is provided by tidyverse (via the tibble package) is called a “tibble”.
The tibble()
class(soups)[1] "spec_tbl_df" "tbl_df" "tbl" "data.frame" soups# A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0 50 -50 10
2 2 70 80 -20 20
3 3 80 97 10 37
4 4 20 60 -80 65
5 5 65 96 37 99
6 6 30 80 -50 20
7 7 75 90 40 80
8 8 70 80 50 -10
9 9 40 90 -60 20
10 10 15 65 25 20
# ℹ 14 more rowsYou’ll notice a few thing about these:
- They print nicely: delimited number of lines, explicit printing of column types
- They do have
classofdata.frame, but alsotblandtbl_df–this means everything that works ondata.frameworks on them, but they also have additional capabilities - Tibbles disallow row names, which is a good habit to get into but can cause some compatibility issues with non-
tidyversefunctions
Tibbles can be created using pretty simple syntax that is parallel but simpler than data frames:
Pick columns: select()
Here’s where tidyverse starts getting kind of spicy! R’s system for indexing data frames is clear and sensible for those who are used to programming languages, but it is not necessarily easy to read.
A common situation in R is wanting to select some rows and some columns of our data–this is called “subsetting” our data. We might want to do it manually (when we are doing exploration) or write a function that will do it automatically (maybe as part of a conditional statement or a loop). We give an example of wanting to do this in our pseudocode example for pipes, above. But this is less easy than it might be for the beginner in R.
In our pseudocode example, we wanted to pick all columns in soups that were about “minestrone”. In base R, we can get at these in a number of ways, but the most “foolproof” way is the following:
# A tibble: 24 × 2
minestrone1 minestrone2
<dbl> <dbl>
1 0 50
2 70 80
3 80 97
4 20 60
5 65 96
6 30 80
7 75 90
8 70 80
9 40 90
10 15 65
# ℹ 14 more rowsYikes! The reason this is “foolproof” is because it doesn’t rely on these columns being in a particular place, and it does not require the reconstruction of the data frame from constituent parts. (grep() is a function for finding regular expressions, a powerful, compact, difficult way to do character matching.) We could do a less foolproof but easier search in the following ways:
- Use numeric indexing to get the right numbers of the columns. This relies on the indexes, rather than the names containing minestrone.
- Extract each minestrone columns using
$and paste them together into a new data frame.
As an exercise, let’s use the following chunk to actually do both.
# Numeric indexing
# Reconstructing from extracted vectorsThe select() function in tidyverse (actually from the dplyr package) is the smarter, easier way to do this. It works on data frames, and it can be read as “from <data frame>, select the columns that meet the criteria we’ve set.”
The simplest way to use select() is just to name the columns you want!
# note the lack of quoting on the column names
select(soups, minestrone1, minestrone2) # A tibble: 24 × 2
minestrone1 minestrone2
<dbl> <dbl>
1 0 50
2 70 80
3 80 97
4 20 60
5 65 96
6 30 80
7 75 90
8 70 80
9 40 90
10 15 65
# ℹ 14 more rowsBut select() has a bunch of helper functions that make it easy to do more foolproof selection. Look them up by running, for example, ?one_of. We’re going to use contains(), which is the easy-to-read version of the grep() command I ran above.
We’ll also do this with a pipe (|>) just to practice.
# A tibble: 24 × 2
minestrone1 minestrone2
<dbl> <dbl>
1 0 50
2 70 80
3 80 97
4 20 60
5 65 96
6 30 80
7 75 90
8 70 80
9 40 90
10 15 65
# ℹ 14 more rowsWe can also use logical negation (!, which we learned about last class) to, for example, get everything that isn’t “hotandsour”.
# A tibble: 24 × 3
subject minestrone1 minestrone2
<dbl> <dbl> <dbl>
1 1 0 50
2 2 70 80
3 3 80 97
4 4 20 60
5 5 65 96
6 6 30 80
7 7 75 90
8 8 70 80
9 9 40 90
10 10 15 65
# ℹ 14 more rows# A tibble: 24 × 2
minestrone1 minestrone2
<dbl> <dbl>
1 0 50
2 70 80
3 80 97
4 20 60
5 65 96
6 30 80
7 75 90
8 70 80
9 40 90
10 15 65
# ℹ 14 more rowsBesides being easier to write conditions for, select() is code that is much closer to how you or I think about what we’re actually doing, making code that is more human readable.
Pick rows: filter()
So select() lets us pick which columns we want. Can we also use it to pick particular observations? No. But for that, there’s filter().
In the pseudocode above, we stated we wanted to only get the first 10 subjects. Just like selecting columns we can do this using base R with a combination of indexing approaches. Here is, again, the most foolproof way to select the first 10 subjects in the soups data:
# A tibble: 10 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0 50 -50 10
2 2 70 80 -20 20
3 3 80 97 10 37
4 4 20 60 -80 65
5 5 65 96 37 99
6 6 30 80 -50 20
7 7 75 90 40 80
8 8 70 80 50 -10
9 9 40 90 -60 20
10 10 15 65 25 20This works by (reading inside out), first using %in% to test whether each entry in soups$subject is in 1:10, then using which(), a function that turns a vector of logical (true/false) data into a set of integers based on the positions of the TRUE readings in the vector, and then finally using that as the index to get the rows of soups. That’s a lot to think about. There’s another, less foolproof way of doing this: select only the first 10 rows of soups.
As an exercise, write R code to select only the first 10 rows of soups. Then, tell me why this is a less foolproof way to select the first 10 subjects.
# Select only the first 10 rows of soupsThe filter() function works to do this in a more human way. Read it as “In , give me only the rows that meet the filter conditions.” So, for our example:
# A tibble: 10 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0 50 -50 10
2 2 70 80 -20 20
3 3 80 97 10 37
4 4 20 60 -80 65
5 5 65 96 37 99
6 6 30 80 -50 20
7 7 75 90 40 80
8 8 70 80 50 -10
9 9 40 90 -60 20
10 10 15 65 25 20We can combine filter() and select() using pipes (|>) to get most of our pseudocode done.
# A tibble: 10 × 3
subject minestrone1 minestrone2
<dbl> <dbl> <dbl>
1 1 0 50
2 2 70 80
3 3 80 97
4 4 20 60
5 5 65 96
6 6 30 80
7 7 75 90
8 8 70 80
9 9 40 90
10 10 15 65In the code chunk below, right this so that it first selects for “minestrone”, rather than dropping “hotandsour”. What happens?
# rewrite the above code chunk to first select for "minestrone". What happens, and why?These are trivial examples, but they quickly become more powerful. In our tidy polyphenols dataset, we have 3 treatment variables (Catechin, PC.B2, and Chlorogenic.acid). A very common application for filter() would be to get a subset of our data that only matches certain treatment conditions:
# get only the observations where `Chlorogenic acid` == 5. Note the double '==' for testing equality.
polyphenols |>
filter(`Chlorogenic acid` == 5) # A tibble: 27 × 4
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L`
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 30 5 15.2
5 0.2 30 5 15.3
6 0.2 30 5 14.0
7 0.2 100 5 46.6
8 0.2 100 5 46.1
9 0.2 100 5 50.9
10 30 1 5 14.0
# ℹ 17 more rowsMake new columns: mutate()
You hopefully are starting to be excited by the relative ease of doing some things in R with tidyverse that are otherwise a little bit abstruse. Here’s where I think things get really, really cool. The mutate() function creates a new column in the existing dataset.
Let’s start with a very simple example–say we wanted to add an “observation ID” column to our polyphenols data that had the row number in it - row 1 will have obs_id = 1, etc.
polyphenols |>
mutate(obs_id = row_number())# A tibble: 81 × 5
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L` obs_id
<dbl> <dbl> <dbl> <dbl> <int>
1 0.2 1 5 2.79 1
2 0.2 1 5 2.73 2
3 0.2 1 5 2.74 3
4 0.2 1 200 57.7 4
5 0.2 1 200 58.2 5
6 0.2 1 200 69.2 6
7 0.2 1 500 150. 7
8 0.2 1 500 162. 8
9 0.2 1 500 161. 9
10 0.2 30 5 15.2 10
# ℹ 71 more rowsNote that the obs_id column got created to the far “right” of the tibble, and the row_number() function is a little helper that tidyverse provides for exactly this sort of situation.
We can do this easily in base R by setting a new name for a column and using the assign (<-) operator, maybe with the new column defined by something like 1:dim(polyphenols)[1], but this is clumsy.
Often, we want to create a new column temporarily, or to combine several existing columns. We can do this using the mutate() function.
# A tibble: 24 × 6
subject minestrone1 minestrone2 hotandsour1 hotandsour2 minestrone_liker
<dbl> <dbl> <dbl> <dbl> <dbl> <chr>
1 1 0 50 -50 10 no
2 2 70 80 -20 20 yes
3 3 80 97 10 37 yes
4 4 20 60 -80 65 yes
5 5 65 96 37 99 yes
6 6 30 80 -50 20 yes
7 7 75 90 40 80 yes
8 8 70 80 50 -10 yes
9 9 40 90 -60 20 yes
10 10 15 65 25 20 yes
# ℹ 14 more rowsWhat does the above function do?
mutate() is a very easy way to edit your data mid-pipe. So we might want to do some calculations, create a temporary variable using mutate(), and then continue our pipe.
# A tibble: 12 × 5
Catechin `PC B2` `Chlorogenic acid` equivalents in gallic acid m…¹ is_outlier
<dbl> <dbl> <dbl> <dbl> <lgl>
1 0.2 1 5 2.79 FALSE
2 0.2 1 5 2.73 FALSE
3 0.2 1 5 2.74 FALSE
4 0.2 30 5 15.2 FALSE
5 0.2 30 5 15.3 FALSE
6 0.2 30 5 14.0 FALSE
7 30 1 5 14.0 FALSE
8 30 1 5 13.8 FALSE
9 30 1 5 13.9 FALSE
10 30 30 5 21.9 FALSE
11 30 30 5 18.0 FALSE
12 30 30 5 17.3 FALSE
# ℹ abbreviated name: ¹`equivalents in gallic acid mg/L`What did we do above? Was it a good idea? Most importantly, did we actually change polyphenols? Check in your work environment to see.
Split-apply-combine: group_by(), count(), and summarize()
We are only scratching the surface of what we can do with tidyverse, and you will both be reading more about this and we’ll be returning throughout the semester as we learn more application. But here is an exploratory data analysis application that we will see here and will come back to us frequently.
The group_by() function takes a data frame and groups it by whatever variable is specified. It looks for distinct values, so it will work with even numeric variables (although not well, if they are not truly grouping variables, such as actual observed values).
polyphenols |>
group_by(Catechin) # note no quotes# A tibble: 81 × 4
# Groups: Catechin [3]
Catechin `PC B2` `Chlorogenic acid` `equivalents in gallic acid mg/L`
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 1 200 57.7
5 0.2 1 200 58.2
6 0.2 1 200 69.2
7 0.2 1 500 150.
8 0.2 1 500 162.
9 0.2 1 500 161.
10 0.2 30 5 15.2
# ℹ 71 more rowsBy itself, this does not appear to do anything, although if you look closely at the output from printing the tibble you’ll notice that it now lists Groups: Catechin [3].
We can use this to split the tibble into groups and do things to each group. For example, the n() function gives the number of rows in a group within a tibble (when used in mutate(), etc) and so, if we have 3 groups, n() will tell us for each group how many rows are in that group:
# A tibble: 81 × 5
# Groups: Catechin [3]
Catechin `PC B2` `Chlorogenic acid` equivalents in gallic ac…¹ number_of_rows
<dbl> <dbl> <dbl> <dbl> <int>
1 0.2 1 5 2.79 27
2 0.2 1 5 2.73 27
3 0.2 1 5 2.74 27
4 0.2 1 200 57.7 27
5 0.2 1 200 58.2 27
6 0.2 1 200 69.2 27
7 0.2 1 500 150. 27
8 0.2 1 500 162. 27
9 0.2 1 500 161. 27
10 0.2 30 5 15.2 27
# ℹ 71 more rows
# ℹ abbreviated name: ¹`equivalents in gallic acid mg/L`…of course, this is very boring because this is a balanced factorial-design experiment with complete data, so each group of Catechin levels has 27 rows!
But this now allows us to use filter(), mutate(), and a host of other tidyverse functions in “grouped” mode, which means when we do all of those they will act WITHIN the group you have defined. So we can see the effect, we will first filter polyphenols, and then create a new variable with a grouped mutate() that is the average for that group.
# A tibble: 27 × 5
# Groups: Catechin [3]
Catechin `PC B2` `Chlorogenic acid` equivalents in gallic aci…¹ average_eq_gc
<dbl> <dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79 21.8
2 0.2 1 5 2.73 21.8
3 0.2 1 5 2.74 21.8
4 0.2 30 5 15.2 21.8
5 0.2 30 5 15.3 21.8
6 0.2 30 5 14.0 21.8
7 0.2 100 5 46.6 21.8
8 0.2 100 5 46.1 21.8
9 0.2 100 5 50.9 21.8
10 30 1 5 14.0 28.4
# ℹ 17 more rows
# ℹ abbreviated name: ¹`equivalents in gallic acid mg/L`But that’s not all! Now that we have the idea of a group, we can use it for all kinds of useful things. For example, the count() function will just return the number of rows in each group. Let’s use that to check if our polyphenols was from a balanced design (e.g., has the same number of observations at every treatment level and intersection):
# A tibble: 27 × 4
# Groups: Catechin, Chlorogenic acid, PC B2 [27]
Catechin `Chlorogenic acid` `PC B2` n
<dbl> <dbl> <dbl> <int>
1 0.2 5 1 3
2 0.2 5 30 3
3 0.2 5 100 3
4 0.2 200 1 3
5 0.2 200 30 3
6 0.2 200 100 3
7 0.2 500 1 3
8 0.2 500 30 3
9 0.2 500 100 3
10 30 5 1 3
# ℹ 17 more rowsThe count() function gets way more useful when we have large data that is unbalanced, and we want to be able to easily pull counts without knowing what they are beforehand.
Much more useful, but slightly more complex is the summarize() function, which applies arbitrary summary functions to the subsets defined by group_by(). So, say we want to find the average and standard deviation at each of the groups we defined before, instead of the count:
# A tibble: 27 × 5
# Groups: Catechin, PC B2 [9]
Catechin `PC B2` `Chlorogenic acid` mean_eq_gc sd_eq_gc
<dbl> <dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.75 0.0321
2 0.2 1 200 61.7 6.52
3 0.2 1 500 158. 6.42
4 0.2 30 5 14.8 0.700
5 0.2 30 200 63.3 0.824
6 0.2 30 500 155. 16.3
7 0.2 100 5 47.9 2.64
8 0.2 100 200 90.6 5.78
9 0.2 100 500 155. 5.52
10 30 1 5 13.9 0.107
# ℹ 17 more rowsEven I think that’s pretty cool, and I wrote the code! This can be a bit tricky to get your head around, so let’s talk through it.
The key intuition is that summarize() takes multiple rows, identified by the same grouping variable (here the triplicate measurement reps identified by the combinations of treatment variables) and applies a function to each group that returns a single number: remember that mean() gives the mean (average) of a set of numeric observations, and sd() gives the standard deviaiton.
NB: One final tip: you can always run part of a piped data flow by selecting just that part and hitting “run” in the right top corner of your RStudio coding panel or hitting ctrl/cmd + enter. This is useful for testing your code flow.
Utilities for data management
Honestly, the amount of power in tidyverse is way more than we can cover today, and is covered more comprehensively (obviously) by Wickham and Grolemund (2017), including your reading for this week. Instead, I want to draw attention to a few “quality of life” functions that tidyverse also provides that make tasks that are overly difficult in R a little easier.
rename()
A common need in data analysis is renaming your columns, because they are called something clear but not machine-readable/easily typable in your data file, and they turn into a mess in R. We have that problem in our polyphenols dataset:
names(polyphenols)[1] "Catechin" "PC B2"
[3] "Chlorogenic acid" "equivalents in gallic acid mg/L"Even with tab-completion, those are a pain to deal with. Renaming these in base R is a huge pain, in fact. You can copy them to a new column by using the <- operator, then remove the original column using the [] operator with negative indices. Or you can use the names() <- operator. Neither is very easy to do on the fly.
In tidyverse, the rename() function is there to help. It works in two, equally useful ways.
# New names go on the left hand side, old names on the right
polyphenols |>
rename(chlor = `Chlorogenic acid`,
pcb2 = `PC B2`,
cat = Catechin,
eq_ga = `equivalents in gallic acid mg/L`)# A tibble: 81 × 4
cat pcb2 chlor eq_ga
<dbl> <dbl> <dbl> <dbl>
1 0.2 1 5 2.79
2 0.2 1 5 2.73
3 0.2 1 5 2.74
4 0.2 1 200 57.7
5 0.2 1 200 58.2
6 0.2 1 200 69.2
7 0.2 1 500 150.
8 0.2 1 500 162.
9 0.2 1 500 161.
10 0.2 30 5 15.2
# ℹ 71 more rows# Alternatively, we can rename by INDEX, with new name on the left and column #
# on the right. This is easier for programming, but is less safe because you
# have to be certain you're matching columns correctly.
# don't forget we have to actually save the changes if we want them to stick
polyphenols <-
polyphenols |>
rename(cat = 1, pcb2 = 2, chlor = 3, eq_ga = 4)arrange()
Another thing we often want to do is sort the rows of a data table according to some criterion. Doing this in base R requires the use of sort(), which works only on vectors, not on data frames, and so requires a similar approach to the base R filtering approach we saw above.
The tidyverse version is very simple: arrange() just requires that you tell it a data frame and one or more columns to sort by.
# show us the data in ascending order of liking for hotandsour1
soups |>
arrange(hotandsour1) # A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 12 75 25 -100 -100
2 21 20 100 -100 60
3 23 10 100 -100 85
4 4 20 60 -80 65
5 13 -25 50 -75 -10
6 14 20 40 -75 60
7 9 40 90 -60 20
8 18 40 90 -60 -15
9 22 20 60 -60 -10
10 1 0 50 -50 10
# ℹ 14 more rows# writing "-" before the sorting column goes in descending order
soups |>
arrange(-hotandsour1) # A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 8 70 80 50 -10
2 7 75 90 40 80
3 5 65 96 37 99
4 24 10 75 35 55
5 17 15 90 30 50
6 10 15 65 25 20
7 3 80 97 10 37
8 11 -50 75 -15 50
9 2 70 80 -20 20
10 15 70 90 -25 -60
# ℹ 14 more rows# this sorts first ascending by hotandsour1, then within that descending by hotandsour2
soups |>
arrange(hotandsour1, -hotandsour2) # A tibble: 24 × 5
subject minestrone1 minestrone2 hotandsour1 hotandsour2
<dbl> <dbl> <dbl> <dbl> <dbl>
1 23 10 100 -100 85
2 21 20 100 -100 60
3 12 75 25 -100 -100
4 4 20 60 -80 65
5 14 20 40 -75 60
6 13 -25 50 -75 -10
7 9 40 90 -60 20
8 22 20 60 -60 -10
9 18 40 90 -60 -15
10 16 20 90 -50 30
# ℹ 14 more rowsrelocate()
R doesn’t really care about the order of columns in a data frame, but we might, for example
- If we have a lot of columns, they might not print properly in the console, requiring some work every time we want to check their values.
- We might be outputting data to a
knitr::kable()or other display mode
We can reorder columns using similar syntax to our base R select operations above, but this requires many steps. Again, the simple tidyverse solution is a single function: relocate():
# by default the column(s) are moved to the "front" of the table
soups |>
relocate(hotandsour2) # A tibble: 24 × 5
hotandsour2 subject minestrone1 minestrone2 hotandsour1
<dbl> <dbl> <dbl> <dbl> <dbl>
1 10 1 0 50 -50
2 20 2 70 80 -20
3 37 3 80 97 10
4 65 4 20 60 -80
5 99 5 65 96 37
6 20 6 30 80 -50
7 80 7 75 90 40
8 -10 8 70 80 50
9 20 9 40 90 -60
10 20 10 15 65 25
# ℹ 14 more rows# .before and .after (note the ".") can be used to move to specific spots
soups |>
relocate(hotandsour2, .after = subject) # A tibble: 24 × 5
subject hotandsour2 minestrone1 minestrone2 hotandsour1
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 10 0 50 -50
2 2 20 70 80 -20
3 3 37 80 97 10
4 4 65 20 60 -80
5 5 99 65 96 37
6 6 20 30 80 -50
7 7 80 75 90 40
8 8 -10 70 80 50
9 9 20 40 90 -60
10 10 20 15 65 25
# ℹ 14 more rowsReview of functions
Finally, I want to leave some time today, if we have any, to discuss the material from last week on functions. This is a huge topic, and I am sure you still have some questions. Let’s have some answers!
Reading
This week, you should read:
- R for Data Science, Chapters 5, 9-11, 18
- Stat 545, Chapters 5-9
- Data Visualization, Chapter 2, A.2
Session info
R version 4.5.3 (2026-03-11)
Platform: aarch64-apple-darwin20
Running under: macOS Tahoe 26.5.1
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: America/New_York
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] readxl_1.4.5 lubridate_1.9.5 forcats_1.0.1 stringr_1.6.0
[5] dplyr_1.2.1 purrr_1.2.2 readr_2.2.0 tidyr_1.3.2
[9] tibble_3.3.1 ggplot2_4.0.2 tidyverse_2.0.0
loaded via a namespace (and not attached):
[1] rematch_2.0.0 bit_4.6.0 gtable_0.3.6 jsonlite_2.0.0
[5] crayon_1.5.3 compiler_4.5.3 tidyselect_1.2.1 parallel_4.5.3
[9] scales_1.4.0 yaml_2.3.12 fastmap_1.2.0 R6_2.6.1
[13] generics_0.1.4 knitr_1.51 htmlwidgets_1.6.4 pillar_1.11.1
[17] RColorBrewer_1.1-3 tzdb_0.5.0 rlang_1.2.0 utf8_1.2.6
[21] stringi_1.8.7 xfun_0.57 S7_0.2.1 bit64_4.6.0-1
[25] otel_0.2.0 timechange_0.4.0 cli_3.6.6 withr_3.0.2
[29] magrittr_2.0.5 digest_0.6.39 grid_4.5.3 vroom_1.7.1
[33] rstudioapi_0.18.0 hms_1.1.4 lifecycle_1.0.5 vctrs_0.7.3
[37] evaluate_1.0.5 glue_1.8.1 cellranger_1.1.0 farver_2.1.2
[41] rmarkdown_2.31 tools_4.5.3 pkgconfig_2.0.3 htmltools_0.5.9 References
Wickham, Hadley, and Garrett Grolemund. 2017. R for Data Science. O’Reilly.