- •brief contents
- •contents
- •preface
- •acknowledgments
- •about this book
- •What’s new in the second edition
- •Who should read this book
- •Roadmap
- •Advice for data miners
- •Code examples
- •Code conventions
- •Author Online
- •About the author
- •about the cover illustration
- •1 Introduction to R
- •1.2 Obtaining and installing R
- •1.3 Working with R
- •1.3.1 Getting started
- •1.3.2 Getting help
- •1.3.3 The workspace
- •1.3.4 Input and output
- •1.4 Packages
- •1.4.1 What are packages?
- •1.4.2 Installing a package
- •1.4.3 Loading a package
- •1.4.4 Learning about a package
- •1.5 Batch processing
- •1.6 Using output as input: reusing results
- •1.7 Working with large datasets
- •1.8 Working through an example
- •1.9 Summary
- •2 Creating a dataset
- •2.1 Understanding datasets
- •2.2 Data structures
- •2.2.1 Vectors
- •2.2.2 Matrices
- •2.2.3 Arrays
- •2.2.4 Data frames
- •2.2.5 Factors
- •2.2.6 Lists
- •2.3 Data input
- •2.3.1 Entering data from the keyboard
- •2.3.2 Importing data from a delimited text file
- •2.3.3 Importing data from Excel
- •2.3.4 Importing data from XML
- •2.3.5 Importing data from the web
- •2.3.6 Importing data from SPSS
- •2.3.7 Importing data from SAS
- •2.3.8 Importing data from Stata
- •2.3.9 Importing data from NetCDF
- •2.3.10 Importing data from HDF5
- •2.3.11 Accessing database management systems (DBMSs)
- •2.3.12 Importing data via Stat/Transfer
- •2.4 Annotating datasets
- •2.4.1 Variable labels
- •2.4.2 Value labels
- •2.5 Useful functions for working with data objects
- •2.6 Summary
- •3 Getting started with graphs
- •3.1 Working with graphs
- •3.2 A simple example
- •3.3 Graphical parameters
- •3.3.1 Symbols and lines
- •3.3.2 Colors
- •3.3.3 Text characteristics
- •3.3.4 Graph and margin dimensions
- •3.4 Adding text, customized axes, and legends
- •3.4.1 Titles
- •3.4.2 Axes
- •3.4.3 Reference lines
- •3.4.4 Legend
- •3.4.5 Text annotations
- •3.4.6 Math annotations
- •3.5 Combining graphs
- •3.5.1 Creating a figure arrangement with fine control
- •3.6 Summary
- •4 Basic data management
- •4.1 A working example
- •4.2 Creating new variables
- •4.3 Recoding variables
- •4.4 Renaming variables
- •4.5 Missing values
- •4.5.1 Recoding values to missing
- •4.5.2 Excluding missing values from analyses
- •4.6 Date values
- •4.6.1 Converting dates to character variables
- •4.6.2 Going further
- •4.7 Type conversions
- •4.8 Sorting data
- •4.9 Merging datasets
- •4.9.1 Adding columns to a data frame
- •4.9.2 Adding rows to a data frame
- •4.10 Subsetting datasets
- •4.10.1 Selecting (keeping) variables
- •4.10.2 Excluding (dropping) variables
- •4.10.3 Selecting observations
- •4.10.4 The subset() function
- •4.10.5 Random samples
- •4.11 Using SQL statements to manipulate data frames
- •4.12 Summary
- •5 Advanced data management
- •5.2 Numerical and character functions
- •5.2.1 Mathematical functions
- •5.2.2 Statistical functions
- •5.2.3 Probability functions
- •5.2.4 Character functions
- •5.2.5 Other useful functions
- •5.2.6 Applying functions to matrices and data frames
- •5.3 A solution for the data-management challenge
- •5.4 Control flow
- •5.4.1 Repetition and looping
- •5.4.2 Conditional execution
- •5.5 User-written functions
- •5.6 Aggregation and reshaping
- •5.6.1 Transpose
- •5.6.2 Aggregating data
- •5.6.3 The reshape2 package
- •5.7 Summary
- •6 Basic graphs
- •6.1 Bar plots
- •6.1.1 Simple bar plots
- •6.1.2 Stacked and grouped bar plots
- •6.1.3 Mean bar plots
- •6.1.4 Tweaking bar plots
- •6.1.5 Spinograms
- •6.2 Pie charts
- •6.3 Histograms
- •6.4 Kernel density plots
- •6.5 Box plots
- •6.5.1 Using parallel box plots to compare groups
- •6.5.2 Violin plots
- •6.6 Dot plots
- •6.7 Summary
- •7 Basic statistics
- •7.1 Descriptive statistics
- •7.1.1 A menagerie of methods
- •7.1.2 Even more methods
- •7.1.3 Descriptive statistics by group
- •7.1.4 Additional methods by group
- •7.1.5 Visualizing results
- •7.2 Frequency and contingency tables
- •7.2.1 Generating frequency tables
- •7.2.2 Tests of independence
- •7.2.3 Measures of association
- •7.2.4 Visualizing results
- •7.3 Correlations
- •7.3.1 Types of correlations
- •7.3.2 Testing correlations for significance
- •7.3.3 Visualizing correlations
- •7.4 T-tests
- •7.4.3 When there are more than two groups
- •7.5 Nonparametric tests of group differences
- •7.5.1 Comparing two groups
- •7.5.2 Comparing more than two groups
- •7.6 Visualizing group differences
- •7.7 Summary
- •8 Regression
- •8.1 The many faces of regression
- •8.1.1 Scenarios for using OLS regression
- •8.1.2 What you need to know
- •8.2 OLS regression
- •8.2.1 Fitting regression models with lm()
- •8.2.2 Simple linear regression
- •8.2.3 Polynomial regression
- •8.2.4 Multiple linear regression
- •8.2.5 Multiple linear regression with interactions
- •8.3 Regression diagnostics
- •8.3.1 A typical approach
- •8.3.2 An enhanced approach
- •8.3.3 Global validation of linear model assumption
- •8.3.4 Multicollinearity
- •8.4 Unusual observations
- •8.4.1 Outliers
- •8.4.3 Influential observations
- •8.5 Corrective measures
- •8.5.1 Deleting observations
- •8.5.2 Transforming variables
- •8.5.3 Adding or deleting variables
- •8.5.4 Trying a different approach
- •8.6 Selecting the “best” regression model
- •8.6.1 Comparing models
- •8.6.2 Variable selection
- •8.7 Taking the analysis further
- •8.7.1 Cross-validation
- •8.7.2 Relative importance
- •8.8 Summary
- •9 Analysis of variance
- •9.1 A crash course on terminology
- •9.2 Fitting ANOVA models
- •9.2.1 The aov() function
- •9.2.2 The order of formula terms
- •9.3.1 Multiple comparisons
- •9.3.2 Assessing test assumptions
- •9.4 One-way ANCOVA
- •9.4.1 Assessing test assumptions
- •9.4.2 Visualizing the results
- •9.6 Repeated measures ANOVA
- •9.7 Multivariate analysis of variance (MANOVA)
- •9.7.1 Assessing test assumptions
- •9.7.2 Robust MANOVA
- •9.8 ANOVA as regression
- •9.9 Summary
- •10 Power analysis
- •10.1 A quick review of hypothesis testing
- •10.2 Implementing power analysis with the pwr package
- •10.2.1 t-tests
- •10.2.2 ANOVA
- •10.2.3 Correlations
- •10.2.4 Linear models
- •10.2.5 Tests of proportions
- •10.2.7 Choosing an appropriate effect size in novel situations
- •10.3 Creating power analysis plots
- •10.4 Other packages
- •10.5 Summary
- •11 Intermediate graphs
- •11.1 Scatter plots
- •11.1.3 3D scatter plots
- •11.1.4 Spinning 3D scatter plots
- •11.1.5 Bubble plots
- •11.2 Line charts
- •11.3 Corrgrams
- •11.4 Mosaic plots
- •11.5 Summary
- •12 Resampling statistics and bootstrapping
- •12.1 Permutation tests
- •12.2 Permutation tests with the coin package
- •12.2.2 Independence in contingency tables
- •12.2.3 Independence between numeric variables
- •12.2.5 Going further
- •12.3 Permutation tests with the lmPerm package
- •12.3.1 Simple and polynomial regression
- •12.3.2 Multiple regression
- •12.4 Additional comments on permutation tests
- •12.5 Bootstrapping
- •12.6 Bootstrapping with the boot package
- •12.6.1 Bootstrapping a single statistic
- •12.6.2 Bootstrapping several statistics
- •12.7 Summary
- •13 Generalized linear models
- •13.1 Generalized linear models and the glm() function
- •13.1.1 The glm() function
- •13.1.2 Supporting functions
- •13.1.3 Model fit and regression diagnostics
- •13.2 Logistic regression
- •13.2.1 Interpreting the model parameters
- •13.2.2 Assessing the impact of predictors on the probability of an outcome
- •13.2.3 Overdispersion
- •13.2.4 Extensions
- •13.3 Poisson regression
- •13.3.1 Interpreting the model parameters
- •13.3.2 Overdispersion
- •13.3.3 Extensions
- •13.4 Summary
- •14 Principal components and factor analysis
- •14.1 Principal components and factor analysis in R
- •14.2 Principal components
- •14.2.1 Selecting the number of components to extract
- •14.2.2 Extracting principal components
- •14.2.3 Rotating principal components
- •14.2.4 Obtaining principal components scores
- •14.3 Exploratory factor analysis
- •14.3.1 Deciding how many common factors to extract
- •14.3.2 Extracting common factors
- •14.3.3 Rotating factors
- •14.3.4 Factor scores
- •14.4 Other latent variable models
- •14.5 Summary
- •15 Time series
- •15.1 Creating a time-series object in R
- •15.2 Smoothing and seasonal decomposition
- •15.2.1 Smoothing with simple moving averages
- •15.2.2 Seasonal decomposition
- •15.3 Exponential forecasting models
- •15.3.1 Simple exponential smoothing
- •15.3.3 The ets() function and automated forecasting
- •15.4 ARIMA forecasting models
- •15.4.1 Prerequisite concepts
- •15.4.2 ARMA and ARIMA models
- •15.4.3 Automated ARIMA forecasting
- •15.5 Going further
- •15.6 Summary
- •16 Cluster analysis
- •16.1 Common steps in cluster analysis
- •16.2 Calculating distances
- •16.3 Hierarchical cluster analysis
- •16.4 Partitioning cluster analysis
- •16.4.2 Partitioning around medoids
- •16.5 Avoiding nonexistent clusters
- •16.6 Summary
- •17 Classification
- •17.1 Preparing the data
- •17.2 Logistic regression
- •17.3 Decision trees
- •17.3.1 Classical decision trees
- •17.3.2 Conditional inference trees
- •17.4 Random forests
- •17.5 Support vector machines
- •17.5.1 Tuning an SVM
- •17.6 Choosing a best predictive solution
- •17.7 Using the rattle package for data mining
- •17.8 Summary
- •18 Advanced methods for missing data
- •18.1 Steps in dealing with missing data
- •18.2 Identifying missing values
- •18.3 Exploring missing-values patterns
- •18.3.1 Tabulating missing values
- •18.3.2 Exploring missing data visually
- •18.3.3 Using correlations to explore missing values
- •18.4 Understanding the sources and impact of missing data
- •18.5 Rational approaches for dealing with incomplete data
- •18.6 Complete-case analysis (listwise deletion)
- •18.7 Multiple imputation
- •18.8 Other approaches to missing data
- •18.8.1 Pairwise deletion
- •18.8.2 Simple (nonstochastic) imputation
- •18.9 Summary
- •19 Advanced graphics with ggplot2
- •19.1 The four graphics systems in R
- •19.2 An introduction to the ggplot2 package
- •19.3 Specifying the plot type with geoms
- •19.4 Grouping
- •19.5 Faceting
- •19.6 Adding smoothed lines
- •19.7 Modifying the appearance of ggplot2 graphs
- •19.7.1 Axes
- •19.7.2 Legends
- •19.7.3 Scales
- •19.7.4 Themes
- •19.7.5 Multiple graphs per page
- •19.8 Saving graphs
- •19.9 Summary
- •20 Advanced programming
- •20.1 A review of the language
- •20.1.1 Data types
- •20.1.2 Control structures
- •20.1.3 Creating functions
- •20.2 Working with environments
- •20.3 Object-oriented programming
- •20.3.1 Generic functions
- •20.3.2 Limitations of the S3 model
- •20.4 Writing efficient code
- •20.5 Debugging
- •20.5.1 Common sources of errors
- •20.5.2 Debugging tools
- •20.5.3 Session options that support debugging
- •20.6 Going further
- •20.7 Summary
- •21 Creating a package
- •21.1 Nonparametric analysis and the npar package
- •21.1.1 Comparing groups with the npar package
- •21.2 Developing the package
- •21.2.1 Computing the statistics
- •21.2.2 Printing the results
- •21.2.3 Summarizing the results
- •21.2.4 Plotting the results
- •21.2.5 Adding sample data to the package
- •21.3 Creating the package documentation
- •21.4 Building the package
- •21.5 Going further
- •21.6 Summary
- •22 Creating dynamic reports
- •22.1 A template approach to reports
- •22.2 Creating dynamic reports with R and Markdown
- •22.3 Creating dynamic reports with R and LaTeX
- •22.4 Creating dynamic reports with R and Open Document
- •22.5 Creating dynamic reports with R and Microsoft Word
- •22.6 Summary
- •afterword Into the rabbit hole
- •appendix A Graphical user interfaces
- •appendix B Customizing the startup environment
- •appendix C Exporting data from R
- •Delimited text file
- •Excel spreadsheet
- •Statistical applications
- •appendix D Matrix algebra in R
- •appendix E Packages used in this book
- •appendix F Working with large datasets
- •F.1 Efficient programming
- •F.2 Storing data outside of RAM
- •F.3 Analytic packages for out-of-memory data
- •F.4 Comprehensive solutions for working with enormous datasets
- •appendix G Updating an R installation
- •G.1 Automated installation (Windows only)
- •G.2 Manual installation (Windows and Mac OS X)
- •G.3 Updating an R installation (Linux)
- •references
- •index
- •Symbols
- •Numerics
- •23.1 The lattice package
- •23.2 Conditioning variables
- •23.3 Panel functions
- •23.4 Grouping variables
- •23.5 Graphic parameters
- •23.6 Customizing plot strips
- •23.7 Page arrangement
- •23.8 Going further
Creating dynamic reports with R and Markdown |
517 |
The examples in this chapter are based on descriptive statistics, regression, and ANOVA problems. None of them represent full analyses of the data. The goal in this chapter is to learn how to incorporate the R results into various types of reports. Feel free to jump around in this chapter, reading the sections that are most relevant to you.
Depending on the template file you start with and the functions used to process it, different report formats (HTML web pages, Microsoft Word documents, OpenOffice Writer documents, PDF reports, articles, and books) are created. The reports are dynamic in the sense that changing the data and reprocessing the template file will result in a new report.
In this chapter, you’ll work with four types of templates: an R Markdown template, an ODT template, a DOCX template, and a LaTeX template. R Markdown templates can be used to create HTML, PDF, and MS Word documents. ODT and DOCX templates are used to create Open Document and Microsoft Word documents, respectively. LaTeX templates are used to create publication-quality PDF documents, including reports, articles, and books. Let’s consider each in turn.
22.2 Creating dynamic reports with R and Markdown
In this section, you’ll use the rmarkdown package to create documents generated from Markdown syntax and R code. When the document is processed, the R code is executed, and the output is formatted and embedded in the finished document. You can use this approach to generate reports as HTML, Word, or PDF documents. Here are the steps:
1Install the rmarkdown package (install.packages("rmarkdown")). This will install several other packages including knitr. If you’re using a recent version of RStudio, you can skip this step because you already have the necessary packages.
2Install the xtable package (install.packages("xtable")). The xtable() function in this package attractively formats data frames and matrices for inclusion in reports. xtable() can also format objects produced by the lm(), glm(), aov(), table(), ts(), and coxph() functions. After loading the package, use methods(xtable) to view a comprehensive list of the objects it can format.
3Install Pandoc (http://johnmacfarlane.net/pandoc/index.html). Pandoc is a free application available for Windows, Mac OS X, and Linux. It converts files from one markup format to another. Again, RStudio users can skip this step.
4If you want to create PDF documents, install a LaTeX compiler. A LaTeX compiler converts a LaTeX document into a high-quality typeset PDF document. I recommend MiKTeX (www.miktex.org) for Windows, MacTeX for Macs (http:// tug.org/mactex), and TeX Live for Linux (www.tug.org/texlive).
With the software set up, you’re ready to go.
To incorporate R output (values, tables, graphs) in a document using Markdown syntax, first create a text document that contains
■Report text
■Markdown syntax
■R code chunks (R code surrounded by delimiters)
518 |
CHAPTER 22 Creating dynamic reports |
By convention, the text file has the filename extension .Rmd.
A sample file (named women.Rmd) is provided in listing 22.1. To generate an HTML document, process this file using
library(rmarkdown) render("women.Rmd", "html_document")
The results are displayed in figure 22.1.
Listing 22.1 women.Rmd: a Markdown template with embedded R code
# Regression Report
```{r echo=FALSE, results='hide'} n <- nrow(women)
fit <- lm(weight ~ height, data=women) sfit <- summary(fit)
b <- coefficients(fit)
```
b Markdown syntax
c R code chunk
R inline code d
Linear regression was used to model the relationship between weights and height in a sample of `r n` women. The equation
**weight = `r b[[1]]` + `r b[[2]]` * height** accounted for `r round(sfit$r.squared,2)`% of the variance in weights. The ANOVA table is given below.
```{r echo=FALSE, results='asis'} library(xtable) options(xtable.comment=FALSE)
print(xtable(sfit), type="html", html.table.attributes="border=0")
```
The regression is plotted in the following figure.
```{r echo=FALSE, fig.width=5, fig.height=4} library(ggplot2)
ggplot(data=women, aes(x=height, y=weight)) + geom_point() + geom_smooth(method="lm")
```
e Formats output with xtable
The report starts with a first-level header b. It indicates that “Regression Report” should be printed in a large, bold font. Examples of other Markdown syntax are given in table 22.1.
Table 22.1 Markdown code and the resulting output
Markdown syntax |
Resulting HTML output |
|
|
# Heading 1 |
<h1>Heading 1</h1> |
## Heading 2 |
<h2>Heading 2</h2> |
... |
... |
###### Heading 6 |
<h6>Heading 2</h6> |
One or more blank lines between text |
Separates text into paragraphs |
|
|
Creating dynamic reports with R and Markdown |
519 |
|
Table 22.1 Markdown code and the resulting output |
|
|
|
|
|
Markdown syntax |
Resulting HTML output |
|
|
|
|
Two or more spaces at the end of a line |
Adds a line break |
|
*I mean it* |
<em>I mean it</em> |
|
**I really mean it** |
<strong>I really mean it</strong> |
|
* item 1 |
<ul> |
|
* item 2 |
<li> item 1 </li> |
|
|
<li> item 2 </li> |
|
|
</ul> |
|
1. item 1 |
<ol> |
|
2. item 2 |
<li> item 1 </li> |
|
|
<li> item 2 </li> |
|
|
</ol> |
|
[Google](http://google.com) |
<a href="http://google.com">Google</a> |
|
 |
<img src="path to image", alt="My text"> |
|
|
|
|
Next comes an R code chunk. R code in Markdown documents is delimited by ```{r
options} and ``` c. When the file is processed, the R code is executed and the results are inserted. Code chunk options are described in table 22.2.
Table 22.2 Code chunk options
Option |
Description |
|
|
echo |
Whether to include the R source code in the output (TRUE) or not (FALSE) |
results |
Whether to output raw results (asis) or hide the results (hide) |
warning |
Whether to include warnings in the output (TRUE) or not (FALSE) |
message |
Whether to include informational messages in the output (TRUE) or not (FALSE) |
error |
Whether to include error messages in in the output (TRUE) or not (FALSE) |
fig.width |
Figure width for plots (inches) |
fig.height |
Figure height for plots (inches) |
|
|
Simple R output (a number or string) can also be placed directly within report text. This inline R code allows you to customize the text in individual sentences. Inline code is placed between `r and ` tags d. In the regression example, the sample size, prediction equation, and R-squared value are embedded in the first paragraph.
Finally, you use the xtable() function to format the regression results e. The statement options(xtable.comment=FALSE) suppresses superfluous messages. The type="html" option in the print() function outputs the xtable object as an HTML table. By default, this table has an unattractive 1-pixel border that’s removed by
520 |
CHAPTER 22 Creating dynamic reports |
adding |
html.table.attributes="border=0". See help(print.xtable) for addi- |
tional formatting options.
To render the file as a PDF document, you only have to make one change. Replace
print(xtable(sfit), type="html", html.table.attributes="border=0")
with
print(xtable(sfit), type="latex")
Then process the file using
library(rmarkdown) render("women.Rmd", "pdf_document")
to get a nicely formatted PDF document.
Unfortunately, the xtable() function doesn’t work for Word documents. You’ll have to get a bit more creative to render statistical output in an attractive fashion. One possibility is to replace xtable() with the kable() function in the knitr package. It can render matrices and data frames in a simple and appealing manner.
Replace
library(xtable)
options(xtable.comment=FALSE)
print(xtable(sfit), type="html", html.table.attributes="border=0")
with
library(knitr) kable(sfit$coefficients)
Then render the file using
library(rmarkdown) render("women.Rmd", "word_document")
The result is an attractive Word document that you can edit using Word. Note that you had to replace the sfit object with sfit$coefficients. The xtable() function can handle lm() objects, but the kable() function can only handle matrices and data frames. Therefore, you have to extract the parts you want to print from more complicated objects. See help(kable) for more details.
Using RStudio to create and process R Markdown documents
Throughout this book, I’ve tried to keep the presentation independent of the interface used to access R. Each of the techniques described will work in the basic R Console. But there are several other options, including RStudio (see appendix A). RStudio makes it particularly easy to render reports from Markdown documents.
If you choose File > New File > R Markdown from the GUI menu, you’ll see the dialog box shown next.
Creating dynamic reports with R and Markdown |
521 |
Dialog box for creating a new
R Markdown document in
RStudio
Choose the type of report you want to generate, and RStudio will create a skeleton file for you. Edit it with your text and code, and then select the rendering option from the Knit drop-down list. That’s it!
Drop-down menu for generating an HTML, PDF, or Word report from an R Markdown document
RStudio has many useful features for programmers. It’s by far my favorite way to work in R.
Markdown syntax is convenient for creating simple documents quickly. To learn more about Markdown, visit the homepage at http://daringfireball.net/projects/markdown and the rmarkdown documentation at http://rmarkdown.rstudio.com. If you want to create complex documents such as publication-quality articles and books, then you may want to look at using LaTeX as your markup language. In the next section, you’ll use LaTeX and the knitr package to create high-quality typeset documents.
