- •brief contents
- •contents
- •preface
- •acknowledgments
- •about this book
- •What’s new in the second edition
- •Who should read this book
- •Roadmap
- •Advice for data miners
- •Code examples
- •Code conventions
- •Author Online
- •About the author
- •about the cover illustration
- •1 Introduction to R
- •1.2 Obtaining and installing R
- •1.3 Working with R
- •1.3.1 Getting started
- •1.3.2 Getting help
- •1.3.3 The workspace
- •1.3.4 Input and output
- •1.4 Packages
- •1.4.1 What are packages?
- •1.4.2 Installing a package
- •1.4.3 Loading a package
- •1.4.4 Learning about a package
- •1.5 Batch processing
- •1.6 Using output as input: reusing results
- •1.7 Working with large datasets
- •1.8 Working through an example
- •1.9 Summary
- •2 Creating a dataset
- •2.1 Understanding datasets
- •2.2 Data structures
- •2.2.1 Vectors
- •2.2.2 Matrices
- •2.2.3 Arrays
- •2.2.4 Data frames
- •2.2.5 Factors
- •2.2.6 Lists
- •2.3 Data input
- •2.3.1 Entering data from the keyboard
- •2.3.2 Importing data from a delimited text file
- •2.3.3 Importing data from Excel
- •2.3.4 Importing data from XML
- •2.3.5 Importing data from the web
- •2.3.6 Importing data from SPSS
- •2.3.7 Importing data from SAS
- •2.3.8 Importing data from Stata
- •2.3.9 Importing data from NetCDF
- •2.3.10 Importing data from HDF5
- •2.3.11 Accessing database management systems (DBMSs)
- •2.3.12 Importing data via Stat/Transfer
- •2.4 Annotating datasets
- •2.4.1 Variable labels
- •2.4.2 Value labels
- •2.5 Useful functions for working with data objects
- •2.6 Summary
- •3 Getting started with graphs
- •3.1 Working with graphs
- •3.2 A simple example
- •3.3 Graphical parameters
- •3.3.1 Symbols and lines
- •3.3.2 Colors
- •3.3.3 Text characteristics
- •3.3.4 Graph and margin dimensions
- •3.4 Adding text, customized axes, and legends
- •3.4.1 Titles
- •3.4.2 Axes
- •3.4.3 Reference lines
- •3.4.4 Legend
- •3.4.5 Text annotations
- •3.4.6 Math annotations
- •3.5 Combining graphs
- •3.5.1 Creating a figure arrangement with fine control
- •3.6 Summary
- •4 Basic data management
- •4.1 A working example
- •4.2 Creating new variables
- •4.3 Recoding variables
- •4.4 Renaming variables
- •4.5 Missing values
- •4.5.1 Recoding values to missing
- •4.5.2 Excluding missing values from analyses
- •4.6 Date values
- •4.6.1 Converting dates to character variables
- •4.6.2 Going further
- •4.7 Type conversions
- •4.8 Sorting data
- •4.9 Merging datasets
- •4.9.1 Adding columns to a data frame
- •4.9.2 Adding rows to a data frame
- •4.10 Subsetting datasets
- •4.10.1 Selecting (keeping) variables
- •4.10.2 Excluding (dropping) variables
- •4.10.3 Selecting observations
- •4.10.4 The subset() function
- •4.10.5 Random samples
- •4.11 Using SQL statements to manipulate data frames
- •4.12 Summary
- •5 Advanced data management
- •5.2 Numerical and character functions
- •5.2.1 Mathematical functions
- •5.2.2 Statistical functions
- •5.2.3 Probability functions
- •5.2.4 Character functions
- •5.2.5 Other useful functions
- •5.2.6 Applying functions to matrices and data frames
- •5.3 A solution for the data-management challenge
- •5.4 Control flow
- •5.4.1 Repetition and looping
- •5.4.2 Conditional execution
- •5.5 User-written functions
- •5.6 Aggregation and reshaping
- •5.6.1 Transpose
- •5.6.2 Aggregating data
- •5.6.3 The reshape2 package
- •5.7 Summary
- •6 Basic graphs
- •6.1 Bar plots
- •6.1.1 Simple bar plots
- •6.1.2 Stacked and grouped bar plots
- •6.1.3 Mean bar plots
- •6.1.4 Tweaking bar plots
- •6.1.5 Spinograms
- •6.2 Pie charts
- •6.3 Histograms
- •6.4 Kernel density plots
- •6.5 Box plots
- •6.5.1 Using parallel box plots to compare groups
- •6.5.2 Violin plots
- •6.6 Dot plots
- •6.7 Summary
- •7 Basic statistics
- •7.1 Descriptive statistics
- •7.1.1 A menagerie of methods
- •7.1.2 Even more methods
- •7.1.3 Descriptive statistics by group
- •7.1.4 Additional methods by group
- •7.1.5 Visualizing results
- •7.2 Frequency and contingency tables
- •7.2.1 Generating frequency tables
- •7.2.2 Tests of independence
- •7.2.3 Measures of association
- •7.2.4 Visualizing results
- •7.3 Correlations
- •7.3.1 Types of correlations
- •7.3.2 Testing correlations for significance
- •7.3.3 Visualizing correlations
- •7.4 T-tests
- •7.4.3 When there are more than two groups
- •7.5 Nonparametric tests of group differences
- •7.5.1 Comparing two groups
- •7.5.2 Comparing more than two groups
- •7.6 Visualizing group differences
- •7.7 Summary
- •8 Regression
- •8.1 The many faces of regression
- •8.1.1 Scenarios for using OLS regression
- •8.1.2 What you need to know
- •8.2 OLS regression
- •8.2.1 Fitting regression models with lm()
- •8.2.2 Simple linear regression
- •8.2.3 Polynomial regression
- •8.2.4 Multiple linear regression
- •8.2.5 Multiple linear regression with interactions
- •8.3 Regression diagnostics
- •8.3.1 A typical approach
- •8.3.2 An enhanced approach
- •8.3.3 Global validation of linear model assumption
- •8.3.4 Multicollinearity
- •8.4 Unusual observations
- •8.4.1 Outliers
- •8.4.3 Influential observations
- •8.5 Corrective measures
- •8.5.1 Deleting observations
- •8.5.2 Transforming variables
- •8.5.3 Adding or deleting variables
- •8.5.4 Trying a different approach
- •8.6 Selecting the “best” regression model
- •8.6.1 Comparing models
- •8.6.2 Variable selection
- •8.7 Taking the analysis further
- •8.7.1 Cross-validation
- •8.7.2 Relative importance
- •8.8 Summary
- •9 Analysis of variance
- •9.1 A crash course on terminology
- •9.2 Fitting ANOVA models
- •9.2.1 The aov() function
- •9.2.2 The order of formula terms
- •9.3.1 Multiple comparisons
- •9.3.2 Assessing test assumptions
- •9.4 One-way ANCOVA
- •9.4.1 Assessing test assumptions
- •9.4.2 Visualizing the results
- •9.6 Repeated measures ANOVA
- •9.7 Multivariate analysis of variance (MANOVA)
- •9.7.1 Assessing test assumptions
- •9.7.2 Robust MANOVA
- •9.8 ANOVA as regression
- •9.9 Summary
- •10 Power analysis
- •10.1 A quick review of hypothesis testing
- •10.2 Implementing power analysis with the pwr package
- •10.2.1 t-tests
- •10.2.2 ANOVA
- •10.2.3 Correlations
- •10.2.4 Linear models
- •10.2.5 Tests of proportions
- •10.2.7 Choosing an appropriate effect size in novel situations
- •10.3 Creating power analysis plots
- •10.4 Other packages
- •10.5 Summary
- •11 Intermediate graphs
- •11.1 Scatter plots
- •11.1.3 3D scatter plots
- •11.1.4 Spinning 3D scatter plots
- •11.1.5 Bubble plots
- •11.2 Line charts
- •11.3 Corrgrams
- •11.4 Mosaic plots
- •11.5 Summary
- •12 Resampling statistics and bootstrapping
- •12.1 Permutation tests
- •12.2 Permutation tests with the coin package
- •12.2.2 Independence in contingency tables
- •12.2.3 Independence between numeric variables
- •12.2.5 Going further
- •12.3 Permutation tests with the lmPerm package
- •12.3.1 Simple and polynomial regression
- •12.3.2 Multiple regression
- •12.4 Additional comments on permutation tests
- •12.5 Bootstrapping
- •12.6 Bootstrapping with the boot package
- •12.6.1 Bootstrapping a single statistic
- •12.6.2 Bootstrapping several statistics
- •12.7 Summary
- •13 Generalized linear models
- •13.1 Generalized linear models and the glm() function
- •13.1.1 The glm() function
- •13.1.2 Supporting functions
- •13.1.3 Model fit and regression diagnostics
- •13.2 Logistic regression
- •13.2.1 Interpreting the model parameters
- •13.2.2 Assessing the impact of predictors on the probability of an outcome
- •13.2.3 Overdispersion
- •13.2.4 Extensions
- •13.3 Poisson regression
- •13.3.1 Interpreting the model parameters
- •13.3.2 Overdispersion
- •13.3.3 Extensions
- •13.4 Summary
- •14 Principal components and factor analysis
- •14.1 Principal components and factor analysis in R
- •14.2 Principal components
- •14.2.1 Selecting the number of components to extract
- •14.2.2 Extracting principal components
- •14.2.3 Rotating principal components
- •14.2.4 Obtaining principal components scores
- •14.3 Exploratory factor analysis
- •14.3.1 Deciding how many common factors to extract
- •14.3.2 Extracting common factors
- •14.3.3 Rotating factors
- •14.3.4 Factor scores
- •14.4 Other latent variable models
- •14.5 Summary
- •15 Time series
- •15.1 Creating a time-series object in R
- •15.2 Smoothing and seasonal decomposition
- •15.2.1 Smoothing with simple moving averages
- •15.2.2 Seasonal decomposition
- •15.3 Exponential forecasting models
- •15.3.1 Simple exponential smoothing
- •15.3.3 The ets() function and automated forecasting
- •15.4 ARIMA forecasting models
- •15.4.1 Prerequisite concepts
- •15.4.2 ARMA and ARIMA models
- •15.4.3 Automated ARIMA forecasting
- •15.5 Going further
- •15.6 Summary
- •16 Cluster analysis
- •16.1 Common steps in cluster analysis
- •16.2 Calculating distances
- •16.3 Hierarchical cluster analysis
- •16.4 Partitioning cluster analysis
- •16.4.2 Partitioning around medoids
- •16.5 Avoiding nonexistent clusters
- •16.6 Summary
- •17 Classification
- •17.1 Preparing the data
- •17.2 Logistic regression
- •17.3 Decision trees
- •17.3.1 Classical decision trees
- •17.3.2 Conditional inference trees
- •17.4 Random forests
- •17.5 Support vector machines
- •17.5.1 Tuning an SVM
- •17.6 Choosing a best predictive solution
- •17.7 Using the rattle package for data mining
- •17.8 Summary
- •18 Advanced methods for missing data
- •18.1 Steps in dealing with missing data
- •18.2 Identifying missing values
- •18.3 Exploring missing-values patterns
- •18.3.1 Tabulating missing values
- •18.3.2 Exploring missing data visually
- •18.3.3 Using correlations to explore missing values
- •18.4 Understanding the sources and impact of missing data
- •18.5 Rational approaches for dealing with incomplete data
- •18.6 Complete-case analysis (listwise deletion)
- •18.7 Multiple imputation
- •18.8 Other approaches to missing data
- •18.8.1 Pairwise deletion
- •18.8.2 Simple (nonstochastic) imputation
- •18.9 Summary
- •19 Advanced graphics with ggplot2
- •19.1 The four graphics systems in R
- •19.2 An introduction to the ggplot2 package
- •19.3 Specifying the plot type with geoms
- •19.4 Grouping
- •19.5 Faceting
- •19.6 Adding smoothed lines
- •19.7 Modifying the appearance of ggplot2 graphs
- •19.7.1 Axes
- •19.7.2 Legends
- •19.7.3 Scales
- •19.7.4 Themes
- •19.7.5 Multiple graphs per page
- •19.8 Saving graphs
- •19.9 Summary
- •20 Advanced programming
- •20.1 A review of the language
- •20.1.1 Data types
- •20.1.2 Control structures
- •20.1.3 Creating functions
- •20.2 Working with environments
- •20.3 Object-oriented programming
- •20.3.1 Generic functions
- •20.3.2 Limitations of the S3 model
- •20.4 Writing efficient code
- •20.5 Debugging
- •20.5.1 Common sources of errors
- •20.5.2 Debugging tools
- •20.5.3 Session options that support debugging
- •20.6 Going further
- •20.7 Summary
- •21 Creating a package
- •21.1 Nonparametric analysis and the npar package
- •21.1.1 Comparing groups with the npar package
- •21.2 Developing the package
- •21.2.1 Computing the statistics
- •21.2.2 Printing the results
- •21.2.3 Summarizing the results
- •21.2.4 Plotting the results
- •21.2.5 Adding sample data to the package
- •21.3 Creating the package documentation
- •21.4 Building the package
- •21.5 Going further
- •21.6 Summary
- •22 Creating dynamic reports
- •22.1 A template approach to reports
- •22.2 Creating dynamic reports with R and Markdown
- •22.3 Creating dynamic reports with R and LaTeX
- •22.4 Creating dynamic reports with R and Open Document
- •22.5 Creating dynamic reports with R and Microsoft Word
- •22.6 Summary
- •afterword Into the rabbit hole
- •appendix A Graphical user interfaces
- •appendix B Customizing the startup environment
- •appendix C Exporting data from R
- •Delimited text file
- •Excel spreadsheet
- •Statistical applications
- •appendix D Matrix algebra in R
- •appendix E Packages used in this book
- •appendix F Working with large datasets
- •F.1 Efficient programming
- •F.2 Storing data outside of RAM
- •F.3 Analytic packages for out-of-memory data
- •F.4 Comprehensive solutions for working with enormous datasets
- •appendix G Updating an R installation
- •G.1 Automated installation (Windows only)
- •G.2 Manual installation (Windows and Mac OS X)
- •G.3 Updating an R installation (Linux)
- •references
- •index
- •Symbols
- •Numerics
- •23.1 The lattice package
- •23.2 Conditioning variables
- •23.3 Panel functions
- •23.4 Grouping variables
- •23.5 Graphic parameters
- •23.6 Customizing plot strips
- •23.7 Page arrangement
- •23.8 Going further
Creating dynamic reports with R and Microsoft Word |
527 |
By default, odfWeave renders data frames, matrices, and vectors in an attractive table format. The odfTable() function can be used to format tables with a higher degree of precision and control. The function produces XML code, so be sure to specify result=xml in code chunks employing this function. Unfortunately, the xtable() function doesn’t work with odfWeave.
Once you have your report in ODF format, you can continue to edit it, tighten up the formatting, and save the results to the ODT, HTML, DOC, or DOCX file format. To learn more, read the odfWeave manual and vignette. Additional information, including a tutorial on document formatting with odfWeave, can be found in the Examples folder installed in the odfWeave directory in your R library. (The function
.libPaths() will display your library location.)
22.5 Creating dynamic reports with R and Microsoft Word
For good or ill, Microsoft Word is the standard for report writing in corporate settings. You’ve already seen how to create a Word document from a Markdown file using rmarkdown (section 22.2). In this section, we’ll look at a method for creating dynamic reports by embedding R code directly into Word documents using the R2wd package. The methods in the section will only work on Windows platforms (sorry, Mac and Linux users).
To follow the examples in this section, you’ll need to install the R2wd package (install.packages("R2wd")). R2wd also requires the RDCOMClient package, available from the Omega Project for Statistical Computing.
At the time of this writing, RDCOMClient must be installed from source. First be sure Rtools is installed (http://cran.r-project.org/bin/windows/Rtools). Next, download the source file (RDCOMClient_0.93-0.tar.gz) from www.omegahat.org/RDCOMClient. Note that the version numbers are likely to change over time. Finally, install the package using
install.packages(RDCOMClient_0.93-0.tar.gz, repos = NULL, type = "source")
The R2wd package provides functions that allow you to create a blank Word document; insert sections and titles; insert text, tables, and graphs; add formatting; and save the results. Although the package is versatile, building and formatting Word documents programmatically can be time-consuming.
The easiest way to create a dynamic report in Word using the R2wd package is a two-step process:
1Create a Word document that contains bookmarks indicating where you want your R results to be placed.
2Create an R script that inserts the results in the Word document at the bookmarked locations and saves the finished document.
Let’s try this approach.
Open a new Word document, and call it salaryTemplate2.docx. Add the text and bookmarks displayed in figure 22.6. (In reality, the bookmarks in figure 22.6 aren’t
528 |
CHAPTER 22 Creating dynamic reports |
Sample Report
Introduction
A two-way analysis of variance was employed to investigate the relationship between gender, academic rank, and annual salary in dollars. Data were collected from n professors in 2008. The ANOVA table is given in Table 1.
aovTable
The interaction between gender and rank is plotted in Figure 1.
effectsPlot
Figure 22.6 A Microsoft Word document named salaryTemplate2.docx containing text and bookmarks. The file is processed by the script salary.R (listing 22.3), results are inserted at the bookmark locations, and the document is saved as salaryReport2.docx (figure 22.7). Note that the bookmarks (in bold shaded text) aren’t actually visible on the page; the image has been annotated so that you can see where to place them.
visible. I’ve annotated the image, adding the bookmark names with a bold colored background, so that you can see where each should go.)
To insert a bookmark, place the cursor where you want the bookmark, choose Insert > Bookmark, give the bookmark a name, and click Add. The bookmarks in this example are named n, aovTable, and effectsPlot.
TIP Selecting Options > Advanced > Show Bookmarks in Microsoft Word will help you see where bookmarks are placed.
Next, create the R script given in listing 22.3. When the script is executed, it produces the necessary analyses, inserts them into the Word document, and saves the final document to disk. The script uses the functions listed in table 22.4.
Table 22.4 R2wd functions
Function |
Use |
|
|
wdGet() |
Returns a handle to a Word document. If Word isn’t running, it’s started, a |
|
blank document is opened, and a handle is returned. |
wdGoToBookmark() |
Places the cursor at a bookmark. |
wdWrite() |
Writes text at the cursor. |
wdTable() |
Writes a data frame or an array as a Word table at the current cursor location. |
wdPlot() |
Creates an R plot, and pastes it into Word at the current cursor location. |
wdSave() |
Saves the Word document. If no file name is given, Word prompts the user for |
|
one. |
wdQuit() |
Closes Word, and removes the handle. |
|
|
Creating dynamic reports with R and Microsoft Word |
529 |
Listing 22.3 salary.R: R script for inserting results in salary.docx
require(R2wd)
require(car)
df <- Salaries n <- nrow(df)
fit <- lm(salary ~ rank*sex, data=df) aovTable <- Anova(fit, type=3)
aovTable <- round(as.data.frame(aovTable), 3) aovTable[is.na(aovTable)] <- ""
wdGet("salaryTemplate2.docx", method="RDCOMClient") wdGoToBookmark("n")
wdWrite(n)
b Opens the document
c Inserts text
wdGoToBookmark("aovTable")
wdTable(aovTable, caption="Two-way Analysis of Variance", caption.pos="above", pointsize=12, autoformat=4)
wdGoToBookmark("effectsPlot") myplot <- function(){
require(effects)
par(mar=c(2,2,2,2)) plot(allEffects(fit), main="")
}
wdPlot(plotfun=myplot, caption="Mean Effects Plot",
height=4, width=5, method="metafile")
wdSave("SalaryReport2.docx") f Saves and quits wdQuit()
d Inserts a table
e Inserts a plot
First, the salary2Template.docx file is opened. If Word isn’t running, it’s automatically launched b. Next, the data analyses are performed. The cursor then is moved to the bookmark named n, and the sample size is inserted c.
Next, the cursor is moved to the bookmark named aovTable and the ANOVA results are inserted as a Word table d. Because R2wd doesn’t support the xtable() function, it’s important that the table be an R data frame, a matrix, or an array. Options can control the text and location of the table caption, font size, and table formatting. Try autoformat = 1, 2, 3, and so on, to see the various formats available. Currently there is no way to suppress the caption.
Two of the statements in the ANOVA code require additional explanation. The aovTable object is a data frame containing the two-way ANOVA results. The round() function is used to limit the number of decimal places printed in the table. The statement
aovTable[is.na(aovTable)] <- ""
is a trick that replaces NAs with blanks. This is necessary because there are no values in the F and Pr(>F) columns for the Residuals row, and you don’t want NA to print in these cells of the table.
530 |
CHAPTER 22 Creating dynamic reports |
The cursor next moves to the bookmark named effectsPlot. The wdPlot() function requires that the user specify a plotting function. Here, the myplot() function produces an effects plot via the allEffects() function in the effects package e.
The wdPlot() function supports method="bitmap" and method="metafile". Use metafile whenever possible—it looks better in a Word document. Unfortunately, the metafile option doesn’t support transparency, so you have to use the bitmap option when transparency is present. You’re most likely to encounter transparency when using the ggplot2 package to produce graphs.
When the code in salary.R is executed, it runs the R code, inserts the results into salaryTemplate2.docx, and saves the finished Word document as salaryReport2.docx f. Then the Microsoft Word application quits. The resulting document is displayed in figure 22.7.
Sample Report
Introduction
A two-way analysis of variance was employed to investigate the relationship between gender, academic rank, and annual salary in dollars. Data were collected from 397 professors in 2008. The ANOVA table is given in Table 1.
Table 1 Two-way Analysis of Variance
|
Sum Sq |
Df |
F value |
Pr(>F) |
(Intercept) |
67009671400 |
1 |
119.538 |
0 |
rank |
15266607695 |
2 |
13.617 |
0 |
sex |
97803720 |
1 |
0.174 |
0.676 |
rank:sex |
43603063 |
2 |
0.039 |
0.962 |
Residuals |
219184457146 |
391 |
|
|
|
|
|
|
|
The interaction between gender and rank is plotted in Figure 1.
|
|
|
AsstProf |
AssocProf |
Prof |
|
|
sex : Female |
|
sex : Male |
|
|
130000 |
|
|
|
|
|
120000 |
|
|
|
|
|
110000 |
|
|
|
|
salary |
100000 |
|
|
|
|
90000 |
|
|
|
|
|
|
|
|
|
|
|
|
80000 |
|
|
|
|
|
70000 |
|
|
|
|
|
AsstProf |
AssocProf |
Prof |
|
|
|
|
|
rank |
|
|
Figure 1 Mean Effects Plot
Figure 22.7 Final report in DOCX format (salaryReport2.docx)