# Example multiverse implementation: Re-evaluating the efficiency of physical visualizations

#### Pierre Dragicevic, Inria

#### Yvonne Jansen, CNRS & Sorbonne Universite

#### Abhraneel Sarma, Northwestern University

#### 2023-11-25

Source:`vignettes/frequentist-multiverse-analysis.Rmd`

`frequentist-multiverse-analysis.Rmd`

```
library(dplyr)
library(tidyr)
library(ggplot2)
library(purrr)
library(broom)
library(gganimate)
library(multiverse)
```

## Multiverse case study #2

In this vignette, we will recreate the multiverse analysis, Re-evaluating the efficiency of Physical Visualisations, performed by Dragicevic et al. in Increasing the transparency of research papers with explorable multiverse analyses using the package.

## Introduction

The original study investigated the effects of moving 3D data visualizations to the physical world and found that it can improve users’ efficiency at information retrieval tasks. The original study consisted of two experiments. Dragicevic et al. only re-analyze the second experiment, whose goal was to better understand why physical visualizations appear to be superior.

## The data

The experiment involved an “enhanced” version of the on-screen 3D chart and an “impoverished” version of the physical 3D chart. The enhanced on-screen chart was rotated using a 3D-tracked tangible prop instead of a mouse. The impoverished physical chart consisted of the same physical object but participants were instructed not to use their fingers for marking. There were 4 conditions:

- physical touch: physical 3D bar charts where touch was explicitly encouraged in the instructions.
- physical no touch: same charts as above except subjects were told not to use their fingers to mark points of interest (labels and bars).
- virtual prop: on-screen 3D bar charts with a tangible prop for controlling 3D rotation.
- virtual mouse: same charts as above, but 3D rotation was mouse-controlled.

These manipulations were meant to answer three questions:

- how important is direct touch in the physical condition?
- how important is rotation by direct manipulation?
- how important is visual realism? Visual realism referred to the higher perceptual richness of physical objects compared to on-screen objects, especially concerning depth cues.

We load the data for this study which is contained in
`data(userlogs)`

in the `multiverse`

package.

## Analysis #1: Mean and Confidence Intervals for each condition

In this vignette, we are primarily concerned with the variables:
`duration`

and `modality`

, as the focus of this
analysis is on task completion times.

The first (default) analysis is a one-sided t-test to estimate the means and 95% confidence intervals of the log-transformed task completion time (duration). Since, task completion times are strictly positive, and may have a long tail, this decision makes sense. However, it may be reasonable to use the untransformed data as well. On the other hand, it is also reasonable to use a bootstrap test instead of a t-test.

This results in four possible analysis combinations, two each for data transformation (log and untransformed), and model (t-test and BCa bootstrap).

### Average task completion time (arithmetic mean) for each condition.

We need a few helper functions to so that they take the same arguments and return the same output.These functions will help us calculate the mean point estimate and the upper and lower bounds of the 95% confidence interval using the bootstrap method and t-test method.

```
bootstrappedCI <- function(observations, conf.level, seed = 0) {
samplemean <- function(x, d) {return(mean(x[d]))}
pointEstimate <- samplemean(observations)
if (!(is.na(seed) | is.null(seed))){
set.seed(seed) # make deterministic
}
bootstrap_samples <- boot::boot(data = observations, statistic = samplemean, R = 5000)
bootci <- boot::boot.ci(bootstrap_samples, type = "bca", conf = conf.level)
c(pointEstimate, bootci$bca[4], bootci$bca[5])
}
tCI <- function(observations, conf.level) {
pointEstimate <- mean(observations)
sampleSD <- sd(observations)
sampleN <- length(observations)
sampleError <- qt(1-(1-conf.level)/2, df = sampleN-1) * sampleSD/sqrt(sampleN)
c(pointEstimate, pointEstimate - sampleError, pointEstimate + sampleError)
}
```

Next we initialise the multiverse object within which this analysis will take place.

`M = multiverse()`

Now we define the parameters we want to consider in the multiverse:
confidence interval method (`ci_method`

) and data
transformation method (`data_transform`

). We also define a
parameter for confidence level, as the choice of a 95% confidence level
is arbitrary and we can choose to instead present our results with
alternate confidence levels. Here we vary between: 50%, 89%, 95%, and
99.9%. Thus our multiverse consists of: 2 \(\times\) 2 \(\times\) 7 different analysis
combinations.

#### Note

In this vignette, we make use of *multiverse
code chunks*, a custom engine designed to work with the
multiverse package, to implement the multiverse analyses. Please refer
to the vignette (`vignette("multiverse-in-rmd")`

) for more
details. Users could instead make use of the function which is more
suited for a script-style implementation. Please refer to the vignettes
(`vignette("complete-multiverse-analysis")`

and
`vignette("basic-multiverse")`

) for more details.

```
```{multiverse default-m-1, inside = M}
ci_method <- branch(ci_method,
"t based" ~ "tCI",
"bootstrap" ~ "bootstrappedCI"
)
data_transform <- branch(data_transform,
"log-transformed" ~ log,
"untransformed" ~ identity
)
conf_level <- branch(conf_level,
"50%" ~ 0.5,
"89" ~ 0.89,
"95%" ~ 0.95,
"99%" ~ 0.99
)
```
```

We now look at the multiverse table and see that it has created all the possible combinations:

`expand(M)`

We then actually perform the steps within the multiverse to get results from the different possible combinations of analysis options. First, we perform the data transformation operation within the multiverse. This will result in the data being appropriately transformed (log or identity) in the corresponding multiverse.

```
```{multiverse default-m-2, inside = M}
duration <- do.call(data_transform, list(data.userlogs.raw$duration))
```
```

Next, we calculate the mean point estimates and 95% confidence
intervals for each condition in the experiment. We also need to format
the data so that the results could be neatly stored in a
`data.frame`

. We strongly recommend sorting the results that
you would wish to extract from the multiverse in a
`data.frame`

as that would make it much easier for analysing
and visualising the results.

object ‘duration’ not foundeval(expr, envir, enclos)object ‘ci.physical_notouch’ not foundeval(expr, envir, enclos)object ‘duration’ not foundeval(expr, envir, enclos)object ‘ci.physical_touch’ not foundeval(expr, envir, enclos)object ‘duration’ not foundeval(expr, envir, enclos)object ‘ci.virtual_prop’ not foundeval(expr, envir, enclos)object ‘duration’ not foundeval(expr, envir, enclos)object ‘ci.virtual_mouse’ not foundeval(expr, envir, enclos)object ‘ci.physical_notouch’ not foundeval(expr, envir, enclos)

```
```{multiverse default-m-3, inside = M}
modality <- data.userlogs.raw$modalityname
ci.physical_notouch <- do.call(ci_method, list(duration[modality == 'physical-notouch'], conf_level))
ci.physical_notouch <- setNames(as.list(c("physical_notouch", ci.physical_notouch)), c("modality", "estimate", "conf.low", "conf.high"))
ci.physical_touch <- do.call(ci_method, list(duration[modality == 'physical-touch'], conf_level))
ci.physical_touch <- setNames(as.list(c("physical_touch", ci.physical_touch)), c("modality", "estimate", "conf.low", "conf.high"))
ci.virtual_prop <- do.call(ci_method, list(duration[modality == 'virtual-prop'], conf_level))
ci.virtual_prop <- setNames(as.list(c("virtual_prop", ci.virtual_prop)), c("modality", "estimate", "conf.low", "conf.high"))
ci.virtual_mouse <- do.call(ci_method, list(duration[modality == 'virtual-mouse'], conf_level))
ci.virtual_mouse <- setNames(as.list(c("virtual_mouse", ci.virtual_mouse)), c("modality", "estimate", "conf.low", "conf.high"))
df <- rbind.data.frame(ci.physical_notouch, ci.physical_touch, ci.virtual_prop, ci.virtual_mouse, make.row.names = FALSE, stringsAsFactors = FALSE)
df <- transform(df, estimate = as.numeric(estimate), conf.low = as.numeric(conf.low), conf.high = as.numeric(conf.high))
```
```

Since the multiverse only executes the default analysis, we then run the following command to run all the analysis that we have defined in the multiverse.

## Results #1

### Extracting the results from the multiverse object

Next, we need to extract the results from the multiverse. The results
for each unique analysis combination (a universe in our multiverse), is
stored in an environment in the `.results`

column. We can
extract data frames from this column using the function. This creates a
new column in our data frame, `summary`

which itself consists
of data frames.

As we can see above, each row in the `summary`

column
consists of a \(4 \times 4\), data
frame which we will need to unpack. We will use the function to expand
the different columns of the data frame into their own columns. Finally,
we use the function to unnest the rows of the data frame into their own
row. Note that we have a `.universe`

column which indexes
each universe in our multiverse i.e. each unique analysis
combination.

Below we can see the result of this transformation. You can see that
we have created four new columns (`modality`

,
`estimate`

, `conf.low`

, `conf.high`

).
In addition, we have four rows for each universe corresponding to the
results for each of the four conditions in our experiment.

```
df.mtbl <- unnest_wider(df.mtbl, c(summary))
df.mtbl <- unnest(df.mtbl, cols = c(modality, estimate, conf.low, conf.high))
head(df.mtbl)
```

### Visualising the results

We will then sort the results, and transform the log transformed variables back on to the natural scale. We are then ready to visualise the result by animating over each universe.

```
df.mtbl <- arrange(df.mtbl, conf_level, desc(data_transform), desc(ci_method))
df.results <- df.mtbl
df.results$estimate[df.mtbl$data_transform == "log-transformed"] = exp(df.mtbl$estimate[df.mtbl$data_transform == "log-transformed"])
df.results$conf.low[df.mtbl$data_transform == "log-transformed"] = exp(df.mtbl$conf.low[df.mtbl$data_transform == "log-transformed"])
df.results$conf.high[df.mtbl$data_transform == "log-transformed"] = exp(df.mtbl$conf.high[df.mtbl$data_transform == "log-transformed"])
df.results %>% head()
```

```
p <- df.results %>%
ggplot() +
geom_vline( xintercept = 0, colour = '#979797' ) +
geom_point( aes(x = estimate, y = modality)) +
geom_errorbarh( aes(xmin = conf.low, xmax = conf.high, y = modality), height = 0) +
transition_manual( .universe )
animate(p, nframes = 28, fps = 2)
```

The figure above shows the (geometric) mean completion time for each condition. At first sight, physical touch appears to be consistently faster than the other conditions, across all possible analysis combinations specified in the multiverse. However, since condition is a within-subject factor, it is preferable to examine within-subject differences, which we show in the next section.

### Sidebar: Using the tidyverse to extract and visualize the results

So far we have tried to keep this analysis in base R. However, the
steps involved in extracting and visualising results from the multiverse
maybe more convenient for some using the tidyverse API. We can implement
the steps that we have taken in the previous three code block in the
following way using the tidyverse. The result of this is the same data
frame that was created in the previous step, and was used as input data
to ggplot, `df.results`

. The `multiverse`

package
is flexible, and does not restrict you towards a particular style of R
programming.

```
expand(M) %>%
mutate( index = seq(1:nrow(.)) ) %>%
mutate( summary = map(.results, "df") ) %>%
unnest_wider( c(summary)) %>%
unnest(cols = c(modality, estimate, conf.low, conf.high)) %>%
mutate(
estimate = ifelse(data_transform == "log-transformed", exp(estimate), estimate),
conf.low = ifelse(data_transform == "log-transformed", exp(conf.low), conf.low),
conf.high = ifelse(data_transform == "log-transformed", exp(conf.high), conf.high)
) %>%
arrange(conf_level, desc(data_transform), desc(ci_method))
```

## Analysis #2: Differences between mean completion times between conditions

Next, we compute the pairwise ratios between mean completion times to examine the within-subject differences.

object ‘diff.touch_notouch’ not foundeval(expr, envir, enclos)object ‘physical_touch - physical_notouch’ not foundeval(expr, envir, enclos)object ‘duration’ not foundeval(expr, envir, enclos)object ‘diff.notouch_prop’ not foundeval(expr, envir, enclos)object ‘physical_notouch - virtual_prop’ not foundeval(expr, envir, enclos)object ‘duration’ not foundeval(expr, envir, enclos)object ‘diff.propr_mouse’ not foundeval(expr, envir, enclos)object ‘virtual_prop - virtual_mouse’ not foundeval(expr, envir, enclos)object ‘physical_touch - physical_notouch’ not foundeval(expr, envir, enclos)

```
```{multiverse default-m-4, inside = M}
diff.touch_notouch <- duration[modality == 'physical-notouch'] - duration[modality == 'physical-touch']
`physical_touch - physical_notouch` <- do.call(ci_method, list(diff.touch_notouch, conf_level))
`physical_touch - physical_notouch` <- setNames(as.list(c("physical_touch - physical_notouch", `physical_touch - physical_notouch`)), c("modality", "estimate", "conf.low", "conf.high"))
diff.notouch_prop <- duration[modality == 'physical-notouch'] - duration[modality == 'virtual-prop']
`physical_notouch - virtual_prop` <- do.call(ci_method, list(diff.notouch_prop, conf_level))
`physical_notouch - virtual_prop` <- setNames(as.list(c("physical_notouch - virtual_prop", `physical_notouch - virtual_prop`)), c("modality", "estimate", "conf.low", "conf.high"))
diff.propr_mouse <- duration[modality == 'virtual-prop'] - duration[modality == 'virtual-mouse']
`virtual_prop - virtual_mouse` <- do.call(ci_method, list(diff.propr_mouse, conf_level))
`virtual_prop - virtual_mouse` <- setNames(as.list(c("virtual_prop - virtual_mouse", `virtual_prop - virtual_mouse`)), c("modality", "estimate", "conf.low", "conf.high"))
df.diffs <- rbind.data.frame(`physical_touch - physical_notouch`, `physical_notouch - virtual_prop`, `virtual_prop - virtual_mouse`, make.row.names = FALSE, stringsAsFactors = FALSE)
df.diffs <- transform(df.diffs, estimate = as.numeric(estimate), conf.low = as.numeric(conf.low), conf.high = as.numeric(conf.high))
```
```

We can output the data frame that we have created inside the multiverse code block, as we would for a data frame in R. This would output the result in the default universe () of the multiverse. We can see that this data frame, as intended, has computed the mean differences and 95% confidence intervals between the conditions we care about.

```
```{multiverse default-m-5, inside = M}
df.diffs
```
```

`df.diffs`

We then execute all the other analysis combinations (universes) in our multiverse.

## Results #2

A value lower than 1 (i.e., on the left side of the dark line) means the condition on the left is faster than the condition on the right. The confidence intervals are not corrected for multiplicity. The results from this study appear to be relatively robust and consistent across all the possible combinations that we have tried.

```
p <- df.results.diff %>%
ggplot() +
geom_vline( xintercept = 0, colour = '#979797' ) +
geom_point( aes(x = estimate, y = modality)) +
geom_errorbarh( aes(xmin = conf.low, xmax = conf.high, y = modality), height = 0) +
transition_manual( .universe ) +
theme_minimal()
animate(p, nframes = 28, fps = 4)
```

Correction for multiplicity can be another analysis option in the multiverse analysis. Since the individual confidence level is 95%, an interval that does not contain the value 1 indicates a statistically significant difference at the α = .05 level. The probability of getting at least one such interval if all 3 population means were zero (i.e., the family-wise error rate) is α=.14. Likewise, the simultaneous confidence level is 86%, meaning that if we replicate our experiment over and over, we should expect the 3 confidence intervals to capture all 3 population means 86% of the time.

## Conclusion

This example was adapted from Dragicevic et al.’s study Increasing the transparency of research papers with explorable multiverse analyses to show how previously performed multiverse analysis can be reproduced using the package in a flexible, and easily readable manner. It also shows how a multiverse analysis can be implemented in mostly base R syntax.