29 Functionals

29.1 My first functional: lapply()

  1. Q: Why are the following two invocations of lapply() equivalent?

    A: In the first statement each element of trims is explicitly supplied to mean()’s second argument. In the latter statement this happens via positional matching, since mean()’s first argument is supplied via name in lapply()’s third argument (...).

  2. Q: The function below scales a vector so it falls in the range [0, 1]. How would you apply it to every column of a data frame? How would you apply it to every numeric column in a data frame?

    A: Since this function needs numeric input, one can check this via an if clause. If one also wants to return non-numeric input columns, these can be supplied to the else argument of the if() “function”:

  3. Q: Use both for loops and lapply() to fit linear models to the mtcars using the formulas stored in this list:

    A: Like in the first exercise, we can create two lapply() versions:

    Note that all versions return the same content, but they won’t be identical, since the values of the “call” element will differ between each version.

  4. Q: Fit the model mpg ~ disp to each of the bootstrap replicates of mtcars in the list below by using a for loop and lapply(). Can you do it without an anonymous function?

    A:

  5. Q: For each model in the previous two exercises, extract \(R^2\) using the function below.

    A: For the models in exercise 3:

    And the models in exercise 4:

29.2 For loops functionals: friends of lapply():

  1. Q: Use vapply() to:

    1. Compute the standard deviation of every column in a numeric data frame.

    2. Compute the standard deviation of every numeric column in a mixed data frame. (Hint: you’ll need to use vapply() twice.)

    A: As a numeric data.frame we choose cars:

    And as a mixed data.frame we choose iris:

  2. Q: Why is using sapply() to get the class() of each element in a data frame dangerous?

    A: Columns of data.frames might have more than one class, so the class of sapply()’s output may differ from time to time (silently). If …

    • all columns have one class: sapply() returns a character vector
    • one column has more classes than the others: sapply() returns a list
    • all columns have the same number of classes, which is more than one: sapply() returns a matrix

    For example:

    Note that this case often appears, wile working with the POSIXt types, POSIXct and POSIXlt.

  3. Q: The following code simulates the performance of a t-test for non-normal data. Use sapply() and an anonymous function to extract the p-value from every trial.

    Extra challenge: get rid of the anonymous function by using [[ directly.

    A:

  4. Q: What does replicate() do? What sort of for loop does it eliminate? Why do its arguments differ from lapply() and friends?

    A: As stated in ?replicate:

    replicate is a wrapper for the common use of sapply for repeated evaluation of an expression (which will usually involve random number generation).

    We can see this clearly in the source code:

    #> function (n, expr, simplify = "array") 
    #> sapply(integer(n), eval.parent(substitute(function(...) expr)), 
    #>     simplify = simplify)
    #> <bytecode: 0x687da20>
    #> <environment: namespace:base>

    Like sapply() replicate() eliminates a for loop. As explained for Map() in the textbook, also every replicate() could have been written via lapply(). But using replicate() is more concise, and more clearly indicates what you’re trying to do.

  5. Q: Implement a version of lapply() that supplies FUN with both the name and the value of each component.

    A:

  6. Q: Implement a combination of Map() and vapply() to create an lapply() variant that iterates in parallel over all of its inputs and stores its outputs in a vector (or a matrix). What arguments should the function take?

    A As we understand this exercise, it is about working with a list of lists, like in the following example:

    So we can get the same result with a more specialized function:

  7. Q: Implement mcsapply(), a multicore version of sapply(). Can you implement mcvapply(), a parallel version of vapply()? Why or why not?

29.3 Manipulating matrices and data frames

  1. Q: How does apply() arrange the output? Read the documentation and perform some experiments.

    A:

    apply() arranges its output columns (or list elements) according to the order of the margin. The rows are ordered by the other dimensions, starting with the “last” dimension of the input object. What this means should become clear by looking at the three and four dimensional cases of the following example:

  2. Q: There’s no equivalent to split() + vapply(). Should there be? When would it be useful? Implement one yourself.

    A: We can modify the tapply2() approach from the book, where split() and sapply() were combined:

    tapply() has a SIMPLIFY argument. When you set it to FALSE, tapply() will always return a list. It is easy to create cases where the length and the types/classes of the list elements vary depending on the input. The vapply() version could be useful, if you want to control the structure of the output to get an error according to some logic of a specific usecase or you want typestable output to build up other functions on top of it.

  3. Q: Implement a pure R version of split(). (Hint: use unique() and subsetting.) Can you do it without a for loop?

    A:

  4. Q: What other types of input and output are missing? Brainstorm before you look up some answers in the plyr paper.

    A: From the suggested plyr paper, we can extract a lot of possible combinations and list them up on a table. Sean C. Anderson already has done this based on a presentation from Hadley Wickham and provided the following result here.

    object type array data frame list nothing
    array apply . . .
    data frame . aggregate by .
    list sapply . lapply .
    n replicates replicate . replicate .
    function arguments mapply . mapply .

    Note the column nothing, which is specifically for usecases, where sideeffects like plotting or writing data are intended.

29.4 Manipulating lists

  1. Q: Why isn’t is.na() a predicate function? What base R function is closest to being a predicate version of is.na()?

    A: Because a predicate function always returns TRUE or FALSE. is.na(NULL) returns logical(0), which excludes it from being a predicate function. The closest in base that we are aware of is anyNA(), if one applies it elementwise.

  2. Q: Use Filter() and vapply() to create a function that applies a summary statistic to every numeric column in a data frame.

    A:

  3. Q: What’s the relationship between which() and Position()? What’s the relationship between where() and Filter()?

    A: which() returns all indices of true entries from a logical vector. Position() returns just the first (default) or the last integer index of all true entries that occur by applying a predicate function on a vector. So the default relation is Position(f, x) <=> min(which(f(x))).

    where(), defined in the book as:

    is useful to return a logical vector from a condition asked on elements of a list or a data frame. Filter(f, x) returns all elements of a list or a data frame, where the supplied predicate function returns TRUE. So the relation is Filter(f, x) <=> x[where(f, x)].

  4. Q: Implement Any(), a function that takes a list and a predicate function, and returns TRUE if the predicate function returns TRUE for any of the inputs. Implement All() similarly.

    A: Any():

    All():

  5. Q: Implement the span() function from Haskell: given a list x and a predicate function f, span returns the location of the longest sequential run of elements where the predicate is true. (Hint: you might find rle() helpful.)

    A: Our span_r() function returns the first index of the longest sequential run of elements where the predicate is true. In case of more than one longest sequenital, more than one first_index is returned.

29.5 Mathematical functionals

  1. Q: Implement arg_max(). It should take a function and a vector of inputs, and return the elements of the input where the function returns the highest value. For example, arg_max(-10:5, function(x) x ^ 2) should return -10. arg_max(-5:5, function(x) x ^ 2) should return c(-5, 5). Also implement the matching arg_min() function.

    A: arg_max():

    arg_min():

  2. Q: Challenge: read about the fixed point algorithm. Complete the exercises using R.

29.6 A family of functions

  1. Q: Implement smaller and larger functions that, given two inputs, return either the smaller or the larger value. Implement na.rm = TRUE: what should the identity be? (Hint: smaller(x, smaller(NA, NA, na.rm = TRUE), na.rm = TRUE) must be x, so smaller(NA, NA, na.rm = TRUE) must be bigger than any other value of x.) Use smaller and larger to implement equivalents of min(), max(), pmin(), pmax(), and new functions row_min() and row_max().

    A: We can do almost everything as shown in the case study in the textbook. First we define the functions smaller_() and larger_(). We use the underscore suffix, to built up non suffixed versions on top, which will include the na.rm parameter. In contrast to the add() example from the book, we change two things at this step. We won’t include errorchecking, since this is done later at the top level and we return NA_integer_ if any of the arguments is NA (this is important, if na.rm is set to FALSE and wasn’t needed by the add() example, since + already returns NA in this case.)

    We can take na.rm() from the book:

    To find the identity value, we can apply the same argument as in the textbook, hence our functions are also associative and the following equation should hold:

    3 = smaller(smaller(3, NA), NA) = smaller(3, smaller(NA, NA)) = 3

    So the identidy has to be greater than 3. When we generalize from 3 to any real number this means that the identity has to be greater than any number, which leads us to infinity. Hence identity has to be Inf for smaller() (and -Inf for larger()), which we implement next:

    Like min() and max() can act on vectors, we can implement this easyly for our new functions. As shown in the book, we also have to set the init parameter to the identity value.

    We can also create vectorised versions as shown in the book. We will just show the smaller() case to become not too verbose.

    Of course, we are also able to copy paste the rest from the textbook, to solve the last part of the exercise:

  2. Q: Create a table that has and, or, add, multiply, smaller, and larger in the columns and binary operator, reducing variant, vectorised variant, and array variants in the rows.

    1. Fill in the cells with the names of base R functions that perform each of the roles.

    2. Compare the names and arguments of the existing R functions. How consistent are they? How could you improve them?

    3. Complete the matrix by implementing any missing functions.

    A In the following table we can see the requested base R functions, that we are aware of:

    and or add multiply smaller larger
    binary && ||
    reducing all any sum prod min max
    vectorised & | + * pmin pmax
    array

    Notice that we were relatively strict about the binary row. Since the vectorised and reducing versions are more general, then the binary versions, we could have used them twice. However, this doesn’t seem to be the intention of this exercise.

    The last part of this exercise can be solved via copy pasting from the book and the last exercise for the binary row and creating combinations of apply() and the reducing versions for the array row. We think the array functions just need a dimension and an rm.na argument. We don’t know how we would name them, but sth. like sum_array(1, na.rm = TRUE) could be ok.

    The second part of the exercise is hard to solve complete. But in our opinion, there are two important parts. The behaviour for special inputs like NA, NaN, NULL and zero length atomics should be consistent and all versions should have a rm.na argument, for which the functions also behave consistent. In the follwing table, we return the output of `f`(x, 1), where f is the function in the first column and x is the special input in the header (the named functions also have an rm.na argument, which is FALSE by default). The order of the arguments is important, because of lazy evaluation.

    NA NaN NULL logical(0) integer(0)
    && NA NA error NA NA
    all NA NA TRUE TRUE TRUE
    & NA NA error logical(0) logical(0)
    || TRUE TRUE error TRUE TRUE
    any TRUE TRUE TRUE TRUE TRUE
    | TRUE TRUE error logical(0) logical(0)
    sum NA NaN 1 1 1
    + NA NaN numeric(0) numeric(0) numeric(0)
    prod NA NaN 1 1 1
    * NA NaN numeric(0) numeric(0) numeric(0)
    min NA NaN 1 1 1
    pmin NA NaN numeric(0) numeric(0) numeric(0)
    max NA NaN 1 1 1
    pmax NA NaN numeric(0) numeric(0) numeric(0)

    We can see, that the vectorised and reduced numerical functions are all consistent. However it is not, that the first three logical functions return NA for NA and NaN, while the 4th till 6th function all return TRUE. Then FALSE would be more consistent for the first three or the return of NA for all and an extra na.rm argument. In seems relatively hard to find an easy rule for all cases and especially the different behaviour for NULL is relatively confusing. Another good opportunity for sorting the functions would be to differentiate between “numerical” and “logical” operators first and then between binary, reduced and vectorised, like below (we left the last colum, which is redundant, because of coercion, as intended):

    `f(x,1)` NA NaN NULL logical(0)
    && NA NA error NA
    || TRUE TRUE error TRUE
    all NA NA TRUE TRUE
    any TRUE TRUE TRUE TRUE
    & NA NA error logical(0)
    | TRUE TRUE error logical(0)
    sum NA NaN 1 1
    prod NA NaN 1 1
    min NA NaN 1 1
    max NA NaN 1 1
    + NA NaN numeric(0) numeric(0)
    * NA NaN numeric(0) numeric(0)
    pmin NA NaN numeric(0) numeric(0)
    pmax NA NaN numeric(0) numeric(0)

    The other point are the naming conventions. We think they are clear, but it could be useful to provide the missing binary operators and name them for example ++, **, <>, >< to be consistent.

  3. Q: How does paste() fit into this structure? What is the scalar binary function that underlies paste()? What are the sep and collapse arguments to paste() equivalent to? Are there any paste variants that don’t have existing R implementations?

    A paste() behaves like a mix. If you supply only length one arguments, it will behave like a reducing function, i.e. :

    If you supply at least one element with length greater then one, it behaves like a vectorised function, i.e. :

    We think it should be possible to implement a new paste() starting from

    The sep argument is equivalent to bind sep on every ... input supplied to paste(), but the last and then bind these results together. In relations:

    paste(n1, n2, ...,nm , sep = sep) <=>
    paste0(paste0(n1, sep), paste(n2, n3, ..., nm, sep = sep)) <=>
    paste0(paste0(n1, sep), paste0(n2, sep), ..., paste0(nn, sep), paste0(nm))

    We can check this for scalar and non scalar input

    collapse just binds the outputs for non scalar input together with the collapse input. In relations:

    for input A1, ..., An, where Ai = a1i:ami,
    
    paste(A1 , A2 , ...,  An, collapse = collapse) 
    <=>
    paste0(
          paste0(paste(  a11,   a12, ...,   a1n), collapse),
          paste0(paste(  a21,   a22, ...,   a2n), collapse),
          .................................................
          paste0(paste(am-11, am-12, ..., am-1n), collapse),      
                 paste(  am1,   am2, ...,   amn)
          )

    One can see this easily by intuition from examples:

    We think the only paste version that is not implemented in base R is an array version. At least we are not aware of sth. like row_paste or paste_apply etc.