A group-filter-select is easy to perform with dplyr. In the example below, we have some data on companies for different quarters this year. I now want to filter to the first quarter of companies which don't have data for the fourth quarter (in this case, the second company), dropping the quarter-label.
df <- data.frame(companyId = c(rep(1, 4),
rep(2, 3),
rep(3, 4)),
Quarter = c(1:4, 1:3, 1:4),
Year = 2019)
q <- 4
df %>%
group_by(
companyId,
) %>%
filter(
Quarter == 1 &
!(q %in% Quarter)
) %>%
select(companyId,
Year)
> # A tibble: 1 x 3
> # Groups: companyId, Ticker [1]
> companyId Year
> <dbl> <dbl>
> 1 2 2019
However, doing the same with dtplyr returns an empty table:
dt <- lazy_dt(data.table(companyId = c(rep(1, 4),
rep(2, 3),
rep(3, 4)),
Quarter = c(1:4, 1:3, 1:4),
Year = 2019))
q <- 4
dt %>%
group_by(
companyId
) %>%
filter(
Quarter == 1 &
!(q %in% Quarter)
) %>%
select(companyId
Year)
> Source: local data table [?? x 3]
> Call: `_DT1`[Quarter == 1 & !(q %in% Quarter), .(companyId,
> Year)]
>
> # ... with 3 variables: companyId <dbl>, Year <dbl>
>
> # Use as.data.table()/as.data.frame()/as_tibble() to access results
What's odd is the displayed translation:
`_DT1`[Quarter == 1 & !(q %in% Quarter),
.(companyId, Year)]
which is incorrect. As described in the dtplyr's own docs, the correct call would need to use a filtered .SD:
`_DT1`[, .SD[Quarter == 1 & !(q %in% Quarter)],
by = .(companyId),
.SDcols = c("Year")]
(the by-columns are automatically included, so .SDcols should omit them to avoid duplication)
Interestingly, if we omit the select, the translation (and therefore output) is correct:
dt %>%
group_by(
companyId
) %>%
filter(
Quarter == 1 &
!(q %in% Quarter)
)
> Source: local data table [?? x 4]
> Call: `_DT2`[, .SD[Quarter == 1 & !(q %in% Quarter)],
> keyby = .(companyId)]
>
> companyId Quarter Year
> <dbl> <int> <dbl>
> 1 2 1 2019
Therefore, as a workaround, I can perform an as.data.table() prior to the select. This works, but throws an annoying warning:
dt %>%
group_by(
companyId
) %>%
filter(
calendarQuarter == 1 &
!(q %in% calendarQuarter)
) %>%
as.data.table() %>%
select(companyId,
calendarYear)
> companyId calendarYear
> 1: 2 2019
> Warning message:
> You are using a dplyr method on a raw data.table, which will call the data frame implementation,
> and is likely to be inefficient.
> *
> * To suppress this message, either generate a data.table translation with `lazy_dt()` or convert
> * to a data frame or tibble with `as.data.frame()`/`as_tibble()`.
I have a hard time thinking this is expected behavior, but would like to check here before throwing this on the dtplyr Github tracker.
This is currently a bug in dtplyr. I have posted it to the package's Github.