Minimal reproducible example
This is a follow-up on this StackOverflow question/answer.
dplyr::arrange seems to interfere with auto-indexing, leading to unexpected wrong results.
A work-around is to disable auto-indexing with options(datatable.auto.index = FALSE).
As this issue is quite tricky, a warning, a mention to it in FAQ or a protection against this happening would be useful!
library(dplyr);
library(data.table)
DT <-
fread(
"iso3c country income
MOZ Mozambique LIC
ZMB Zambia LMIC
ALB Albania UMIC
MOZ Mozambique LIC
ZMB Zambia LMIC
ALB Albania UMIC
"
)
codes <- c("ALB", "ZMB")
options(datatable.auto.index = TRUE) # Default
DT <- distinct(DT) %>% as.data.table()
# Index creation because %in% is used for the first time
DT[iso3c %in% codes,verbose=T]
# Index mixed up by arrange
DT <- DT %>% arrange(iso3c) %>% as.data.table()
# this is wack because data.table uses the old index where row were rearranged:
DT[iso3c %in% codes,verbose=T]
#> iso3c country income
#> 1: ALB Albania UMIC
# this works because (...) prevents the parser to use auto-index
DT[(iso3c %in% codes)]
#> iso3c country income
#> 1: ALB Albania UMIC
#> 2: ZMB Zambia LMIC
Output of sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18362)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.14.0 dplyr_1.0.6
Reactions are currently unavailable
Minimal reproducible example
This is a follow-up on this StackOverflow question/answer.
dplyr::arrange seems to interfere with auto-indexing, leading to unexpected wrong results.
A work-around is to disable auto-indexing with options(datatable.auto.index = FALSE).
As this issue is quite tricky, a warning, a mention to it in FAQ or a protection against this happening would be useful!
library(dplyr); library(data.table) DT <- fread( "iso3c country income MOZ Mozambique LIC ZMB Zambia LMIC ALB Albania UMIC MOZ Mozambique LIC ZMB Zambia LMIC ALB Albania UMIC " ) codes <- c("ALB", "ZMB") options(datatable.auto.index = TRUE) # Default DT <- distinct(DT) %>% as.data.table() # Index creation because %in% is used for the first time DT[iso3c %in% codes,verbose=T] # Index mixed up by arrange DT <- DT %>% arrange(iso3c) %>% as.data.table() # this is wack because data.table uses the old index where row were rearranged: DT[iso3c %in% codes,verbose=T] #> iso3c country income #> 1: ALB Albania UMIC # this works because (...) prevents the parser to use auto-index DT[(iso3c %in% codes)] #> iso3c country income #> 1: ALB Albania UMIC #> 2: ZMB Zambia LMICOutput of sessionInfo()