Уменьшите время обработки применения функции к вектору и группировки по столбцам

Медленная часть кажется сгруппированной 1 - x. Попробуйте удалить его из сгруппированных вычислений:

t <- Sys.time()
data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_2]
data[, (vec_3) := lapply(.SD, prod), by = c("group_1", "group_2"), .SDcols = vec_3]
data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_3]
Sys.time() - t

data.table имеет оптимизированную версию prod. Авторы data.table называют это оптимизацией «GForce». Смотрите раздел «Что это за волшебство» здесь.

Проблема здесь в том, что это не сработает, если несколько операций выполняются вместе и внутри prod.

Нам нужно построить его шаг за шагом:

# First, what is inside the prod. No need for grouping
data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_2]

# Now, the prod alone, by groups. Now we take vec_3 in .SDcols

data[, (vec_3) := lapply(.SD, prod), 
     by = c("group_1", "group_2"), .SDcols = vec_3]

# Now, the outer operation, no need for grouping
data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_3]

Мои результаты сравнения обоих подходов:

> data.table::setkey(data, group_1, group_2)
> # lapply step 2
> t <- Sys.time()
> data[, (vec_3) := lapply(.SD, function(x) 1 - prod((1 - x))), 
+      by=c("group_1", "group_2"), .SDcols=vec_2]
> t <- Sys.time()
> data[, (vec_3) := lapply(.SD, function(x) 1 - prod((1 - x))), 
+      by=c("group_1", "group_2"), .SDcols=vec_2]
> Sys.time() - t
Time difference of 1.847846 mins

> # Optmized
> t <- Sys.time()
> # First, what is inside the prod. No need for grouping
> data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_2]
> 
> # Now, the prod alone, by groups. Now we take vec_3 in .SDcols
> 
> data[, (vec_3) := lapply(.SD, prod), 
+      by = c("group_1", "group_2"), .SDcols = vec_3]
> 
> # Now, the outer operation, no need for grouping
> data[, (vec_3) := lapply(.SD, function(x) 1 - x), .SDcols = vec_3]
> Sys.time() - t
Time difference of 4.312201 secs

Итак, 1,9 минуты против 4,31 секунды.

Посмотрите, когда GForce оптимизируется или нет, с помощью: (есть некоторые накладные расходы)

options(datatable.verbose = TRUE)

Finding groups using uniqlist on key ... 0.540s elapsed (0.390s cpu) 
Finding group sizes from the positions (can be avoided to save RAM) ... 0.110s elapsed (0.080s cpu) 
lapply optimization changed j from 'lapply(.SD, prod)' to 'list(prod(PD_3_N0), prod(PD_3_N1), prod(PD_3_N2), prod(PD_3_N3), prod(PD_3_N4))'
GForce optimized j to 'list(gprod(PD_3_N0), gprod(PD_3_N1), gprod(PD_3_N2), gprod(PD_3_N3), gprod(PD_3_N4))' (see ?GForce)
Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.028
gforce assign high and low took 0.025
gforce eval took 1.122
1.310s elapsed (0.970s cpu) 
Assigning to 30000000 row subset of 30000000 rows
RHS_list_of_columns == true

В этом примере знак + деактивирует GForce, поэтому большая часть оптимизации сохраняется.

> data[, (vec_3) := lapply(.SD, prod), .SDcols = vec_3, by = c("group_1", "group_2")]
Finding groups using uniqlist on key ... 0.510s elapsed (0.420s cpu) 
Finding group sizes from the positions (can be avoided to save RAM) ... 0.110s elapsed (0.090s cpu) 
lapply optimization changed j from 'lapply(.SD, prod)' to 'list(prod(PD_3_N0), prod(PD_3_N1), prod(PD_3_N2), prod(PD_3_N3), prod(PD_3_N4))'
GForce optimized j to 'list(gprod(PD_3_N0), gprod(PD_3_N1), gprod(PD_3_N2), gprod(PD_3_N3), gprod(PD_3_N4))' (see ?GForce)
Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.028
gforce assign high and low took 0.030
gforce eval took 1.212
1.390s elapsed (0.750s cpu) 
Assigning to 30000000 row subset of 30000000 rows
RHS_list_of_columns == true
> Sys.time() - t
Time difference of 3.031504 secs
> t <- Sys.time()
> data[, (vec_3) := lapply(.SD, function(x) +prod(x)), .SDcols = vec_3, by = c("group_1", "group_2")]
Finding groups using uniqlist on key ... 0.530s elapsed (0.480s cpu) 
Finding group sizes from the positions (can be avoided to save RAM) ... 0.110s elapsed (0.090s cpu) 
lapply optimization changed j from 'lapply(.SD, function(x) +prod(x))' to 'list(..FUN1(PD_3_N0), ..FUN1(PD_3_N1), ..FUN1(PD_3_N2), ..FUN1(PD_3_N3), ..FUN1(PD_3_N4))'
GForce is on, but not activated for this query; left j unchanged (see ?GForce)
Old mean optimization is on, left j unchanged.
Making each group and running j (GForce FALSE) ... 
  memcpy contiguous groups took 7.584s for 18964290 groups
  eval(j) took 63.648s for 18964290 calls
00:01:30 elapsed (00:01:22 cpu) 
> Sys.time() - t
Time difference of 1.514286 mins