Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[sanjaisahu]

Hi folks,

I need to merge different large tables (up to 10Gb each) into a single one. To do so I am using a computer cluster with 50+ cores and 10+Gb Ram that runs on Linux.

I always end up with an error message like: "Cannot allocate vector of size X Mb". Given that commands like memory.limit(size=X) are Windows-specific and not accepted, I cannot find a way around to merge my large tables.

Any suggestion welcome!

This is the code I use:

library(parallel)

no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)

temp = list.files(pattern="*.txt$")
gc()
Here the error occurs:

myfiles = parLapply(cl,temp, function(x) read.csv(x,
                                        header=TRUE,
                                        sep=";",
                                        stringsAsFactors=F,
                                        encoding = "UTF-8",
                                        na.strings = c("NA","99","")))


myfiles.final = do.call(rbind, myfiles)



Thank you

[bigpaws]

You have not mentioned the database you are using. The type of table and is the
merge from the same OS and the same database that the original tables are on.

Bigpaws

[Edney]

Can we expect problems if any of those things don't match while trying to merge tables, Bigpaws?