R asked the OS for one contiguous block it could not get
R needs a single contiguous allocation for a vector or matrix, so this can fail while plenty of RAM appears free. Copy-on-modify makes it worse: an innocuous-looking operation on a large object can transiently need twice its size.
Work on a subset first to confirm the logic, then reduce peak memory rather than just asking for more: drop objects you no longer need with rm() plus gc(), avoid growing objects in loops, and use sparse or on-disk representations (Matrix, DelayedArray, HDF5) for single-cell-scale data.