The JVM hit its heap ceiling — which is set on the JVM, not on the tool
Java tools default to a heap far smaller than the machine's RAM, so a 300 GB server still runs the tool in a couple of gigabytes unless told otherwise. Passing a memory flag to the tool often does nothing; the ceiling belongs to the JVM.
Set it on the JVM: java -Xmx8g -jar tool.jar …. If the tool starts from its own launcher, the flag goes in that launcher's config file — and see the .vmoptions entry above, which silently fails to parse on Windows when saved with CRLF endings.