I'm assuming that this error is occurring when you start RVC.
Since RVC is running on your vCenter server (by default), it might mean that you need to add additional memory resources to your vCenter.
An alternative is to deploy a new vCenter server VM, and not to use it for VC tasks but just use it for RVC and VSAN Observer activity.
HTH
Cormac