I just spent a bunch of time with an odd problem when I upgraded from vmware workstation 10 to 11 on a fedora 20 linux hosts running a windows 7 guest OS.
When I logged in, with the vm configured with 3D display setting enabled and 2 gigs of shared memory, the OS would spit out a message saying there wasn't enough resources for the current display setting and that it reverted the display settings back to a basic setting. It would then turn off the Areo display stuff. (transparent borders etc.)
I have two UHD displays and an initial physical memory configured at 8 gigs.
After messing around a bit with the display settings under the Windows 7 guest OS, I stumbled across an odd solution to the problem.
First I expanded the guest physical memory from 8 gigs to 16 gigs. I booted up the Windows 7 OS and the error went away. I then shut the system down, and shrank the physical memory back to 8 gigs. I booted up and the error message didn't come back. So for some reason, expanding and contracting the physical memory seems to have made the problem go away.
Because of this, I think the underlying problem was that I didn't have enough "virtual" memory or swap space configured on the guest Windows 7 OS. The expanding of physical memory to 16 gigs from 8 gigs forced the system to reconfigure the "virtual" memory or swap space and then when I shrank the physical memory back to 8 gigs from 16 gigs, the swap or virtual memory settings remained where they were.
That's all just a theory. In any case, I wanted to post this in case someone else ended up with this same problem.
Cheers.