PondScum

Warning - geek stuff ahead! I do this in my day job

Out of memory in this case means out of virtual address space, which as you say is 2 GB for a normal 32-bit process (the OS is mapped into the other 2 GB). Everything accessed in a particular run of the game, both code and data, needs to find somewhere to sit in that 2 GB. The operating system handles all of this for the programmer, who just says "link this DLL" or "allocate some heap space": the operating system then decides where to put those new objects in the 2 GB address space. Then the 2 GB of virtual addresses are mapped onto real physical addresses in RAM during execution - but these physical addresses change on the fly as bits of code and data are paged in and out of RAM to disk. Your code still has the illusion that "hey, I've got 2 GB of virtual addresses, no problem!" no matter whether it's running on a 1 GB, 2 GB, or 4 GB machine. This is why machines with low physical RAM can run large programs - just verrrrry slowwwwly, because they are constantly swapping code and data in and out of RAM.

An out of memory error means the program tried to load or create something new, and the operating system said "your 2 GB of virtual addresses is full, tough luck!", at which point most programs just roll over and die (it's VERY hard to reliably recover from out-of-memory errors). This isn't always because the 2 GB is actually full - it just means that it's gotten so fragmented that the only free space left is in very small bits, and your code is asking for a contiguous range of addresses that won't fit in any of the small bits left. Sadly there is no way to "defragment" your virtual address space. In my day job I work with code that bumps up against the 1.6-1.7 GB range - once we get to 1.8 GB, a crash is almost certain to occur.

Root causes can be just trying to load too much (very large scenarios), and/or a slow memory leak which gradually fills up the 2 GB with allocated objects that aren't being properly freed (this shows itself as a crash after some playing time). Obviously if you have a very large scenario AND a slow memory leak, then you'll crash sooner.

If you want to look at the virtual address space of a process, then the free VMMap tool from SysInternals (now bought out by Microsoft) is the #1 choice. You can get it from http://technet.microsoft.com/en-us/sysinternals/dd535533.

Here's a VMMap screenshot showing what CMBN's address space looks like just after launch: the important numbers are to the right of the top three bars. Note that it has "committed" (used) 239,856 KB out of that 2 GB, and of that 125,848 KB is in my working set (i.e. in RAM at this instant)



Now I load a turn into CMBN, and look how the numbers have shot up. I'm up to 641,500 KB committed, and all of the growth is in heap space (the orange section of the bar), which makes sense since that's used to store game data



My bet is that if you load a huge scenario, play for a while, and then take a VMMap snapshot, you'll see that the committed space is up above 1.5 GB. If you keep playing and it keeps steadily increasing, that's probably a leak. And it'll crash before you get to 2 GB.