I have installed OIM [11GR2 PS2] and OAM [R2PS2] in my PC, but system hangs with 12Gb of RAM. I have I3 5th generation processor along with 12 Gb of RAM.I use win10 as my basic OS; however for installing oracle product I use VM where I have installed win7[ultimate version ]. Though as per oracle pre-requisite chart, 8GB of RAM is enough to run single instance of OIM / OAM, however I have allocated almost 10.5 GB of RAM to those VM's running OIM / OAM, but each time, after admin server start, whenever I try to start any of the manage server, the CPU consumption reaches 100% and everything hangs, I had to shut down my VM.
Though the question is a basic one, but have not found exact answer anywhere. Looking for help/suggestion .
The memory requirement of 8 GB is bare minimum and 16 GB is recommended. See this 11gR2 memory requirements and 11gR2 requirements. Also Refer to 3.1 Minimum Memory Requirements for Oracle Identity and Access Management and the section 3.3 Examples: Determining Memory Requirements for an Oracle Identity and Access Management Production Environment. (Even though it is mentioned Production but is valid for your instance since you have one VM, which is hosting all the components, inlcuding WebLogic server, OIM server, SOA server and also OAM server.
Here is the estimate of RAM from the above Oracle 11gR2 reference
To estimate the suggested memory requirements, you could use the following formula:
4 GB for the operating system and other software
+ 4 GB for the Administration Server
+ 8 GB for the two Managed Servers (OIM Server and SOA Server)
-----------------------------------------------------------
16 GB
With 4 GB for OS and 4 GB for Admin, that makes 8 GB RAM consumed already. And as you start a Managed server which would make it 12 GB, which the VM does not have... Hence as soon as you start your Managed server the all RAM is consumed which makes your VM to hang.
As you can see Oracle is recommending 16 GB and that too it is without OAM server (which also you have installed on the same VM). So definitely you are constrained with your current 10.5 GB. Since your PC max is 12 GB, suggest you install only OIM on one VM on the current PC and OAM on a different VM on separate PC if possible. Yes Oracle IAM software is definitely a memory hog.
BTW, I have two suggestions for you, first if you want to install 11gR2 version then go for PS3 (11.1.2.3) or better go with 12c which is latest. 11.1.2.2 is considered old now. Here is link for PS3 download. And second consider Oracle's free downloadable Pre-built VMs here. Although the pre-built VMs will be on linux.