Solution : https://service.sap.com/sap/support/notes/1467134 (SAP Service marketplace login required)
Key words : 
/3 kernel=> 64 bit aix kernel=> heap limit      = unlimited=> stack limit     = 4294967296=> core limit      = unlimited=> file size limit = unlimited=> heap address  = 0x0x113b09420=> stack address = 0xfffffffffffecd0=> stack low     =  0xfffffffffffc3e0=> stack high    =  0xffffffffffff890-------------------------------------------=> stack trace, size=103219256 callerid='ictcm2' [ictxxem, aix 64-bit platforms, aix 64-bit system, relevant kernel patch, sample trace log, memory management configuration problem, memory management profile parameters, 0x101a2b2f8------------------------------------------- solution correctly configure, memory allocation fails due
Related Notes : 
       
| 619235 | SYSTEM_CORE_DUMPED in the IctCmDeleteCache function | 
| 445533 | Lots of extended memory on AIX (64 bit) | 
| 329021 | SPOOL_INTERNAL_ERROR, ERROR => EmIAllocMmResourceEg | 
| 146528 | Configuration of R/3 on hosts with much RAM | 
| 103747 | Performance: Parameter recommendations as of Release 4.0 |