Basis Errors and Solutions

Symptom

The db2diag.log fills up with the following messages:
2009-05-20-12.54.45.235332-240 E1660A535 LEVEL: Warning
PID : 2257034 TID : 14138 PROC : db2sysc
0
INSTANCE: xxxxxx NODE : 000 DB : xxx
APPHDL : 0-39 APPID:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AUTHID : xxxxxx
EDUID : 14138 EDUNAME: db2agent (xxx) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool,
probe:2000
DATA #1 : String, 55 bytes
Sending a signal to clean up NOT THREADED FMP process:
DATA #2 : Process ID, 4 bytes
1966198
2009-05-20-12.54.45.235788-240 I2196A1216 LEVEL: Error
PID : 2257034 TID : 14138 PROC : db2sysc
0
INSTANCE: xxxxxx NODE : 000 DB : xxx
APPHDL : 0-39 APPID:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AUTHID : xxxxxx
EDUID : 14138 EDUNAME: db2agent (xxx) 0
FUNCTION: DB2 UDB, oper system services, sqlossig, probe:10
MESSAGE : Sending SIGKILL to the following process id
DATA #1 : signed integer, 4 bytes
1966198
CALLSTCK:
[0] 0×0900000020314580 pdLog + 0xD0
[1] 0×090000001FED0C44 pdLog@glue3E1 + 0xD0
[2] 0×09000000213DA524 sqlossig + 0xEC
[3] 0×09000000238A1C08 sqlossig@glue5E8 + 0×7C
[4] 0×090000001FD74968
sqlerReturnFmpToPool__FcT1P14sqlerFmpHandleP8sqeAgent + 0×620
[5] 0×090000002014DBF0
sqlerReturnFmpToPool__FcT1P14sqlerFmpHandleP8sqeAgent@glue5B3 +
0×7C
[6] 0×09000000204B7E98
sqlri_close_udf_ufob_cleanup__FP10sqlri_ufobP5sqlcaP8sqlrr_cb +
0×3C
[7] 0×090000002035D4D0
sqlri_close_udf_eos__FP25sqlra_sql_context_siblingiP8sqlrr_cb +
0×160
[8] 0×0900000022D2589C
sqlri_close_udf_eos__FP25sqlra_sql_context_siblingiP8sqlrr_cb@gl
ueF80 + 0×78
[9] 0×09000000204FDED8 sqlricls_complex__FP8sqlrr_cbilN23 +
0xA4
2009-05-20-12.54.45.240918-240 E3413A470 LEVEL: Warning
PID : 2523314 TID : 258 PROC : db2wdog
0
INSTANCE: xxxxxx NODE : 000
EDUID : 258 EDUNAME: db2wdog 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool,
probe:999
DATA #1 : String, 34 bytes
Removing FMP from pool FMP handle:
DATA #2 : sqlerFmpHandle, PD_SQLER_TYPE_FMP_HANDLE, 16 bytes
fmpPid: 1966198 pFmpEntry: 0×0000000000000000
2009-05-20-12.54.45.241285-240 E3884A490 LEVEL: Warning
PID : 2523314 TID : 258 PROC : db2wdog
0
INSTANCE: xxxxxx NODE : 000
EDUID : 258 EDUNAME: db2wdog 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool,
probe:1000
DATA #1 : String, 10 bytes
Fmp Entry:
DATA #2 : sqlerFmpThreadList, PD_SQLER_TYPE_FMP_THREAD_LIST, 408
bytes
Object not dumped: Address: 0×0000000000000000 Size: 408 Reason:
Address is NULL
2009-05-20-12.54.45.241590-240 E4375A396 LEVEL: Error
PID : 2523314 TID : 258 PROC : db2wdog
0
INSTANCE: xxxxxx NODE : 000
EDUID : 258 EDUNAME: db2wdog 0
FUNCTION: DB2 UDB, base sys utilities, sqleChildCrashHandler,
probe:15
DATA #1 : A non-EDU child crashed. Process id: 1966198, OSS term code:
0×102, signal: 9
Other terms

db6 db2luw db2fmp sqlerReturnFmpToPool
Reason and Prerequisites

The problem can happen with the following settings:
DB2 Registry:DB2_AGENT_CACHING=OFFDBM CFG:KEEPFENCED=YESFENCED_POOL=x

where x is a small number compared to the DB2 agent pool.DB2 level:DB2 LUW V9.5 FP4 and lowerDB2 LUW V9.7 GA

With KEEPFENCED=YES, there is a pool of fenced processes. Within the
pool there are different types of fenced processes. For example, upon
startup, the fenced pool is generally filled with normal non-threadsafe
C processes. Other types include multi-threaded fmp processes, admin
command fmp processes, db2acd fmp process, etc.
With DB2_AGENT_CACHING=OFF, the fenced process is returned to the pool as soon as the execution is completed. If the current size of the pool
is greater than or equal to FENCED_POOL parameter, the fmp process is
removed from the pool.
When a workload runs SYSPROC.ADMIN_CMD DB2 routine with the current clients, an admin fmp worker is needed. Given that the pool is full
of normal C processes, the admin fmp worker is always being terminated
as removed from the pool as soon as their work is done.
As a result, the db2diag.log is filled with admin fmp worker being terminated messages.

Solution

Set the DB2 DBM CFG parameter:
KEEPFENCED=NO

With KEEPFENCED set to NO, there is no longer a fenced process pool. Whenever the workload requires an fenced process, it is created instead of taking a process from a pool, since a pool no longer exists. There will no longer be messages about removing processes from the pool.
Alternatively, you can apply the following APARs
DB2 LUW V9.5: APAR IC61679, contained in FP5.DB2 LUW V9.7: APAR IC61678, contained in FP1.

After the fix is applied you should use the default parameters again.