Problem: In Oracle Applications Manager, in Applications Dashboard, on the database is appearing up. The remaining servers are shown as down.

Solution:

1. Tried stopping and restarting the entire system (stopPROD.sh and startPROD.sh).

Executing service control script:

/emca/PROD/prodcomn/admin/scripts/PROD_linux-blade/adrepctl.sh stop

script returned:

****************************************************
You are running adrepctl.sh version 115.29
Reports Server for PROD is already stopped.
adrepctl.sh: exiting with status 2
.end std out.
.end err out.
****************************************************
Executing service control script:
/emca/PROD/prodcomn/admin/scripts/PROD_linux-blade/adfmcctl.sh stop
script returned:
****************************************************
You are running adfmcctl.sh version 115.16
forms load balancing client for PROD already stopped.
adfmcctl.sh: exiting with status 2
.end std out.
.end err out.
****************************************************
Executing service control script:
/emca/PROD/prodcomn/admin/scripts/PROD_linux-blade/adfmsctl.sh stop
script returned:
****************************************************

Subject:  Summary of Possible Reasons and Solutions for the Problem Where All Concurrent Requests Stuck in Pending Phase Doc ID:  Note:182154.1

2.  When shutting down the concurrent manager are there any FNDLIBR processes still running at the OS level?   If so, do a kill -9 on them.  Then restart the concurrent manager.

Tried this, but CM stopped successfully with no need to kill FNDLIBR.

3.  Rebuild the concurrent manager views.  As applmgr run the following from the OS:

This is non-destructive.

Concurrent Manager views can be rebuild by running the following command at the command line:

Ensure that concurrent manager is shutdown.

FNDLIBR FND FNDCPBWV apps/apps SYSADMIN 'System Administrator' SYSADMIN

Restart the concurrent manager.

Executed the above, but no change.

Executed the code:

update fnd_concurrent_queues
set control_code = 'A'
where concurrent_queue_name = 'FNDCRM';
commit;
SQL> update fnd_concurrent_queues
2  set control_code = 'A'
3  where control_code = 'N'
4  /
SQL> select control_code, target_node, concurrent_queue_name
2  from fnd_concurrent_queues
3  where control_code = 'N'
4  /
C TARGET_NODE                    CONCURRENT_QUEUE_NAME
- ------------------------------ ------------------------------
N                                FNDCRM
N                                FNDSCH
N                                STANDARD
N                                PODAMGR
N                                RCVOLTM
N                                PASMGR
N                                INVMGR
N                                INVTMRPM
N                                IEU_SH_CS
N                                IEU_WL_CS
N                                OAMCOLMGR
C TARGET_NODE                    CONCURRENT_QUEUE_NAME
- ------------------------------ ------------------------------
N                                OAMGCS_LINUX-BLADE

updated /etc/hosts file:

[applmgr@linux-blade etc]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain   localhost
128.x.x.x      linux-blade.domainname  linux-blade
SQL> create table fnd_concurrent_queues_vseeram
2        as select * from fnd_concurrent_queues;