Archive for November, 2012

Troubleshooting Plugins

Wednesday, November 21st, 2012

Solution:

For reference and to assist in the troubleshooting process, we have listed the following plugins used in the applcation.  The reference shows the plugin name as seen in the corresponding ar.cfg/ar.conf file, the actual plugin name shown in the Plugin log, and what the specific plugin does.
Plugin: arapprove

Actual Plugin Name:ARSYS.ARDBC.PREVIEW
What it does:Runs the approval process
Plugin: libcmdbconsolefilterapi20

Actual Plugin Name:  BMC.ARF.CMDBCONSOLE
What it does: Used to compare instances in CMDB Console
 

Plugin: ardbcQuery

Actual Plugin Name:  REMEDY.ARDBC.APPQUERY
What it does: Used to query the overview console and return data from the vendor form for Incident, Problem, Change and Asset

Plugin: arfcbdata

Actual Plugin Name:  REMEDY.ARF.CBDATA
What it does: Used for chargebacks

Plugin: caieventcmd

Actual Plugin Name:  REMEDY.ARF.CAI
What it does: Takes data from the CAI Event form and creates a record in the appropriate Interface_Create form.

Plugin: omfobjiefilapi

Actual Plugin Name:  REMEDY.ARF.OMFOBJIE
What it does: Used for exporting sla’s

Plugin: arfslasetup

Actual Plugin Name:  REMEDY.ARF.SLASETUP REMEDY.ARF.SLASETUP
What it does: When you Register Application forms in SLA, this builds the _SLA join form
 

 
From the application standpoint we expect to see the ar.cfg/ar.conf file show the Plugin and the plugin path.  The plugin path should only be seen one time for each plugin used in multiple apps.  Whichever application was installed first will update the ar.cfg/ar.conf file with the plugin and plugin path.  For example, the caieventcmd plugin on unix would show the following if Problem Management was the first app installed:
 

Plugin-Path: /usr/arsystem/appslinux02/problem/bin
Plugin: /usr/arsystem/appslinux02/problem/bin/libcaieventcmd.so
 

On a windows server it would show the plugin path and plugin in one line:
Plugin: “D:\AR System Applications\itsm70ars70w2k3\BMC Remedy Problem Management\bin\caieventcmd.dll”
 

 

Troubleshooting ARERR [8755] The specified plug-in does not exist errors: <Plugin Name>

The name of the plugin will show in the above error.  For this type of error message, you must know the actual plugin name from above, for example, REMEDY.ARF.CAI, to know which entry to look for in the ar.cfg/ar.conf file. 
 

1.  Verify the ar.cfg/ar.conf file has the plugin and plugin path.
On windows:
 “D:\AR System Applications\itsm70ars70w2k3\BMC Remedy Problem Management\bin\caieventcmd.dll”
 

On UNIX:
Plugin-Path: /usr/arsystem/appslinux02/problem/bin
Plugin: /usr/arsystem/appslinux02/problem/bin/libcaieventcmd.so
 

2.  For the above location, go to the directory on the server and verify that the actual plugin resides in the directory listed and permissions are consistent with the user arserverd/arserver.exe run as (especially important when transitioning to a non-root install.)

 

Troubleshooting ARERR [8939] The AR System Plug-In server is not responding.  Cannot connect to the system at this time.  Contact your AR System Administrator for assistance. : RPC: Timed out

1.Enable Plugin and API logging on the server and perform the action that generates the timeout message.
2.Review both the Plugin and API log for delays in processing.  3.If the delay occurs in the Plugin log but the API log looks ok, then the issue is with the plugin entries in the ar.cfg/ar.conf file.

You should have the following:
 Server-Plugin-Default-Timeout:<your setting>
 Plugin-Loopback-RPC-Socket: <your setting>
 Plugin-ARDBC-Threads: <your setting>
 Plugin-Filter-API-Threads: 1 15
 
***Please see the AR System Configuration Guide beginning on page 330 for more information on these settings and how to adjust them for your specific environment***** http://documents.bmc.com/supportu/documents/84/66/58466/58466.pdf
 

4.For the REMEDY.ARF.CAI plugin, make sure Approval Server is installed and you have configured the CAI:Plugin registry form as noted on page 124 of the BMC Remedy IT Service Management Architecture Guide http://documents.bmc.com/supportu/documents/57/43/65743/65743.pdf.  As a technique for reducing thread contention, the private queue and number of threads to be used for the CAI plug-in can be defined using this form. To use this feature, you must define a private queue using BMC Remedy Administrator, and then update the CAI Plug-in Registry with the queue number and number of threads. If the applications are running on a specific port and not on portmapper, you will need to update the plugin from patch 004 of the applications. A change was made to this plugin to allow it run on a specific port. 
5.  If the delay is in the API logging, then the server may not have enough fast and list threads to perform plugin operations.  Adjust your threads via the Administrator tool as needed.

7.5 Plugin config and log files

Wednesday, November 21st, 2012

Problem:

  • How to determine which plugin is which
    How to determine which config file goes with which plugin
    How to determine which file to edit to enable plugin logging for a given plugin
    BMC Remedy AR System Server
     

Solution:

  • Your paths and server names will differ. These are all the defaults except we used /data1 instead of /opt for the install; the server name is rhel4-rs-vm1. Also see KB KM-000010023751 for an explanation of what the ITSM plugins do.

 

NOTES:

The configuration for the arplugin daemon is in the ar.conf file. “Plugin:”, “Plugin-Path:” and “Plugin-Port:” only apply to the C arplugin daemon. The configuration for the Java plugins are done via separate .xml files. The port the Java plugin server runs on and which plugins are loaded by that Java plugin server are in the pluginsvr_config.xml file. Logging for each Java plugin server and the individual plugins loaded by that Java plugin server are configured in the log4j_pluginsvr.xml file for each. The actual log file name is listed in each log4j_pluginsvr.xml file.
 

C plugins (loaded by arplugin daemon):
Plugin: ardbcconf.so
Plugin: reportplugin.so
Plugin: ServerAdmin.so
Plugin: FlashboardObject.so
Plugin: /data1/bmc/ARSystem/arealdap/arealdap.so
Plugin: /data1/bmc/ARSystem/ardbcldap/ardbcldap.so
Plugin: /data1/bmc/ARSystem/approval/bin/arapprove.so
Plugin: /data1/bmc/AtriumCore/rhel4-rs-vm1/cmdb/server/bin/libcmdbconsolefilterapi75.so
Plugin: /data1/bmc/BMCRemedyITSMSuite/rhel4-rs-vm1/Shared_Components/bin/libardbcQuery.so
Plugin: /data1/bmc/BMCRemedyITSMSuite/rhel4-rs-vm1/Shared_Components/bin/libarfcbdata.so
Plugin: /data1/bmc/BMCRemedyITSMSuite/rhel4-rs-vm1/Shared_Components/bin/libcaieventcmd.so
Plugin: /data1/bmc/BMCRemedyITSMSuite/rhel4-rs-vm1/Shared_Components/bin/libNextId.so
 

 Java plugins (loaded by 3 separate Java plugin servers, all of which call the same Java plugin server class from the ARServer directory).
Because these don’t run on the C arplugin daemon specified by Plugin-Port: they must be aliased to point to the individual Java plugin server port so that arserverd knows where to call them at.
Server-Plugin-Alias: RMDY.ITSM.RLE RMDY.ITSM.RLE rhel4-rs-vm1:9999
Server-Plugin-Alias: ARSYS.ARF.REGISTRY ARSYS.ARF.REGISTRY rhel4-rs-vm1.labs.bmc.com:9999
Server-Plugin-Alias: ARSYS.ARF.WEBSERVICE ARSYS.ARF.WEBSERVICE rhel4-rs-vm1:9999
Server-Plugin-Alias: BMC.FILTERAPI.NORM.ENGINE BMC.FILTERAPI.NORM.ENGINE rhel4-rs-vm1:9555
Server-Plugin-Alias: AIS.FILTERAPI AIS.FILTERAPI rhel4-rs-vm1:9556
Server-Plugin-Alias: BMC.ARDBC.ATRIUM.API BMC.ARDBC.ATRIUM.API rhel4-rs-vm1:9556
Server-Plugin-Alias: DSM.FILTER DSM.FILTER rhel4-rs-vm1:9556
 

From armonitor.conf
            C plugin daemon:
            /data1/bmc/ARSystem/bin/arplugin -s rhel4-rs-vm1 -i /data1/bmc/ARSystem

             ARSystem Java plugin (port 9999 by default):
            /usr/java/jdk1.6.0_06/jre/bin/java -Xmx512m -classpath /data1/bmc/ARSystem/pluginsvr:/data1/bmc/ARSystem/pluginsvr/arpluginsvr75.jar com.bmc.arsys.pluginsvr.ARPluginServerMain -x rhel4-rs-vm1 -i /data1/bmc/ARSystem
 
            Atrium Java plugin (port 9556 by default):
            /data1/bmc/AtriumCore/rhel4-rs-vm1/cmdb/server/bin/atriumplugin.sh
 
            Normalization Engine – which is a plugin loaded by a Java plugin server (port 9555 by default):
            /data1/bmc/AtriumCore/rhel4-rs-vm1/cmdb/server/bin/normeng.sh
 
ARSystem Java Plugin config file and log4j (logging config file):
            /data1/bmc/ARSystem/pluginsvr
            Filenames:
            Config: pluginsvr_config.xml
            Logging: log4j_pluginsvr.xml
            Plugin logfile: /data1/bmc/ARSystem/db/arjavaplugin.log
 
Atrium Java Plugin config file and log4j (logging config file):
            /data1/bmc/AtriumCore/rhel4-rs-vm1/cmdb/plugins/shared
            Filenames:
            Config: pluginsvr_config.xml
            Logging: log4j_pluginsvr.xml
            Plugin logfile: /data1/bmc/AtriumCore/rhel4-rs-vm1/Logs/AtriumPluginSvr.log
 
Normalization Engine config file and log4j (logging config file):
            /data1/bmc/AtriumCore/rhel4-rs-vm1/cmdb/plugins/ne
            Filenames:
            Config: pluginsvr_config.xml
            Logging: log4j_pluginsvr.xml
            Plugin logfile: /data1/bmc/AtriumCore/rhel4-rs-vm1/Logs/AtriumPlugin.log