Archive for the ‘7.1.x’ Category

Troubleshooting Plugins

Wednesday, November 21st, 2012

Solution:

For reference and to assist in the troubleshooting process, we have listed the following plugins used in the applcation.  The reference shows the plugin name as seen in the corresponding ar.cfg/ar.conf file, the actual plugin name shown in the Plugin log, and what the specific plugin does.
Plugin: arapprove

Actual Plugin Name:ARSYS.ARDBC.PREVIEW
What it does:Runs the approval process
Plugin: libcmdbconsolefilterapi20

Actual Plugin Name:  BMC.ARF.CMDBCONSOLE
What it does: Used to compare instances in CMDB Console
 

Plugin: ardbcQuery

Actual Plugin Name:  REMEDY.ARDBC.APPQUERY
What it does: Used to query the overview console and return data from the vendor form for Incident, Problem, Change and Asset

Plugin: arfcbdata

Actual Plugin Name:  REMEDY.ARF.CBDATA
What it does: Used for chargebacks

Plugin: caieventcmd

Actual Plugin Name:  REMEDY.ARF.CAI
What it does: Takes data from the CAI Event form and creates a record in the appropriate Interface_Create form.

Plugin: omfobjiefilapi

Actual Plugin Name:  REMEDY.ARF.OMFOBJIE
What it does: Used for exporting sla’s

Plugin: arfslasetup

Actual Plugin Name:  REMEDY.ARF.SLASETUP REMEDY.ARF.SLASETUP
What it does: When you Register Application forms in SLA, this builds the _SLA join form
 

 
From the application standpoint we expect to see the ar.cfg/ar.conf file show the Plugin and the plugin path.  The plugin path should only be seen one time for each plugin used in multiple apps.  Whichever application was installed first will update the ar.cfg/ar.conf file with the plugin and plugin path.  For example, the caieventcmd plugin on unix would show the following if Problem Management was the first app installed:
 

Plugin-Path: /usr/arsystem/appslinux02/problem/bin
Plugin: /usr/arsystem/appslinux02/problem/bin/libcaieventcmd.so
 

On a windows server it would show the plugin path and plugin in one line:
Plugin: “D:\AR System Applications\itsm70ars70w2k3\BMC Remedy Problem Management\bin\caieventcmd.dll”
 

 

Troubleshooting ARERR [8755] The specified plug-in does not exist errors: <Plugin Name>

The name of the plugin will show in the above error.  For this type of error message, you must know the actual plugin name from above, for example, REMEDY.ARF.CAI, to know which entry to look for in the ar.cfg/ar.conf file. 
 

1.  Verify the ar.cfg/ar.conf file has the plugin and plugin path.
On windows:
 “D:\AR System Applications\itsm70ars70w2k3\BMC Remedy Problem Management\bin\caieventcmd.dll”
 

On UNIX:
Plugin-Path: /usr/arsystem/appslinux02/problem/bin
Plugin: /usr/arsystem/appslinux02/problem/bin/libcaieventcmd.so
 

2.  For the above location, go to the directory on the server and verify that the actual plugin resides in the directory listed and permissions are consistent with the user arserverd/arserver.exe run as (especially important when transitioning to a non-root install.)

 

Troubleshooting ARERR [8939] The AR System Plug-In server is not responding.  Cannot connect to the system at this time.  Contact your AR System Administrator for assistance. : RPC: Timed out

1.Enable Plugin and API logging on the server and perform the action that generates the timeout message.
2.Review both the Plugin and API log for delays in processing.  3.If the delay occurs in the Plugin log but the API log looks ok, then the issue is with the plugin entries in the ar.cfg/ar.conf file.

You should have the following:
 Server-Plugin-Default-Timeout:<your setting>
 Plugin-Loopback-RPC-Socket: <your setting>
 Plugin-ARDBC-Threads: <your setting>
 Plugin-Filter-API-Threads: 1 15
 
***Please see the AR System Configuration Guide beginning on page 330 for more information on these settings and how to adjust them for your specific environment***** http://documents.bmc.com/supportu/documents/84/66/58466/58466.pdf
 

4.For the REMEDY.ARF.CAI plugin, make sure Approval Server is installed and you have configured the CAI:Plugin registry form as noted on page 124 of the BMC Remedy IT Service Management Architecture Guide http://documents.bmc.com/supportu/documents/57/43/65743/65743.pdf.  As a technique for reducing thread contention, the private queue and number of threads to be used for the CAI plug-in can be defined using this form. To use this feature, you must define a private queue using BMC Remedy Administrator, and then update the CAI Plug-in Registry with the queue number and number of threads. If the applications are running on a specific port and not on portmapper, you will need to update the plugin from patch 004 of the applications. A change was made to this plugin to allow it run on a specific port. 
5.  If the delay is in the API logging, then the server may not have enough fast and list threads to perform plugin operations.  Adjust your threads via the Administrator tool as needed.

Clarification of Server-Name, Server-Connect-Name and IP-Name for server group configuration

Wednesday, May 16th, 2012

Problem

There are several parameters used by the ARSystem to understand how to treat the Server name for various components and actions. Having these set incorrectly could cause errors and lead to ARServer behaving in an unplanned manner.
 
Please explain how to setup the Server-Name, Server-Connect-Name and IP-Name parameters

Remedy AR System Server (version 6.3.00 to 7.6.04)
DR: Remedy AR System Server

 

Solution

There are several parameters used by the ARSystem to understand how to treat the Server name for various components and actions.
 
Having these set incorrectly could cause errors and lead to ARServer behaving in an unplanned manner.
 
The most important parameters are Server-Name, Server-Connect-Name and IP-Name for the server group ar.conf/ar.cfg files.
 
 a) The Server-Name entry must be a common value for each ARServer in server group;  It is an alias for the ARServer environment that will be the same for all the servers. It is a name used as a virtual server where all Remedy client computers will connect.  All clients (Mid-Tier, ARS User Tool, API programs) should connect using that value of Server-Name.  
 
In a load-balanced environment, this name resolves to the IP address of the hardware load balancer, which in turn directs the call to one of the AR System servers in the group.  This should be a short host name that resolves to the Load Balancer; not the Fully Qualified Name or the IP Address.  
 
In some case, such as Notification shortcuts and urls,   ARServer will concatenate the domain name of the ARServer host to this Server-Name value to create a fully qualified Server Name.  
 
If the Domain of the ARServer host causes an invalid Fully qualified hostname to be generated, the Domain-Name: paramater can be used to hard code the domain name
Syntax:
 
  Server-Name: <server_group_server_name>
  Domain-Name: <domain name that will create a valid fully qualified name when added to Server-Name>
 
   where   <server_group_server_name>  is the alias name that resolved to the Load Balancer
 
Example:
  Server-Name: remedybox1

  Domain-Name: company.com
 
Users will connect to the server ‘remedybox1’  or  ‘remedybox1.company.com’, which resolves to the load balancer.
 
Notifications and other actions that uses a fully qualified name will use ‘remedybox1.company.com’
 
 
 b) Server-Connect-Name must be a unique value for each ARServer in server group.  The only important details about this entry is that it is case sensitive and it must resolve directly to the ARServer host, not the load balancer.
 
You can use the FQDN (Fully Qualify Domain Name) format instead of the short name to ensure that it resolves correctly.
 
Please notice that you need to use only one Server-Connect-Name entry per server using that server’s hostname
 
Example:
 
  Server-Connect-Name: serverA.your_domain.com             (in serverA ar.conf/ar.cfg)
 
  Server-Connect-Name: serverB.your_domain.com             (in serverB ar.conf/ar.cfg)
           NOTE:
            Server Name field in the AR System Server Group Operation Ranking form should match the Server-Connect-Name with the exact value, including case.
 
c) IP-Name is used by the ARServer when an Admin operation is executed.   Some Admin operations such as creating a Filter, require that the client use the correct Server Name when connecting.  
 
The ARServer compares the Server Name that the client used  with its own server name.   If these names do not match you may encounter errors such as ARERR 225 “Set fields actions that reference entries in other forms can only reference forms on the same server in a filter or escalation. “
 
If this occurs during an action that stores a server name,  it is possible that a hard coded server name with be stored in the definition.
 
To prevent either of these scenarios,  add IP-Name to ar.cfg/ar.conf.   Multiple IP-Name parameters can exist in a single configuration file.   It is recommended that you set an IP-Name parameter for every possible server name that a client may connect to to perform admin operations.  This includes admin operations triggerred by workflow such as Creating new Classes in CMDB or new Targets in SLM.
 
For example:
       IP-Name: <separate entry for short name or hostname for each server in server group>
       IP-Name: <separate entry for FQDN for each server in server group>
       IP-Name: <separate entry for IP address for each server in server group>
 
 
           NOTE:
            To maintain consitenct ar.cfg.ar.conf files, in minimum server group environment of two ARServers there will be a minimum of six separate entries for IP-Name in each config file of each ARServer in that server group.  Each additional ARServer in server group will have additional three entries in each ar.cfg/conf file of each ARServer in that server group.
         Example:

         IP-Name: serverA
         IP-Name: serverA.your_domain.com
         IP-Name: xxx.xxx.xxx.xxx                       (where xxx.xxx.xxx.xxx is the IP address of serverA)
         IP-Name: serverB
         IP-Name: serverB.your_domain.com
         IP-Name: yyy.yyy.yyy.yyy                       (where yyy.yyy.yyy.yyy is the IP address of serverB)
         
         
 Finally, note the following two basic concepts about server groups:

 1.) It is important that you know that servers in server group MUST be on physically separate server boxes, and you cannot create server group with servers installed and running on the same server box.
 2.) Servers in a server group MUST be the same version and patch because they use the same database, so you cannot create server group with servers of different versions.
 

Find Information about Server Group related tables in database

Wednesday, May 16th, 2012

Problem

Need to know the function of each server group table in database
ARS 7.1/7.5, any OS, any Database

Solution

A) servgrp_applic : This table contains the information about the all application licenses ( Floating) in server group. It also keeps track of Floating App licenses that are being consumed in the server group.

 

B) servgrp_board: This table contains the information about the Server Names in the server group & their TCP Port number if any. In order for any failover to occur, the intervalcount column in the Servgrp_board table has to stop incrementing for that particular server. 

This table can be viewed using a database query tool to obtain some information about the Server Group’s health. 

 

The following columns are useful to look at: 

i) serverName:  this column should contain one record for each server in the group.  The name that appears here should be the same as the Server-Connect-Name and as the ‘Server’ field in the AR System Server Group Operation Ranking form.  If the data is not correct, the Ranking form should be modified to reflect the correct name,  the Server-Connect-Name: parameter  verified in each Configuration file, and the faulty records deleted from this table.  Restarting each ARServer will cause this table to get populated with the correct server names.

 

ii) intervalCount:  gets incremented each time the specified server connects on the interval specified in the Administrator Console for the Server Group Check Interval setting.    If this number is not incrementing or is 0, then the specified server is either not running or is not participating in the server group. 

iii) opFlags:   This is a semi-colon delimited list of 0’s and 1’s indicating which operations are currently owned by the specified server.   If the values are not what you expect, then there may be a problem with the ranking form, such as invalid server names. 

C) servgrp_config: This table contains the information about the checkInterval count. The synchronization between two servers in the server group takes two(2) check interval times (15 seconds each) before the changes are applied on the remaining servers. This means, changes can be piled and delayed if the pending signals keeps changing within the checkInterval time in servgrp_config table. Once the synchronization is completed, the operation is passed to the next server.

 

D) servgrp_ftslic:  This table contains the information about the Full Text Search licenses in the Server Group. This calculates the FTS licenses & store at the central place. 

E) servgrp_op_mstr:    If you do “Select operation from servgrp_op_mstr” on database, it will show you all 13 operations by default. But if you open servgrp_op_mstr table, under the “opNum’ column & there will be a value of -1 (minus 1) for any operation that is not used probably. Only the operation which has positive number value , that server group assumes active & has a rank defined in the AR Server Operation Ranking form. You also have a category of each operation, like what application that operation belong to.

 

F) servgrp_userlic: This table contains the information of all the User Fixed & Floating license in the server group. If you have 10 User floating licenses on server A & 15 User floating licenses on server B, then in this table the User floating licenses calculated as 25 number of available User floating licenses in the server group. You have userName, serverName & groupId columns on the servgrp_userlic table which contain the user name, server name of that user & group id of user which that user is member of.