In the error notification was an ORA-29532:

Event Error Name: -29532
Event Error Message: ORA-29532: Java call terminated by uncaught Java exception: java.lang.ArrayIndexOutOfBoundsException
Event Error Stack:
WF_XML.getResponseDetails()
WF_XML.handleReceiveEvent(oracle.apps.wf.notification.receive.message, C0F79E8FE39B266EE0301490C8C41A0C)

The underlying issue turned out to be a regressed XDK after a database upgrade, and a workaround was to bounce the Workflow Mailer Agent Listeners until the patch (5194357) could be reapplied. But that still left a certain number of workflows in limbo, with the respective users unaware that their actions had not “taken.” So, knowing the identification number of the first errored workflow (from the subject of the error notification email sent to the sysadmin), I wrote a small script to identify these workflows:

select
 recipient_role,
 status,
 mail_status,
 to_user,
 subject
from
 wf_notifications
where
 notification_id in (
 select
 substr(subject, -7) wf_notification_id
 from
 wf_notifications
 where
 subject like 'External Event ERROR%'
 and notification_id > &id_number)
order by
 to_user

References

How to Identify the Version of XDK Installed Inside the Database [ID 177411.1]

Response Workflow Emails Not Being Processed by the Mailer and Still Shows in Process Resulting in ORA-29532 [ID 1288090.1]

SELECT
  gps.period_num,
  gps.period_name,
  gps.set_of_books_id,
  gsob.period_set_name,
  DECODE (gps.closing_status,
    'O', 'Open',
    'C', 'Closed',
    'F', 'Future',
    'N', 'Never') status
FROM
  gl_period_statuses gps,
  gl_sets_of_books gsob
where
  gps.period_name like '%11' and
  gps.application_id = 101 and
  gsob.set_of_books_id = gps.set_of_books_id
order by
  gps.period_num

I’m putting these queries on the blog in case someone else finds them useful. The queries were originally for 10gR2.

Data file locations:

SELECT 
  name 
FROM 
  v$datafile;

Temp file locations:

SELECT
  name 
FROM
  v$tempfile;

Undo file locations:

SELECT
  name
FROM
  v$datafile
WHERE
  ts# = (
         SELECT
           ts#
         FROM
           v$tablespace 
         WHERE
           name = (SELECT
                     tablespace_name
                   FROM 
                     dba_tablespaces
                   WHERE
                     contents = 'UNDO')
        );

Archive log destinations:

ARCHIVE LOG LIST;

- or -

SELECT destination FROM v$archive_dest;

Control file locations:

SELECT name FROM v$controlfile;

Redo log locations:

SELECT member FROM v$logfile;

Checking ARCHIVELOG mode:

SELECT log_mode FROM v$database;

Enabling ARCHIVELOG mode:

SQL> startup mount
ORACLE instance started.

Total System Global Area 184549376 bytes
Fixed Size 1300928 bytes
Variable Size 157820480 bytes
Database Buffers 25165824 bytes
Redo Buffers 262144 bytes
Database mounted.

SQL> alter database archivelog;
Database altered.

SQL> alter database open;
Database altered.

Determine which datafiles have AUTOEXTEND enabled:

SELECT file_name FROM dba_data_files WHERE autoextensible = 'YES';

Disable AUTOEXTEND for a datafile:

ALTER DATABASE DATAFILE '/path/to/datafile.dbf' AUTOEXTEND OFF

Enable AUTOEXTEND for a datafile, but with a maximum size:

ALTER DATABASE DATAFILE '/path/to/datafile.dbf' AUTOEXTEND ON MAXSIZE 500M

Estimate size of SYSAUX tablespace (after system has been running for >1 days):

@$ORACLE_HOME/rdbms/admin/utlsyxsz.SQL;

and answer the questions that the script asks

Change archive log destination:

ALTER SYSTEM SET log_archive_dest_1='LOCATION=/path/to/archivelogs';

Getting ORA-19815 errors

“ORA-19815: WARNING: db_recovery_file_dest_size of 2147483648 bytes is 100.00% used, and has 0 remaining bytes available.”

What’s happening:

* Oracle is trying to write to the flashback area, but is bumping up against
a hard limit defined by the db_recovery_file_dest_size parameter

Possible causes:

* In ARCHIVELOG mode, but no destination is set (defaults to flashback)

Possible fixes:

* Set one or more archive log destinations
* Increase the db_recovery_file_dest_size (if not full)
- BUT: You’ll probably end up having this error again in the future
if you try to use this as a permanent fix

Open Cursors:

SELECT 
  a.VALUE,
  b.name
FROM
  v$mystat a,
  v$statname b
WHERE
  a.statistic# = b.statistic# AND
  a.statistic# = 3;

Who opened cursors:

SELECT user_name, STATUS, osuser, machine, a.sql_text
FROM v$session b, v$open_cursor a
WHERE a.sid = b.sid;

Max number of cursors per user:

SELECT *
FROM v$parameter
WHERE name = 'open_cursors';

As you know, one of the biggest changes in TSM 6.x is that it is now backed by DB2.  Previous releases of TSM used a proprietary database that worked well, but had certain limitations, especially with scaling.

In any case, Chad was using the following query to find old Oracle database backups that had not been properly cleaned:

select \
 cast(sum(b.file_size/1073741824) as decimal(18,2)) AS GB_SIZE \
from \
 backups a, \
 contents b \
where \
 a.node_name in ('DEV01_ORA','DEV02_ORA','DEV03_ORA','PRD01_ORA','PROD02_ORA') and \
 a.backup_date < '2011-11-01 00:00:00' and \
 a.object_id=b.object_id

Chad’s problem was that the query was slow.  Really, really slow.  It sounded like an interesting problem (I know, I’m weird), and since one of my many hats is doing Oracle support, I have a seen a fair amount of poorly-performing database queries.  And usually, the culprit is the query itself, and not the database.  I’m not anyone’s idea of a SQL tuning expert, but it seemed like a good opportunity to learn.

I confirmed what Chad was seeing by issuing a select count(*) against backups and contents, and the contents query in particular took very long to execute.  Additionally, it returned tens of millions of rows, so there was a lot of data to process.

Since the CLI for DB2, db2,  is included with TSM, I used it to query the database directly.  Once connected, the first order of business was to find out what the query was hitting: what are backups and contents?

db2 => set schema tsmdb1
DB20000I  The SQL command completed successfully.
db2 => list tables for schema tsmdb1

Table/View                      Schema          Type  Creation time
------------------------------- --------------- ----- --------------------------
.
.
.
BACKUPS                         TSMDB1          V     2011-09-14-19.05.58.424225
.
.
.
CONTENTS                        TSMDB1          V     2011-09-14-19.06.04.636090
.
.
.

Well.  We can see from the ‘V’ in the Type column that the two objects are views.  A view is essentially a SQL code block that you can query and update just like you would a table.  This is great for abstracting a complicated database, but whenever you execute a statement against it, it gets executed itself.  So there may be a whole lot more processing involved than you expect or need.

In order to examine the SQL code for the views, DB2 provides the db2look command-line utility.  Here are the commands to dump the code for the two views.  Note that the output is being placed into a text file.

db2look -d TSMDB1 -e -v backups -o backups.sql

 and

db2look -d TSMDB1 -e -v contents -o contents.sql

After looking at the source code, I found that most of what Chad was looking for seemed to be in the backup_objects table.  Here’s my rewrite of Chad’s query:

select \
  sum(cast(bk.bfsize/1024/1024 as decimal(14,4))) AS MB \
from \
  backup_objects bk \
where \
  bk.nodeid in (select nodeid from nodesview where node_name in ('DEV01_ORA','DEV02_ORA','DEV03_ORA','PRD01_ORA','PROD02_ORA')) and \
  bk.insdate<'2011-11-01 00:00:00'

Chad pointed out that you can’t query the nodes table from the dsmadmc client (why??), however you are able to query against the nodesview view.  You’ll need to change the column from nodename to node_name when you do.

With the new query, Chad is able to get results within 10 minutes.  Still not blazing fast, but it sounds like the query is usable now.

References:

DB2 Doesn’t Make A Difference

db2look – DB2 statistics and DDL extraction tool command

Here’s a look at the volumes in the storage pool:

tsm: SERVER1>q vol stg=mypool1

Volume Name                  Storage         Device         Estimated       Pct      Volume
                             Pool Name       Class Name      Capacity      Util      Status
------------------------     -----------     ----------     ---------     -----     --------
/path/to/mypoolvol01         MYPOOL1         DISK             500.0 G      55.2     On-Line
/path/to/mypoolvol02         MYPOOL1         DISK             500.0 G      90.0     On-Line

The first step is to disable caching of migrated files on the storage pool.

tsm: SERVER1>update stgpool mypool1 cache=no

As you can see, this has no effect on the storage space used; the data remains in the storage pool.

tsm: SERVER1>q vol stg=mypool1

Volume Name                  Storage         Device         Estimated       Pct      Volume
                             Pool Name       Class Name      Capacity      Util      Status
------------------------     -----------     ----------     ---------     -----     --------
/path/to/mypoolvol01         MYPOOL1         DISK             500.0 G      55.2     On-Line
/path/to/mypoolvol02         MYPOOL1         DISK             500.0 G      90.0     On-Line

However, now that caching has been disabled, if we use the “move data” command, any cached data in the storage pool will be cleared:

tsm: SERVER1>move data /path/to/mypoolvol01
ANR2232W This command will move all of the data stored on volume /path/to/mypoolvol01 to other volumes within the
same storage pool; the data will be inaccessible to users until the operation completes.

Do you wish to proceed? (Yes (Y)/No (N)) y
ANS8003I Process number 1094 started.

tsm: SERVER1>q vol stg=mypool1

Volume Name                  Storage         Device         Estimated       Pct      Volume
                             Pool Name       Class Name      Capacity      Util      Status
------------------------     -----------     ----------     ---------     -----     --------
/path/to/mypoolvol01         MYPOOL1         DISK             500.0 G       0.0     On-Line
/path/to/mypoolvol02         MYPOOL1         DISK             500.0 G      90.0     On-Line

With the volume now empty, we can remove the volume from the storage pool:

tsm: SERVER1>del vol /path/to/mypoolvol01
ANR2220W This command will delete volume /path/to/V02737713.BFS from its storage pool after verifying that the volume
contains no data.

Do you wish to proceed? (Yes (Y)/No (N)) y

The last step is to remove the file from the filesystem:

# rm /path/to/mypoolvol01

ORA-01795: maximum number of expressions in a list is 1000

Which, honestly, is one of the most honest and straightforward ORA codes. So what are you going to do about it?  You can either break your query into separate statements with 1,000 expressions each, or you can make life a lot easier and load what you need into a scratch table using the Oracle SQL*Loader utility (aka sqlldr).  This will let you query against data within the database.

SQL*Loader, unsurprisingly, loads data from a flat file into a table in an Oracle database.  Of course, one of the first things you’ll need to do is to create the table where you want this data to be loaded.  This table will of course be different, depending on what data you are working with:

-------------------------------------------------
-- BEGIN SCRATCH_t.sql

create table scratch_t
  (important_txt   varchar2(30) not null);

-- END SCRATCH_t.sql
-------------------------------------------------

We’ll also need a control file.  This tells sqlldr how to load our data into the database.  Note that the table we just created is referenced here.

-------------------------------------------------
-- BEGIN scratch.ctl

LOAD DATA
INFILE '*'
--INFILE 'scratch_values.txt'
--BADFILE 'scratch.bad'
--DISCARDFILE 'scratch.dsc'
APPEND
INTO TABLE SCRATCH_t
FIELDS TERMINATED BY ' '
OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(
IMPORTANT_TXT
)

-- END scratch.ctl
-------------------------------------------------

Once the table and control file have been created, and the data is in a text file, you are ready to roll.  Invoke SQL*Loader from the command line with appropriate credentials, referencing your input and control files:

[oradood@bigserver ~]$ sqlldr user/pass data=scratch_values.txt,control=scratch.ctl

While the program is running, it will output how many records have been processed.  Once it completes, don’t forget to check the logfile (try scratch.out) for any errors.

Reference

SQL*Loader Command-Line Reference

Some of the checks were things that I’ve looked for (LPNs that have been Shipped Out of Stores, but still have an onhand quantity), but most were ones that I hadn’t built queries for.  In my defense, I am a Unix guy who plays Oracle DBA from time to time… ;^)

If you’re running Oracle E-Business Suite and have WMS-enabled organization(s), it’s definitely useful to check out this script. I opened several SRs after running the script for the first time.  Hopefully this means fewer problems for our warehouse and shipping staff, and therefore fewer tickets for IT.

Reference

WMS Health Check Scripts to Check for Data Corruption [ID 1086783.1]

select
  unique wdd.source_header_number order_number,
  wdd.source_header_id,
  wdd.source_header_type_name order_type,
  wdd.source_line_id line_id,
  msi.segment1 item
from
  wsh_delivery_details wdd,
  mtl_system_items_b msi
where
  wdd.delivery_detail_id in (
    123456,
    789012,
    345678,
    901234) and
  msi.inventory_item_id = wdd.inventory_item_id

• An email to someone at another company bounced back.
• Someone forgot his or her password and/or password.
• A mobile user has poor wireless and/or mobile broadband connectivity.  “I’m at the airport…”
• A mobile user was sitting at home during a power outage.  Wish I were making this up.
• The application is down for scheduled maintenance, as requested and scheduled by the person complaining of the outage.  “You never tell us anything!”
• The person submitting the ticket was told by a coworker that someone on a previous shift had some sort of problem sometime between last night and last week.  “He needs this to do his job.”
• Users were sharing credentials, and then the account was closed when the account owner was terminated for whatever reason.  “How else can I do it?”
• User attempted to log into a system that was decommissioned five years ago.  It *was* a good hostname.
• All of the access points in a large warehouse were physically moved, creating large dead spots.  Of course, no one saw anything or knew how it happened.  Gnomes!
• The cnn.com website was down.  Guess who had to send an outage notification to 16,000 users?
• The application administrator tried something that broke the dev instance, and then proceeded to ‘try’ it in all of the other instances, too.  Because you never know!
• A developer with production access ran a “harmless” SQL statement.  It couldn’t be that – it’s only a SELECT statement!
• The application administrator decided to “secure” the ERP by disabling system accounts.  Since no one could log in, it was really secure.

How about you?  I’d love to hear some other good definitions of “down.”