Enabling deduplication on a primary storage pool in TSM 6.2

Posted by on Aug 2, 2011 in Infrastructure, Storage, TSM | 2 Comments

I recently had the need to enable deduplication on an existing 5.5TB storage pool residing on a TSM 6.2 server.  The storage pool has a FILE (disk-based sequential access) device class, and is used to store virtual volumes from a remote 5.5 server.   So, even though it is configured as a primary pool, its data is offsite.

Normally, deduplication occurs when moving data from a primary pool to a copy pool.  However, in this situation, storage was very limited, budget was nonexistant, and the data was also backed up to a tape pool on the remote site.  So, I chose to go ahead and finally implement dedupe after having run TSM 6.2 for over eight months.

Before proceeding, make sure that you have sufficient disk space dedicated to DB2, both for data files and archived logs.  If you run out of space in either, your DB2 instance will pretty much cease functioning until you give it space to work.  I ended up needing about 25GB for the data files and 40GB for the primary archive log destination.  Also, please be aware that deduplication can and will be very resource-intensive.  It can also take a very long time.

Assuming this still sounds like a good idea, the first thing you will need to do is to enable deduplication in the storage pool.  And, you’ll also need to tell TSM how many Identify Duplicates worker processes you want to run.  You’ll probably want to use a number that is slightly smaller than the number of cores in your system.  Here’s the command to enable deduplication and start up 4 worker processes for the storage pool:

tsm: TSM1>update stgpool mypool1 dedup=yes identifyproc=4

If you issue a “q proc” you will see the workers have started and may be “active.”  That is, they are looking for duplicate blocks in the storage pool.  Here’s what one of my workers looks like; this one is currently “idle.”

tsm: TSM1>q proc

 Process     Process Description      Status
  Number
--------     --------------------     -------------------------------------------------
     332     Identify Duplicates      Storage pool: MYPOOL1. Volume: NONE. State:
                                       idle. State Date/Time: 07/29/2011 03:51:29 PM.
                                       Current Physical File(bytes): 0. Total Files
                                       Processed: 1010. Total Duplicate Extents Found:
                                       6,288,726. Total Duplicate Bytes Found:
                                       401,236,392,235.

While the workers process the storage pool, it’s probably a good idea to keep tabs on the system and make sure you aren’t running out of resources.  I had fun watching the disk subsystem on my Linux system with ‘iostat -x 1’ and running db2top in another xterm.

After everything is finished, you’ll probably do what I did, issue a “q stg mypool1” and then notice that there’s no change in the amount of space used, and reclamation isn’t finding anything.  A “q stg mypool f=d” will confirm that Duplicate Data Not Stored is at 0%.  Not a good feeling!  But what’s actually happening is that TSM is protecting you from a collision or data error causing data loss on a primary storage pool.  After making sure that you understand the consequences, you can disable this protection by changing the option DEDUPREQUIRESBACKUP to “no”:

tsm: TSM1>setopt DEDUPREQUIRESBACKUP no

Now you will need to start reclamation on the storage pool, if it isn’t already enabled.  If you have a lot of data and/or a lot of I/O, this will take a very long time.  You can check how things are going by using the detailed format of a query stgpool:

tsm: TSM1>q stg mypool1 f=d

                    Storage Pool Name: MYPOOL1
                    Storage Pool Type: Primary
                    Device Class Name: FILECLASS
                   Estimated Capacity: 6,381 G
                   Space Trigger Util: 77.2
                             Pct Util: 74.8
                             Pct Migr: 74.8
                          Pct Logical: 86.9
                         High Mig Pct: 90
                          Low Mig Pct: 70
                      Migration Delay: 0
                   Migration Continue: Yes
                  Migration Processes: 1
                Reclamation Processes: 1
                    Next Storage Pool:
                 Reclaim Storage Pool:
               Maximum Size Threshold: No Limit
                               Access: Read/Write
                          Description:
                    Overflow Location:
                Cache Migrated Files?:
                           Collocate?: Group
                Reclamation Threshold: 1
            Offsite Reclamation Limit:
      Maximum Scratch Volumes Allowed: 100
       Number of Scratch Volumes Used: 3
        Delay Period for Volume Reuse: 0 Day(s)
               Migration in Progress?: No
                 Amount Migrated (MB): 0.00
     Elapsed Migration Time (seconds): 0
             Reclamation in Progress?: Yes
       Last Update by (administrator): TKYLE
                Last Update Date/Time: 08/01/2011 04:31:02
             Storage Pool Data Format: Native
                 Copy Storage Pool(s):
                  Active Data Pool(s):
              Continue Copy on Error?: Yes
                             CRC Data: No
                     Reclamation Type: Threshold
          Overwrite Data when Deleted:
                    Deduplicate Data?: Yes
 Processes For Identifying Duplicates: 4
            Duplicate Data Not Stored: 836 G (15%)
                       Auto-copy Mode: Client
Contains Data Deduplicated by Client?: No

As you can see, we’ve deduped 15% of the data in our storage pool.