I recently had the need to enable deduplication on an existing 5.5TB storage pool residing on a TSM 6.2 server. The storage pool has a FILE (disk-based sequential access) device class, and is used to store virtual volumes from a remote 5.5 server. So, even though it is configured as a primary pool, its data is offsite.
Normally, deduplication occurs when moving data from a primary pool to a copy pool. However, in this situation, storage was very limited, budget was nonexistant, and the data was also backed up to a tape pool on the remote site. So, I chose to go ahead and finally implement dedupe after having run TSM 6.2 for over eight months.
Before proceeding, make sure that you have sufficient disk space dedicated to DB2, both for data files and archived logs. If you run out of space in either, your DB2 instance will pretty much cease functioning until you give it space to work. I ended up needing about 25GB for the data files and 40GB for the primary archive log destination. Also, please be aware that deduplication can and will be very resource-intensive. It can also take a very long time.
Assuming this still sounds like a good idea, the first thing you will need to do is to enable deduplication in the storage pool. And, you’ll also need to tell TSM how many Identify Duplicates worker processes you want to run. You’ll probably want to use a number that is slightly smaller than the number of cores in your system. Here’s the command to enable deduplication and start up 4 worker processes for the storage pool:
tsm: TSM1>update stgpool mypool1 dedup=yes identifyproc=4
If you issue a “q proc” you will see the workers have started and may be “active.” That is, they are looking for duplicate blocks in the storage pool. Here’s what one of my workers looks like; this one is currently “idle.”
tsm: TSM1>q proc Process Process Description Status Number -------- -------------------- ------------------------------------------------- 332 Identify Duplicates Storage pool: MYPOOL1. Volume: NONE. State: idle. State Date/Time: 07/29/2011 03:51:29 PM. Current Physical File(bytes): 0. Total Files Processed: 1010. Total Duplicate Extents Found: 6,288,726. Total Duplicate Bytes Found: 401,236,392,235.
While the workers process the storage pool, it’s probably a good idea to keep tabs on the system and make sure you aren’t running out of resources. I had fun watching the disk subsystem on my Linux system with ‘iostat -x 1’ and running db2top in another xterm.
After everything is finished, you’ll probably do what I did, issue a “q stg mypool1” and then notice that there’s no change in the amount of space used, and reclamation isn’t finding anything. A “q stg mypool f=d” will confirm that Duplicate Data Not Stored is at 0%. Not a good feeling! But what’s actually happening is that TSM is protecting you from a collision or data error causing data loss on a primary storage pool. After making sure that you understand the consequences, you can disable this protection by changing the option DEDUPREQUIRESBACKUP to “no”:
tsm: TSM1>setopt DEDUPREQUIRESBACKUP no
Now you will need to start reclamation on the storage pool, if it isn’t already enabled. If you have a lot of data and/or a lot of I/O, this will take a very long time. You can check how things are going by using the detailed format of a query stgpool:
tsm: TSM1>q stg mypool1 f=d Storage Pool Name: MYPOOL1 Storage Pool Type: Primary Device Class Name: FILECLASS Estimated Capacity: 6,381 G Space Trigger Util: 77.2 Pct Util: 74.8 Pct Migr: 74.8 Pct Logical: 86.9 High Mig Pct: 90 Low Mig Pct: 70 Migration Delay: 0 Migration Continue: Yes Migration Processes: 1 Reclamation Processes: 1 Next Storage Pool: Reclaim Storage Pool: Maximum Size Threshold: No Limit Access: Read/Write Description: Overflow Location: Cache Migrated Files?: Collocate?: Group Reclamation Threshold: 1 Offsite Reclamation Limit: Maximum Scratch Volumes Allowed: 100 Number of Scratch Volumes Used: 3 Delay Period for Volume Reuse: 0 Day(s) Migration in Progress?: No Amount Migrated (MB): 0.00 Elapsed Migration Time (seconds): 0 Reclamation in Progress?: Yes Last Update by (administrator): TKYLE Last Update Date/Time: 08/01/2011 04:31:02 Storage Pool Data Format: Native Copy Storage Pool(s): Active Data Pool(s): Continue Copy on Error?: Yes CRC Data: No Reclamation Type: Threshold Overwrite Data when Deleted: Deduplicate Data?: Yes Processes For Identifying Duplicates: 4 Duplicate Data Not Stored: 836 G (15%) Auto-copy Mode: Client Contains Data Deduplicated by Client?: No
As you can see, we’ve deduped 15% of the data in our storage pool.