Introduction

Introduction

VSE/ESA Version 1 introduced Enterprise Systems Architecture (ESA) to VSE users. ESA functionality provided dramatic improvements in throughput and capacity as well as performance. The evolution of ESA support continues in VSE/ESA Version 2.

New functionality included in VSE/ESA Version 2 includes ...

LANRES for VSE
ADSM for VSE
VisualLift for VSE
VSE Workdesk
VSE/ESA Internal Improvements
- Simplified Installation
- New Console Support
- Support for Hardware Data Compression
- New N-Way Turbo Dispatcher
Many Functional Enhancements

With word of these improvements, users rush to order VSE/ESA 2.1 expecting the system to perform better than ever before. Yet, after installation we find slower on-line response time and longer batch run times. Then suddenly we remember, increased functionality probably involves increased overhead.

Increased Overhead

How much increased overhead? In an effort to determine the level of increased overhead in VSE/ESA Version 2.1, I wrote a benchmark program that issued selected SVCs 4000 times. The STCK (STore ClocK) instruction was used to calculate the duration of the SVC request.

The following table shows the execution times of the selected SVCs in micro-seconds. The first percentage shown is the increase from the previous version of VSE/ESA. The second percentage is the increase from VSE/ESA 1.3.

                                               GETFLD          GETFLD         
VSE/ESA        Start I/O       COMREG          MAINTASK        ALET           
Version        SVC 0           SVC 33          SVC 107         SVC 107        

1.3            1217             46             14              109            

2.1 Std        1335  9%         72 71%         37 164%         145 33%        

2.1 TD 1 CPU   1420  6% 16%    143 98% 210%    65  75% 364%    234 61% 114%   

2.1 TD 2 CPU   1566 10% 28%    167 16% 263%    84  29% 500%    281 20% 157%

Table 1: CPU Timings

The SVC 0 (start I/O) test was done by executing a channel program with a single no-operation CCW to SYS000. SYS000 was assigned to a local non-SNA terminal. It should be understood that there can be variations in timing a start I/O operation due to differences in devices, channels and the I/O subsystem used (370, XA or ESA). The test does show that the CPU time used to process a start I/O request increased by 9% simply by converting to VSE/ESA 2.1. Converting to VSE/ESA 2.1 and using Turbo dispatcher added 16% to the cost (CPU) of a start I/O request.

The SVC 33 (COMREG) test is interesting because the SVC 33 routine does nothing except branch to the dispatcher. So this test is a good indication of how much SVC entry and exit linkage overhead increased. Interestingly enough, although this SVC is effectively a no-operation, a large number of applications use it, including many VSE/ESA internal components. Using this SVC is an effective way to waste CPU time.

The SVC 107 (GETFLD FIELD=MAINTASK) test shows the results from a fast reflect SVC that actually returns directly to the issuer without entering the dispatcher's task selection procedure. This test shows that the traditional fast reflect SVC processing is not quite as fast any more.

The SVC 107 (GETFLD FIELD=ALET) test shows the results from a fast reflect SVC that returns to the issuer through the dispatcher's task selection procedure. The results of this test clearly show the extra overhead of the task selection procedure.

What is Causing Overhead?

Changes to the supervisor entry and exit linkage for processing interupts and SVC requests account for a large portion of the CPU overhead increases.

The following table shows the basic supervisor entry processing for an interrupt or SVC request.

        VSE/ESA 1.3                 VSE/ESA 2.1          

Debug                       Dispatcher Interface         

VSE/PT                      Debug                        

VSE Processing ...          Vender Interface             

                            VSE/PT                       

                            VSE Processing ...

Table 2: Interrupt Processing Sequence

Dispatcher The biggest misconception about VSE/ESA 2.1 is the dispatcher. The dispatcher in VSE/ESA 2.1 was totally re-written. The new VSE/ESA 2.1 dispatcher was basically re-written to be a function call processor and has two forms, standard and Turbo. Using the standard dispatcher, processing is functionally equivalent to the VSE/ESA 1.3 dispatcher but it is NOT the same code. The Turbo dispatcher is a new and more complex dispatcher that provides support for multiple CPUs. Under VSE/ESA 2.1 all interrupt processing routines first perform a call to the dispatcher (to enter non-parallel mode) regardless of which dispatcher is actually in use. Having to make these dispatcher function calls, even when running the standard dispatcher, is one source of increased CPU overhead.

Debug The VSE/ESA 2.1 Debug facility was changed from a Monitor Call (MC) to a Branch-And-Link (BAL) interface. This reduced the complexity of the debug facility and decreases the overhead of debug while it is active. However, when the debug facility is disabled (as it is most of the time) the change adds nine instructions to the cost of processing every interrupt and SVC request.

Vender Interface VSE/ESA 2.1 also introduced a new facility called Vendor Exits. These exits allow 3rd party software vendors to establish hooks into VSE/ESA in a standard and controlled fashion. They also introduced a new source of overhead. Every interrupt and SVC request invokes a call to the new vendor interface routines that results in a minimum of 35 instructions being executed. Of course, if an exit is active, much more processing may be required.

Tuning VSE/ESA 2.1 (or How to Minimize the Overhead)

Copy Blocks Over allocate the number of copy blocks defined. This is done with the SYS BUFSIZE= IPL parameter. The default of 1500 appears to be too small for some systems. Use the SIR command to monitor this.
31-bit System Getvis Vastly over allocate the amount of 31-bit system getvis. VSE/ESA 2.1 uses much more 31-bit storage than VSE/ESA 1.3 did and the default is much too small. Add 10-20MB to your SVA GETVIS=(a,b) IPL parameter. Increasing the size of the 31-bit system getvis area should free up storage in the 24-bit system getvis area.
MSECS Try adjusting the MSECS parameter. The default of 976 may be too long for responsive time-slicing on your system. Try 256, 512, 768 or even 1536 instead. Some users have reported that changes to this parameter made a visible difference in throughput.
PRTYIO Set PRTYIO OFF. If you are running OPTI-CACHE, set PRTYIO to the OPTI-CACHE partition. Studies have shown PRTYIO is less effective when running under VSE/ESA 2.1.
Label Area Move the Label Area to a virtual disk. VSE/ESA 2.1 accesses the virtual disk label area as a native data space. This reduces CPU overhead and increases the number of labels that can be stored.
Fetch/Load Processing Use your favorite monitoring tool to monitor phase fetch/load activity. After adjusting the size of the 31-bit system getvis area, start adding phases to the SVA. Add them to the SVA if they are SVA eligible or use MOVE mode support.
Paging Many of the new features of VSE/ESA 1.3 and 2.1 were designed to run on systems with much more memory. Data spaces, virtual disks, data tables, multiple LSR pools, etc., all require more and more memory to be effectively utilized. VSE/ESA 2.1 makes even more use of data spaces (VTAM). Make sure you have enough memory so that VSE is not forced to page (other than on a limited basis). Our testing suggests running VSE/ESA 2.1 on systems with a minimum of 64 to 128MB of memory. The more memory you have, the more you can take advantage of caching and other Data-In-Memory (DIM) storage services.
Data Space Definitions When increasing the size and number of data spaces defined for a system, remember to increase the VSIZE of the system by the same amount of storage. Failure to do this can result in some unusual error messages appearing when the system runs out of virtual storage.
CONSOLE Support Regardless of whether or not you like the new console support, it appears to be a performance problem. Limit the number of terminals being used as consoles and watch out for re-display performance problems.
VSAM Compression Data compression can reduce the size of a VSAM dataset and the number of I/O requests required to process the data. Software data compression requires three times as much CPU as compression done using hardware. Verify that you have compression hardware or sufficient available CPU. If you are currently running at 90% CPU utilization or higher, compression will probably not be very effective. IBM provides a data compression prediction tool (IKQCPRED) to verify that a dataset has good compression characteristics. Use this tool prior to compressing a dataset. IKQCPRED.ZIP is available on IBM's FTP server LSCFTP.KGN.IBM.COM/PUB/VSE. Verify the VSAM backup/restore product that you are using supports VSAM compression BEFORE you compress a dataset. If the backup product does not backup the compression dictionary entries along with the dataset, you may not be able to read the dataset after restoring it.

Tuning POWER

Convert the POWER data file to full track DBLKS. About 47k on 3380s and 57k on 3390s. Try using a DBLK group size of five. This should reduce the number of I/O operations done to the POWER data file.
Use multiple extents for the POWER data file. POWER can process one I/O per extent simultaneously. This allows overlapped I/O operations when POWER processes data.
Define all POWER virtual (dummy) printers as 3800. This allows POWER to process output page-by-page instead of line-by-line. Leave real printer definitions unchanged. This change can reduce run times by 20-60% and CPU time used by 20-40%.

Tuning CICS

Eliminate the use of DFHTEMP auxiliary storage queues in favor of main storage queues. Main storage Temporary Storage (TS) data is stored in 31-bit partition getvis. Many users use recoverable TS queues without any real need. Recovering TS queues also increases CICS startup time.
Eliminate as much FETCH/LOAD processing as possible. Use COBOL II, COBOL/VSE, or other compilers that allow CICS to store programs in 31-bit storage. There may be 3rd party software products available that will assist in this area.
Disable CICS trace and monitors unless needed. On test CICS systems continuously running trace/monitors is acceptable; but on production CICS systems only run trace/monitors when you are having problems. This can account for 5-15% of CICS CPU usage.
Reduce the number of I/O operations CICS performs. Tune VSAM to use all available LSR pools and define the proper number of buffers for running in a 31-bit environment.
Use a software caching product (OPTI-CACHE from Barnard Software, Inc.) to cache the I/O requests done by CICS. Caching CICS I/O requests generally reduces CICS response time by 25-50%.
When running VSE/ESA 2.1 under the Turbo dispatcher with multiple CPUs, use CICS MRO facilities to split the workload into multiple CICS partitions. This allows CICS to take advantage of multiple CPUs.

Tuning VTAM 4.2

Unlike earlier versions of VTAM, all VTAM 4.2 buffers (with the exception of XDBUF buffers) reside in the system getvis. Additionally, all system getvis buffers (with the exception of IOBUF buffers) are allocated out of 31-bit storage. IOBUF buffers are allocated out of 24-bit system getvis.
The default size and number of the buffers in the various buffer pools is not optimal. Use the D NET,BFRUSE command to monitor buffer pool statistics. Try changing the size of the IOBUF buffers to 480 bytes and increase the number of buffers allocated to all pools to the MAX USED value shown in the D NET,BFRUSE output. This should eliminate buffer pool expansion/contraction and improve VTAM performance. Warning: Do not change the size of the buffers in any pool except the IOBUF buffer pool.
Over allocate the size of the dataspace needed by VTAM. The calculations used to determine the exact size are 'fun'. See IBM's VSE/ESA WWW home page and the section on Hints and tips for VTAM 4.2 for details. Of course, you could just allocate 50MB and let it go at that. Over allocating the dataspace just takes up page dataset and does not use real storage unless the space is actually used.
Over allocate the size of the dataspace needed by each VTAM application. Specifing 2-4MB should be sufficient space. VTAM 4.2 uses the dataspace allocated by the application to queue inbound data, store persistent session data, and as work space for data compression.
Useful VTAM 4.2 Statistical Commands GETVIS Fx Getvis information MAP Fx Partition allocations D NET,BFRUSE Buffer information D NET,STATS,TYPE=VTAM VTAM network information D NET,STORUSE,DSPNAME=* Dataspace information D NET,STORUSE,APPL=<applid> Application information D NET,STORUSE,POOL=* Storage pool information
VTAM CPU Accounting CICS and other VTAM application programs may see an increase in CPU time because VTAM's CPU time is accounted to the application partition. This change can make before/after CPU usage reporting difficult for application using VTAM.

Tuning the Turbo Dispatcher

The amount of time your system spends running non-parallel work units is critical to effective use of multiple CPUs. Typical examples of non-parallel work units are most system services and key 0 programs (such as VSE supervisor services, POWER and VTAM). While VSE may be able to support up to 10 CPUs, only one CPU can process non-parallel work units at a time. Vendor products tend to make extensive use of Key 0 programming increasing the amount of non-parallel work units that must be run.

Another problem with non-parallel work is the difference between a task that is 83 bound waiting-to-run parallel and a task that is 83 bound waiting-to-run non-parallel. The task 83 bound waiting-to-run non-parallel is experiencing what I call non-parallel work elongation. Because only one CPU may run non-parallel work at a time, when two (or more) tasks require non-parallel work at the same time, one (or more) tasks must wait. The effective duration of the waiting task's non-parallel work is really the wait time plus the CPU time required to complete the work. This problem can severely limit effective use of multiple CPUs.

The best way to decrease the number of non-parallel work units is to reduce the number of I/O operations and supervisor service requests (SVCs). Sounds easy, right?

Partition Balancing Remember, using the standard dispatcher 10 class C jobs balanced with a static partition resultss in the 10 class C jobs each getting 1/10th of the static partition's time-slice. Using the new Turbo dispatcher, the 10 class C jobs each get the same time-slice as the static partition. This is basically a ten fold increase in CPU time for the dynamic paritions. Your old partition balancing paramters may not be correct when switching to the new Turbo dispatcher.

Excessive Spin Time Applications (generally 3rd party vendor products) that swap the SVC new PSW address in low memory will cause the Turbo dispatcher to spend excessive time 'spinning' on its internal locks attempting to serialize non-parallel time. Users have reported spin times exceeding 10% of each CPU when running software that uses this technique. Swapping the SVC new PSW (or any other PSW) address can be avoided by using the new vendor interface facilities.

The I/O Bottleneck

With the changes to VSE/ESA 2.1's SVC and interupt processing routines, I/O becomes a bigger bottleneck than ever before. Less CPU is used to cache an I/O request than to actually perform the I/O operation. In addition, caching the I/O request requires only one non-parallel work unit. Actually performing the I/O operation requires at least three (and maybe more) non-parallel work units. Caching usually reduces the number of I/O operations by 35-65%, so the non-parallel work unit and CPU savings add up.

Caching

MDCACHE is built-in to VM/ESA 1.2.2 (and higher) and provides good basic track caching features. MDCACHE stores data in VM/ESA's Dynamic Storage Area (DSA) and attempts to balance its use of storage at the expense of system paging. For CMS guest machines, this is an effective method of caching. However, for VSE guest machines, use of MDCACHE does not eliminate the I/O operation. While MDCACHE may reduce a job's run time, CPU time may actually increase. OPTI-CACHE eliminates the I/O operation reducing CPU usage and VM interaction overhead. OPTI-CACHE also understands VSE access methods allowing multi-track read-ahead to improve performance even more.

While MDCACHE is the only effective way to cache VSAM Share Options(4 4) datasets (that is, datasets being updated on multiple VSE systems at the same time) MDCACHE is not available to VSE systems running native or to VM/ESA VSE guest systems using dedicated DASD. Dedicated DASD is very common to VSE systems running as V=R high performance preferred guests. Also, remember MDCACHE will not cache FBA devices that do not start and end with a block number divisible by 8 (page aligned).

The following Tables show the results of a series LIBR TEST commands issued to a 20 cylinder 3380 library. The tests were run on a PC Server 500 System/390 under various versions of VSE/ESA in a V=V VM/ESA 1.2.2 virtual machine. The LIBR TEST commands issued 12810 I/O requests. When OPTI-CACHE or MDCACHE was used no actual I/O was performed, all data was retrieved from cache.

The Tables show the duration and the amount of CPU time as reported by VSE's job accounting and VM's Q TIME output used by the commands. The percentages shown are the calculated increase or decrease in duration and CPU time.

                   No Caching         OPTI-CACHE          MDCACHE            

Duration           135.9 Secs         16.4  (-88%)        25.3  (-81%)       



JA CPU             10.19              13.22                9.65              

Overhead            2.09               0.05                2.05              

Total              12.28              13.27 (+8%)         11.70 (-4%)        



VM CPU             14.69              14.23               13.73              

CP CPU             17.42               0.22               10.27              

Total              32.11              14.45 (-55%)        24.00 (-25%)

Table 3: Results under VSE/ESA 1.3

                   No Caching         OPTI-CACHE          MDCACHE            

Duration           142.5 Secs         16.3  (-88%)        27.3  (-80%)       



JA CPU             12.62              14.16               11.78              

Overhead            2.97               0.12                2.73              

Total              15.59              14.28 (-8%)         14.51 (-6%)        



VM CPU             18.06              14.73               16.05              

CP CPU             17.49               0.24               10.29              

Total              35.55              14.97 (-57%)        26.34 (-25%)

Table 4: Results under VSE/ESA 2.1 Standard Dispatcher

                   No Caching         OPTI-CACHE          MDCACHE            

Duration           148.2 Secs         22.8  (-84%)        35.9  (-77%)       



JA CPU             13.76              14.67               13.32              

Overhead            6.34               2.27                4.97              

Total              20.10              17.07 (-15%)        18.29 (-9%)        



VM CPU             23.41              17.46               19.93              

CP CPU             17.37               0.27               10.24              

Total              40.78              17.73 (-56%)        30.07 (-26%)

Table 5: Results under VSE/ESA 2.1 Turbo w/1 CPU

                   No Caching         OPTI-CACHE          MDCACHE            

Duration           159.7 Secs         24.0  (-84%)        47.2  (-70%)       



JA CPU             14.58              15.27               14.70              

Overhead            7.93               3.07                4.17              

Total              22.51              18.34 (-18%)        18.87 (-16%)       



VM CPU             27.78              18.94               22.43              

CP CPU             17.96               0.34               14.20              

Total              45.74              19.28 (-57%)        36.63 (-19%)

Table 6: Results under VSE/ESA 2.1 Turbo w/2 CPUs

Some obvious conclusions are:

OPTI-CACHE clearly provides the greatest reduction in the commands run-time (duration).
OPTI-CACHE virtually eliminates overhead CPU time. When running under VM or VSE/ESA 2.1 with the Turbo dispatcher, this savings is dramatic.
OPTI-CACHE also reduces the amount of time VSE/ESA 2.1 spends running in non-parallel mode. These results can be seen using the QUERY TD,INTERNAL command (not shown here). This reduction allows the Turbo dispatcher to make more effective use of multiple CPUs.

Converting to VSE/ESA 2.1

Table 7 shows the increase in CPU time used by the LIBR TEST commands as reported by VSE's job accounting and VM's Q TIME output when converting from one version of VSE/ESA to another. Comparison of these times yields some interesting results.

Converting                                        VSE CPU       VM CPU       
From                 To                           Change        Change       

VSE/ESA 1.3          VSE/ESA 2.1 STD              +27%          +10%         

VSE/ESA 1.3          VSE/ESA 2.1 TD (1 CPU)       +63%          +27%         

VSE/ESA 1.3          VSE/ESA 2.1 TD (2 CPU)       +84%          +42%         

VSE/ESA 2.1 STD      VSE/ESA 2.1 TD (1 CPU)       +28%          +14%         

VSE/ESA 2.1 STD      VSE/ESA 2.1 TD (2 CPU)       +44%          +28%         

VSE/ESA 2.1 TD       VSE/ESA 2.1 TD (2 CPU)       +12%          +12%

Table 7: CPU Increase Calculations

Results generated by using LIBR TEST commands probably represent a worst case scenario for calcuating increased overhead when converting from VSE/ESA 1.3 to VSE/ESA 2.1.

CPU time of the LIBR TEST commands showed overhead increasing by 27-84% (as reported by VSE's job accounting). Running a more typical mix of CPU bound and I/O intensive jobs, expect an average increase of 10-25%.

Moral of the Story

VSE/ESA Version 2.1 continues the evolution of VSE into the world of Enterprise Systems Architecture (ESA). The new N-Way Turbo dispatcher combines with new Client/Server features to position VSE/ESA as a super server in our new network centric world.

Yet, as with all new functionality, there is a cost involved. These costs can, however, be controlled and with proper tuning and use of new Data-In-Memory (DIM) technologies (caching) VSE/ESA 2.1 can provide greater capacity and performance than ever before.

All names, products, and services mentioned are the trademarks of their respective organizations.

Back to the BSI Home Page