Pipes and Threads: Performance Testing of Advanced Scalability Features in SAS ® v9

SAS v9 provides many new features of interest to users with large data sets. Many of the commonly used procedures are multi-treaded. This allows users to harness the full power of multi-processor architectures to enhance the efficiency of some tasks. In addition, SAS/Connect allows users to spawn multiple independent processes to execute custom parallel processing solutions. SAS/Connect also enables pipeline parallelism, i.e., piping of records from one DATA/PROC step to another between processes. This potentially relieves some of the I/O burden of the total SAS job, an important consideration when dealing with large data sets. We examined the real world speed of these three features using two different multiprocessor system architectures, a Sun Microsystems Sun Fire 3800 running Unix and an IBM 9672/Y36 running MVS. We investigated a typical usage pattern with a simple DATA step followed by a PROC TABULATE. Test data were from a simulated insurance claims file with test runs ranging from 10 million to approximately 322 million records. Our tests suggest that for an architecture based on traditional synchronous processing using the traditional method of the DATA step writing intermediate output to disk, SAS v9 is not substantially different than SAS v8. However, the enhanced features in SAS v9, piping and multi-threading, yield significantly shorter total run times on more powerful architectures. These performance gains were evident at all file sizes tested (1.2 to 38.4 GB). We noted also that the built-in multi-threading in PROC TABULATE performed better than a manual parallel processing solution. This is an important finding since it suggests that shorter run times can be achieved directly with the base product without the added expense of an additional complicated programming effort.