Porting the ALMA Correlator Data Processor from hard real-time to plain Linux

The ALMA correlator back-end consists of a cluster of 16 computing nodes and a master collector/packager node. The mission of the cluster is to process time domain lags into auto-correlations and complex visibilities, integrate them for some configurable amount of time and package them into a workable data product. Computers in the cluster are organized such that individual workloads per node are kept within achievable levels for different observing modes and antennas in the array. Over the course of an observation the master node transmits enough state information to each involved computing node to specify exactly how to process each set of lags received from the correlator. For that distributed mechanism to work, it is necessary to unequivocally identify each individual lag set arriving at each computing node. The original approach was based on a custom hardware interface to each node in the cluster plus a realtime version of the Linux Operating System. A modification recently introduced in the ALMA correlator consists of tagging each lag set with a time stamp before delivering them to the cluster. The time stamp identifies a precise 16- millisecond window during which that specific data set was streamed to the computing cluster. From the time stamp value a node is able to identify a centroid (in absolute time units), base-lines, and correlator mode during that hardware integration. That is, enough information to let the digital signal processing pipeline in each node to process time domain lags into frequency domain auto-correlations per antenna and visibilities per base-line. The scheme also means that a good degree of concurrency can be achieved in each node by having individual CPU cores process individual lag sets at the same time, thus rendering enough processing power to cope with a maximum 1 GiB/sec output from the correlator. The present paper describes how we time stamp lag sets within the correlator hardware, the implications to their on-line processing in software and the benefits that this extension has brought in terms of software maintainability and overall system simplifications.