Control of large scale distributed DAQ/trigger systems in the networked PC era

The HEP community is moving towards larger and more complex experiments, as exemplified by the coming LHC ATLAS and CMS detectors. The triggering and data acquisition hardware has matched this increase in complexity. With availability of cheap high power PC-based computing, many of the data acquisition (DAQ) and higher-level trigger components have been moving to a PC and network based model. These new trigger and DAQ systems are distributed, sometimes involving 100's of computers, and must be fault tolerant. The DZERO Run 2 upgrade DAQ and trigger are a distributed and fault tolerant trigger/DAQ system. A sophisticated trigger/DAQ resource manager has been implemented that can tolerate component failures and even reconfigure on the fly. We present the control system, the monitor system, and the general software design and integration for the DZERO DAQ and Level 3 trigger, including our design plans for controlling the expansion of the farm and DAQ bandwidth with minimal work, as well as the ability for smart monitoring.