Powerful Data Analysis and Composition with the UNIX-Shell.

In addition to a wide range of commercially available data processing tools for data analysis and knowledge discovery, there are a bundle of Unix-shell scripting and text processing tools practically available on every computer. This paper reports on some of these data processing tools and presents how they can be used together to manipulate and transform data and also to perform some sort of analysis like aggregation, etc. Beside the free availability, these tools have the advantage that they can be used immediately, without prior transformation and loading the data into the target system. Another important point is, that they are typically stream-based and thus, huge amounts of data can be processed without running out of main-memory.