Evolution from Shark to Spark SQL: Preliminary Analysis and Qualitative Evaluation

Spark is a general distributed framework with the abstraction called resilient distributed datasets (RDD). Database analysis is one of the main kinds of workloads supported on Spark. The SQL component on Spark has evolved from Shark to Spark SQL, while the core components of Spark also have evolved a lot comparing with the original version. We analyzed on which aspects Spark have made efforts to support many workloads efficiently and whether the changes make the support for SQL achieve better performance.