How I learned to stop worrying and love compilers

The modern platforms that we want to use to manage our data are far more complex to program efficiently than the machines we used in the past. Every computer we run on is a massively parallel machine with many architectural "surprises" for programmers who are unaware of the way the underlying hardware architecture works. A simple way to think about this is that optimal programs must specify how & where the data should move, not just what computations should be performed and in what order. Architecture-oblivious software that leaves the decisions about data movement to a low-level coherence protocol is becoming much less efficient, relatively speaking. After an extended flirtation with using imperative programming frameworks such as Map-Reduce and NoSQL, many people are returning back to declarative languages like SQL, where the language compiler & runtime are free to make most of the data movement decisions for the programmer. Another way to think about a SQL compiler is that it includes an "algorithm picker" and the runtime includes libraries of useful algorithm implementations (which contain the data movement specifications). This talk will discuss the needs and opportunities for expanding the domain of algorithm-picking languages like SQL. Doing so will require integration with managed-language runtime compilers (e.g. Java or Javascript compilers) that are integrated with the SQL compiler not just to provide efficiency gains during query execution, but also to use managed language runtime profiling to help in algorithm selection as well as assembly-level compilation decisions.