A Data Processing Framework for Cloud Environment Based on Hadoop and Grid Middleware

Owing to performance improvement of mobile devices, number of mobile applications and their variety has increased exponentially in recent years. However, many of these mobile applications are not executed alone and need server-side Internet services which require computing functions such as processing, networking, and storage. The server-side Internet services are usually provided using computing resources at Cloud data center because mobile applications are rapidly increasing in number and they tend to be more and more complex in nature. In addition, the conventional data managing framework, like 3-tier architecture, face additional problems such as heterogeneous external data to import and the vast amount of data to process. In this paper, we propose a data processing framework for mobile applications based on OGSA-DAI for heterogeneous external data import and MapReduce for large data processing. We designed and implemented a data connector based on OGSA-DAI middleware which can access and integrate heterogeneous data in a distributed environment, supporting various data management functions. And then we deployed a data processing framework (we call this data connector) into a Cloud system for mobile applications. We also used MapReduce programming model for data connector. Finally, we conducted various experiments and showed that our proposed framework can be used to access heterogeneous external data and to process large data with negligible or no system overhead.