Crowd Source Framework for Indian Digital Heritage Space

The heritage sites of India with their complex scientific-based architectures and history are a great asset for human kind; 39 of these heritage sites are recognized by UNESCO. We need to preserve the information relevant to the sites in terms of history, art, culture, materials, architecture styles, and their role in the socioeconomic growth. These sites are of interest and value to architects, historians, and tourists for various levels of explorations and interpretations. The heritage data is relevant to the cultural heritage sites as it is a valuable asset of the nation. The geographical distribution of these sites, scale, and variance poses challenges in collection of the data, processing for validation, and efficient storage and retrieval. The advances in networking technology which lead to the availability of huge bandwidth and cloud services, high-speed processors, low-cost memories, and efficient AI and Ml algorithms have made it possible to address challenges of big data storage and analytics. We present our work towards building a crowd source platform for data collection, data preprocessing, classification, storage, and query-based retrieval. The platform is used for uploading and retrieval of relevant information about cultural heritage. We also are working towards building 3D reconstruction of the heritage sites and making walk-throughs for real-time experience. For data collection, we followed a two-pronged approach of manual collection as well as crowdsourcing. There are 70 heritage monuments for which data is collected manually by visiting the places. The scale of Indian Cultural Heritage sites poses challenges. Hence, we have proposed a crowdsourcing framework through which the data assimilation is done by contributions open to the public. All the images uploaded by the users are stored in the private cloud server with their respective heritage site information. We have totally collected 105 heritage sites information resulting in 360GB of data. The crowd-sourced data is characterized by redundancy, blur, and occlusion. Proposed framework discards all the redundant images with an 80% similarity value (using Hamming score) in the dataset and removes blurred images. Further, we have applied a GAN-based method for super-resolution which uses perceptual loss function by comparing two images based on high-level representations like content and style discrepancies to recover photo-realistic textures of these images and to enhance the quality as these will be used for 3D reconstruction and walk-throughs. The preprocessed data needs to be classified. The challenges for classification are the volume and variety. The current state-of-the-art artificial intelligence and machine learning algorithms and their proven efficiency in applications in various domains of signal processing, especially vision-based applications, are directions to solve the big data analytic challenges. The data concerned to the heritage sites majorly consists of images and videos, and vision-based processing is computational and time intensive. The availability of high-end processors, GPU and CPU, has made applying AI algorithms for classification. We applied the well-established DNN algorithm for image classification. We have optimized the algorithm by applying transfer learning for enhancing the accuracy and time on pre-trained MobileNet architecture over ImageNet dataset and retrained on Indian Heritage Digital Space (IHDS) dataset. The suggested transfer learning algorithm has the accuracy of 99.25%. We illustrate our crowd source framework using a web and mobile application that includes preprocessing, classification, and information retrieval. We have optimized the proposed model by applying a quantization technique and ported it on edge devices. Using edge devices, users can click a photo of a heritage monument which they visit, and he can upload them using a designed framework. Further, we have used KLE Tech Private Cloud for storing and retrieving the data. The dataset collected from our proposed crowd source framework can be further used for 3D reconstruction, walk-throughs, and digital documentations of the heritage monuments.