Hardware and networks for Gaia data processing
0
Citation
0
Reference
10
Related Paper
Abstract:
A considerable amount of computing power is needed for Gaia data processing during the mission. A pan European system of six data centres are working together to perform different parts of the processing and combine the results. Data processing estimates suggest around 1020 FLOP total processing is required. Data will be transferred daily around Europe and with a final raw data volume approaching 100 TB. With these needs in mind the centres are already gearing up for Gaia. We present the status and plans of the Gaia Data Processing Centres.Keywords:
Data Processing
Data processing system
A considerable amount of computing power is needed for Gaia data processing during the mission. A pan European system of six data centres are working together to perform different parts of the processing and combine the results. Data processing estimates suggest around 1020 FLOP total processing is required. Data will be transferred daily around Europe and with a final raw data volume approaching 100 TB. With these needs in mind the centres are already gearing up for Gaia. We present the status and plans of the Gaia Data Processing Centres.
Data Processing
Data processing system
Cite
Citations (0)
Data acquisition and pre-processing are important steps in any traffic data collection, analysis or simulation project. In this paper the authors present the results of a case study they have done on traffic simulation for a stretch of the E17 highway near Antwerp, Belgium. They will especially focus on the data collection and processing part. The objective of the work presented here is to illustrate how traffic data can be collected and processed in such a way that it can be used for modeling and prediction purposes. The data collection and processing process consists of three steps: data acquisition, interfacing, and pre-processing. During this process a central database with raw sensor data is created. In general the raw sensor data will contain errors or missing values due to sensor failures, data link errors, biases, and other measurement errors. Therefore, the authors introduce a pre-processing step in order to deal with data that is known to be corrupt or for which the value is temporarily not available (many simulation packages require data records that are and contain no gaps). After the three steps have been carried out, they have a database of clean and complete traffic data that can be used for further analysis or that can be used in other applications such as highway traffic simulation or traffic forecasts. (A) For the covering abstract see ITRD E106484.
Interfacing
Data Processing
Traffic count
Cite
Citations (7)
We designed a data-processing system of Young's modulus experiment taking Mathematica as the main tool and developed a data comparison program to mark right or wrong in student's data-processing paper.We solved the problem of complex calculation,data processing and paper-checking and improved greatly the efficiency and quality of data-processing in the Young's modulus experiment,as well as of the teaching.
Data Processing
Data processing system
Experimental data
Materials processing
Cite
Citations (0)
Data Processing
Data processing system
Cite
Citations (0)
AbstractAbstractData analysis and processing is playing an important role because of the large amount of data generated through various sources of big data. It is an important component in big data-based applications. Data qualities are the main concern in the data acquisition, transformation and data pre-processing under big data applications. Data pre-processing is required because of inconsistent, noisy and incomplete data generation in big data applications. Data analysis basically encompasses different methods and a function applicable to data's to detect characteristics such as data type, size, format, patterns and so on. Based on data format, it's easy to identify data qualities for further use in various applications. Moreover, data analysis and processing includes various steps such as data qualities identification, statistical analysis of data, defining modeling, and hypothetical testing of model and result from analysis. Raw data is unused data and required analysis, filtering, and processing in any system. This paper deals with the analysis and processing aspects of raw data and cleaned data in big data applications. This paper also deals with data cleaning and its implementation concepts.Subject Classification: 68U35Keywords: Data cleaningBig dataData visualizationData acquisitionData filtering
Data Processing
Data transformation
Data Analysis
Data pre-processing
Identification
Data type
Cite
Citations (17)
Traditional computer data processing mode cannot operate efficiently, and is inefficient in processing computer data in big data era. Therefore, the research on computer data processing model in big data era is carried out. This paper plans the overall architecture of computer data processing mode. On the basis of the overall architecture, big data processing mode is divided into three modes: offline batch data processing, query data processing and real-time data processing. Detailed design the core functions of the three modes, implement computer data processing in big data environment, complete the design of computer data processing mode in big data era. The experiment data proved its efficiency, the designed computer data processing mode in big data era is more efficient than the traditional data processing mode, The processing efficiency of this mode increased by 45%.
Data Processing
Data processing system
Mode (computer interface)
Batch processing
Cite
Citations (0)
Data Processing
Data processing system
Electronic data processing
Automatic Data Processing
Multidimensional signal processing
Cite
Citations (0)
The surge in the number of earth observation satellites being launched worldwide is placing significant pressure on the satellite-direct ground receiving stations that are responsible for systematic data acquisition, processing, archiving, and dissemination of earth observation data. Growth in the number of satellite sensors has a bearing on the ground segment payload data processing systems due to the complexity, volume, and variety of the data emanating from the different sensors. In this paper, we have aimed to present a generic, multi-mission, modularized payload data processing system that we are implementing to optimize satellite data processing from historical and current sensors, directly received at the South African National Space Agency's (SANSA) ground receiving station. We have presented the architectural framework for the multi-mission processing system, which is comprised of five processing modules, i.e., the data ingestion module, a radiometric and geometric processing module, atmospheric correction and Analysis Ready Data (ARD) module, Value Added Products (VAPS) module, and lastly, a packaging and delivery module. Our results indicate that the open architecture, multi-mission processing system, when implemented, eliminated the bottlenecks linked with proprietary mono-mission systems. The customizable architecture enabled us to optimize our processing in line with our hardware capacities, and that resulted in significant gains in large-scale image processing efficiencies. The modularized, multi-mission data processing enabled seamless end-to-end image processing, as demonstrated by the capability of the multi-mission system to execute geometric and radiometric corrections to the extent of making it analysis-ready. The processing workflows were highly scalable and enabled us to generate higher-level thematic information products from the ingestion of raw data.
Payload (computing)
Data processing system
Data Processing
Earth observation
Cite
Citations (11)
During this time, the data processing system at the Police Jaluko criminals have used a computer but not as optimal as possible, so it takes a long time to search criminal data on police Jaluko. The purpose of this thesis was to study the data processing system at the police criminal Jaluko, identify weaknesses that may be in it and designed a new application program using Visual Basic 6.0. With the data processing system is expected to change the old system to the new system. By using Visual Basic 6.0, changes to the data processing system is intended facilitate processing of criminal data on police Jaluko, so that later the data can be processed properly and quickly. The results of this study demonstrate using data processing applications properly designed, data validation has been able to do since pengimputan, process data and reports presented during the data entered is true.
Data Processing
Data processing system
Cite
Citations (0)
In this paper, based on the characteristics of the fixed data, this paper compares the various mathematical statistics analysis methods and chooses the improved Grabs criterion to analyze the data, and through the analysis of the data processing, the data processing method is not suitable. It is proved that this method can be applied to the processing of fixed raw data. This paper provides a reference for reasonably determining the effective quota analysis data.
Data Processing
Data Analysis
Cite
Citations (0)