The Software Package for Astronomical Reductions with KMOS: SPARK
R. DaviesAlex Agudo BerbelE. WiezorrekMichele CirasuoloN. M. Förster SchreiberY. JungB. MuschielokThomas OttS. RamsayJ. SchlichterR. M. SharplesMichael Wegner
87
Citation
32
Reference
10
Related Paper
Citation Trend
Abstract:
KMOS is a multi-object near-infrared integral field spectrometer with 24 deployable cryogenic pick-off arms. Inevitably, data processing is a complex task that requires careful calibration and quality control. In this paper we describe all the steps involved in producing science-quality data products from the raw observations. In particular, we focus on the following issues: (i) the calibration scheme which produces maps of the spatial and spectral locations of all illuminated pixels on the detectors; (ii) our concept of minimising the number of interpolations, to the limiting case of a single reconstruction that simultaneously uses raw data from multiple exposures; (iii) a comparison of the various interpolation methods implemented, and an assessment of the performance of true 3D interpolation schemes; (iv) the way in which instrumental flexure is measured and compensated. We finish by presenting some examples of data processed using the pipeline.Keywords:
Interpolation
SPARK (programming language)
Limiting
Data Processing
Holistic energy management in the shipping industry involves reliable data collection, systematic processing and smart analysis. The era of digitisation allows sensor technology to be used on-board vessels, converting different forms of signal into a digital format that can be exported conveniently for further processing. Appropriate sensor selection is important to ensure continuous data collection when vessels sail through harsh conditions. However, without proper processing, this leads to the collection of big data sets but without resulting useful intelligence that benefits the industry. The adoption of digital and computer technology, allows the next phase of fast data processing. This contributes to the growing area of big data analysis, which is now a problem for many technological sectors, including the maritime industry. Enormous databases are often stored without clear goals or suitable uses. Processing of data requires engineering knowledge to ensure suitable filters are applied to raw data. This systematic processing of data leads to transparency in real time data display and contributes to predictive analysis. In addition, the generation of series of raw data when coupled with other external data such as weather information provides a rich database that reflects the true scenario of the vessel. Subsequent processing will then provide improved decision making tools for optimal operations. These advances open the door for different market analyses and the generation of new knowledge. This paper highlights the crucial steps needed and the challenges of sensor installation to obtain accurate data, followed by pre and post processing of data to generate knowledge. With this, big data can now provide information and reveal hidden patterns and trends regarding vessel operations, machinery diagnostics and energy efficient fleet management. A case study was carried out on a tug boat that operates in the North Sea, firstly to demonstrate confidence in the raw data collected and secondly to demonstrate the systematic filtration, aggregation and display of useful information.
Data Processing
Data Analysis
Cite
Citations (0)
SPARK (programming language)
Cite
Citations (45)
A considerable amount of computing power is needed for Gaia data processing during the mission. A pan European system of six data centres are working together to perform different parts of the processing and combine the results. Data processing estimates suggest around 1020 FLOP total processing is required. Data will be transferred daily around Europe and with a final raw data volume approaching 100 TB. With these needs in mind the centres are already gearing up for Gaia. We present the status and plans of the Gaia Data Processing Centres.
Data Processing
Data processing system
Cite
Citations (0)
Data acquisition and pre-processing are important steps in any traffic data collection, analysis or simulation project. In this paper the authors present the results of a case study they have done on traffic simulation for a stretch of the E17 highway near Antwerp, Belgium. They will especially focus on the data collection and processing part. The objective of the work presented here is to illustrate how traffic data can be collected and processed in such a way that it can be used for modeling and prediction purposes. The data collection and processing process consists of three steps: data acquisition, interfacing, and pre-processing. During this process a central database with raw sensor data is created. In general the raw sensor data will contain errors or missing values due to sensor failures, data link errors, biases, and other measurement errors. Therefore, the authors introduce a pre-processing step in order to deal with data that is known to be corrupt or for which the value is temporarily not available (many simulation packages require data records that are and contain no gaps). After the three steps have been carried out, they have a database of clean and complete traffic data that can be used for further analysis or that can be used in other applications such as highway traffic simulation or traffic forecasts. (A) For the covering abstract see ITRD E106484.
Interfacing
Data Processing
Traffic count
Cite
Citations (7)
Data Processing
Mode (computer interface)
Cite
Citations (0)
AbstractAbstractData analysis and processing is playing an important role because of the large amount of data generated through various sources of big data. It is an important component in big data-based applications. Data qualities are the main concern in the data acquisition, transformation and data pre-processing under big data applications. Data pre-processing is required because of inconsistent, noisy and incomplete data generation in big data applications. Data analysis basically encompasses different methods and a function applicable to data's to detect characteristics such as data type, size, format, patterns and so on. Based on data format, it's easy to identify data qualities for further use in various applications. Moreover, data analysis and processing includes various steps such as data qualities identification, statistical analysis of data, defining modeling, and hypothetical testing of model and result from analysis. Raw data is unused data and required analysis, filtering, and processing in any system. This paper deals with the analysis and processing aspects of raw data and cleaned data in big data applications. This paper also deals with data cleaning and its implementation concepts.Subject Classification: 68U35Keywords: Data cleaningBig dataData visualizationData acquisitionData filtering
Data Processing
Data transformation
Data Analysis
Data pre-processing
Identification
Data type
Cite
Citations (17)
A new three-electrode light source, the hollow spark, has been developed by combining an open vacuum spark and a sliding spark. Its design and qualities are described and compared to other spark sources.
SPARK (programming language)
Spark gap
Spark discharge
Cite
Citations (5)
In the previous chapters, the fundamental components of Spark such as Spark Core, Spark SQL, Spark Streaming, Structured Streaming, and Spark MLlib have been covered. In this chapter, we discuss one simple real-time use case to understand how we can use Spark in real-time scenarios.
SPARK (programming language)
Spark chamber
Electric spark
Cite
Citations (0)
일반적으로 C언어는 고신뢰도를 요구하는 소프트웨어에 적합하지 않다고 알려졌음에도 불구하고 적지 않은 수의 안전성이 중요시되는(safety-critical) 시스템들이 C언어로 구현되었거나 C언어를 기반으로 개발되고 있다. 본 연구의 목적은 C언어의 안전한 subset을 정의하고 이 subset 언어로 작성된 프로그램을 SPARK Ada로 변환하여 SPARK의 분석 도구들을 사용해 프로그램의 안전성을 분석하는 데 있다. SPARK은 Ada의 안전한 subset으로 고신뢰도를 요하는 시스템을 구현하는데 성공적으로 사용되어 왔다. SPARK으로 변환된 C 프로그램은 SPARK 수준의 안전성을 갖게 되며 SPARK의 분석 도구인 Examiner를 통해 프로그램의 정확성 검증 등의 분석을 할 수 있다. 본 연구에서는 엘리베이터 컨트롤러 사례 연구를 통해 정의한 subset이 현실적인 시스템을 구현하기에 부족하지 않음을 발견하였으며, SPARK Ada로의 변환을 자동화해주는 변환기를 구현하였다.
SPARK (programming language)
Cite
Citations (0)
In this paper, based on the characteristics of the fixed data, this paper compares the various mathematical statistics analysis methods and chooses the improved Grabs criterion to analyze the data, and through the analysis of the data processing, the data processing method is not suitable. It is proved that this method can be applied to the processing of fixed raw data. This paper provides a reference for reasonably determining the effective quota analysis data.
Data Processing
Data Analysis
Cite
Citations (0)