Open Source File System Selection for Remote Sensing Data Operational Storage and Processing

Significant increase in the number of Earth remote sensing devices requires improvement of ground-based means of automatic stream data processing, which would allow the implementation of a mass remote sensing service in real time. The experience of choosing a freely distributed parallel file system working on the Linux operating system for solving problems of operational storage and processing of remote sensing data (RSD) is presented. RSD processing is organized according to the technology which excludes multiple copying of data between the processing steps and the delivery of the program code to the data. The processed data array is a collection of both target data files with the length of up to tens of gigabytes, and a set of short files associated with them with service data of tens of kilobytes in length. Technology Hadoop is used to implement the complex. To improve data processing performance, it was decided to replace the standard HDFS file system with a more efficient one. As a result of analysis and testing, the parallel OrangeFS file system was chosen. Instead of the previously used AVRO format, HDF5 is used as the internal technological format of data exchange. The issues of optimizing the settings of the data access mechanisms stack are considered, including the mode with dynamic selection of the I/O scheduler. The use of a more advanced parallel file system made it possible to increase the processing speed up to 40% relative to the standard Hadoop file system. © 2020, Springer Nature Switzerland AG.