Application Advantages of Staring Area Imaging Technology (SAIT) for Microsatellites

ZHONG Xing , BAO Songze

1 Chang Guang Satellite Technology Co., Ltd., Jilin 130012

2 University of Chinese Academy of Sciences, Beijing 100049

3 Changchun Institute of Optics Fine Mechanics and Physics, Chinese Academy of Sciences,Jilin 130033

Abstract: The Staring Area Imaging Technology (SAIT) satellite continuously “images” the target over a certain time range, and can realize continuous imaging and multi-angle imaging of the area of interest. It has the characteristics of flexible imaging parameter setting and fast image preprocessing speed, enabling dynamic target detection and tracking,super-resolution, surface 3D model construction, night-time imaging and many other application tasks. Based on the technical characteristics of the SAIT satellite, this paper analyzes the challenges in satellite development and data processing, focuses on the quasi-realtime application of SAIT satellite data, and looks at the development trend of the SAIT satellite.

Key words: Staring Area Imaging Technology (SAIT), Jilin-1 video satellite, remote sensing application

1 INTRODUCTION

After more than 30 years of rapid development, China's space technology has made tremendous progress, creating a series of Earth observation and remote sensing satellite systems with applications for resources, meteorology, ocean, environment and national defense. Significant innovations have been made in remote sensing satellite development, constellation construction, and ground data processing. At the same time,spatial resolution, time resolution, and data quality have been greatly improved, which provides better information services and intelligence support for modern agriculture, disaster prevention and mitigation, resource and environment, and public safety.

The video micro-satellite emerged in recent years and is a new type of earth observation satellite. Compared with traditional Earth observation satellites, it has a staring imaging capability, as shown in Figure1. In such staring imaging mode, the satellite can continuously observe a region and obtain more dynamic information about the target than a traditional satellite. The integration time of the detector is no longer limited by the satellite motion speed ratio and the ground sample distance (GSD). The frame interval is limited only by the detector's ability, data acquisition speed and data transmission capability, hence the satellite can acquire dynamic video with high time resolution.

Figure 1 Staring imaging mode of video micro-satellite

In recent years, countries around the world have developed SAIT satellites. Indonesia's LAPAN-TUBSAT[1]weighing only 55 kg, launched in January 2007, is located in a sun-synchronous orbit, with an orbital height of 635 km and orbital inclination of 97.9°. The main payload is a high-resolution color camera. The spatial resolution is 5 m and the imaging width is 3.5 km. South Africa launched SumbandilaSat in September 2009[2]. The SKYSAT series of video satellites launched by SKYBOX Corporation of the United States located at a height of 600 km in a sun-synchronous orbit has a mass of 91 kg. The optical imaging system is manufactured from tantalum carbide with 2 m multi-spectral resolution and 0.9 m panchromatic resolution, and the imaging swath width is up to 8 km. The video imaging mode has a panchromatic resolution of 1.1m and a width up to 2 km × 1.1 km. The frame rate is 30 frames and can capture video up to 90 s[3]. Domestically, the National University of Defense Technology launched the Tiantuo-2 video verification satellite in 2014. This is China's first video satellite with a GSD of 5 m.China's first all-industry-chain satellite company, Chang Guang Satellite Technology Co. , Ltd. (CGSTL) has launched a number of Jilin-1 series of video satellites since 2015. The satellite model diagram and technical indicators are shown in Figure 1 and Table 1. The first generation of video satellites (Jilin-1sp 01 - sp 02)were launched in October 2015. The second-generation (Jilin-1sp 03) has a spatial resolution of 0.92 m and was launched in January 2017. The latest generation (Jilin-1sp 04 - sp 08)launched in November 2017 and January 2018 are compatible with pushbroom, staring imaging, multi-point imaging, spatial imaging as well as a variety of other imaging modes such as night-time imaging, which has reached an advanced world level.

Figure 2 Jilin-1 video satellite series

2 ADVANTAGES FOR SAIT SATELLITES

Firstly, a SAIT satellite can image ground targets and space targets, and has multiple imaging modes such as staring/cruising/tracking (as shown in Figure 3), which can perform multiple shooting tasks. The cruising mode is based on the staring mode,adding a certain angular velocity to the yaw and pitch axes of the satellite, so that the ground track of the satellite optical axis moves as expected and increases the coverage area. It is similar to the lens shaking in photography, but for cruising mode, the moving range and speed of the satellite is controllable. Secondly,the SAIT satellite can extend the target information dimension by multi-angle imaging and shooting time series images. Thirdly, as the satellite is staring at the target during imaging, which has a long exposure time, different exposure parameters can be flexibly selected, so that a high dynamic range image can be obtained. Finally, since the SAIT satellite take a single shoot for a region, all pixels in the image share the same geometric model,so fast geometric processing is possible.

Table 1 Jilin-1 video satellite series technical indicators

Figure 3 Different frames of video acquired by Jilin-1 satellites

3 CHALLENGES FOR SAIT SATELLITES

In order to make the optical axis of the video satellite camera always aim at the same target scene for uninterrupted imaging, the attitude control system needs to adjust the attitude of the satellite in real time to compensate for image displacement,so the imaging process has high dynamics. The main challenges of SAIT technology comes from two aspects, satellite development and image processing. For the former, high dynamic and high precision attitude measuring and determination, with high precision tracking modeling and pointing control must be implemented. In addition, structural optimization of the satellite and micro-vibration isolation of flexible components is required.In image processing, geometric, radiometric calibration and uniformity processing for large area-array detector with multiple imaging angles are required.

3.1 Challenges for SAIT Satellites Development

During stare imaging, the satellite needs to be continuously maneuvered. Highly precise point maneuver control is mainly based on a stare tracking algorithm, which is realized by nonlinear control of attitude. During the maneuvering process, the satellite independently calculates and plans the tracking path, and controls the satellite's triaxial motion to make the satellite maneuver as expected. To that end, SAIT needs a nonlinear continuous change of attitude, which makes the exported moment of the execution components, and hence induces a disturbance to the high resolution imaging. In addition, for high dynamic attitude determination the geometric quality of image products mainly depends on the attitude determination. Maneuvering during SAIT will also decrease the measuring accuracy. High dynamic sensors with high accuracy at the same time are needed.

3.2 Challenges for SAIT Satellite Image Processing

For satellite data processing, it is necessary to ensure that each frame of the satellite video has a uniform radiance, and at the same time, the stationary target in the video has the same position between each frame. However, during the staring imaging, the motion and attitude control errors of the satellite platform will cause a spatial orientation change and jitter of the camera's optical axis. The geometric difference between different frames and the radiation difference when viewed from multiple angles are large. Geometric and radiometric calibration are required during data preprocessing.

3.2.1 Radiometric calibration

The uniformity of the radiometric response in video data is caused by various reasons such as relative displacement between different frames, non-uniform illumination with the optical system, defect of detector, dust on detector or optical surfaces, abnormal pixel presence, etc. Therefore, it is necessary to perform pixel-level relative radiometric calibration for the satellite video, which is very important for SAIT images.

After the satellite launches, multiple images can be captured at specific angles for a scene of uniform brightness, such as the ocean, desert and cloud, enabling field-free relative radiometric calibration. As shown in the equation1, the relative radiometric calibration coefficient for each pixel can be obtained by dividing the average grayscale of all pixels in the image by the time-averaged grayscale of each pixel.

Figure 4 Relative radiometric calibration effect

3.2.2 Geometric calibration

According to the satellite imaging mode, combined with satellite orbit, attitudes, time and other parameters, a high-precision geometric positioning model is constructed to realize image and ground coordinate conversion. Based on analyzing the influence of orbital error, attitude error, time error and elements of interior orientation for the geometric positioning accuracy, high-precision positioning is determined by solving the RPC model. By taking the advantages of Jilin-1 video satellite multi-point imaging mode to obtain geometric calibration field video data. Then select the appropriate image to register with the high precision DOM of the calibration field and combine the DEM data to obtain the 3D coordinates for the registration point, and use the geometric positioning model to establish the relationship between the image point and the ground control point, and solve the bias matrix. Elements of the exterior orientation calibration flow chart are shown in Figure 5.

Figure 5 Elements of exterior orientation calibration flow chart

Satellite video stabilization is the basis for the video application, in order to eliminate the inconsistencies of the same ground target in different frames. The video captured by the US Skysat satellite is processed with image stabilization enabling an accuracy of 0.6 pixel[3]. For the Jilin-1 video data, using the image stabilization method based on image matching and geometric orientation calibration, the registration accuracy after image stabilization is better than 0.5 pixels enabling continuous observation of the ground target for up to 300 seconds, typically providing a satellite video product with duration of 30 s,60 s, 90 s and 120 s. The flow chart is shown in Figure 6. First using distortion of the principal point of the photograph, the principal distance of photo, original orbit and attitude parameter one can calculate a rigorous geometric imaging model and the RPC coefficients for each frame. Combined with DEM,the conversion relationship between image coordinates and object coordinates is established. High-precision registration between the main frame and the sub-frames is then performed to achieve inter-frame motion estimation and motion compen-sation. According to the above result, orientation processing is performed, and the grayscale of each sub-frame is resampled according to the main frame. Finally each frame can be framed to obtain a stabilized video.

Figure 6 Satellite video stabilization flow chart

4 SAIT APPLICATION

The main advantage of satellite video data compared to general satellite data is that it can achieve long-term, multi-angle imaging of an area. Continuous imaging enables dynamic monitoring of hotspots, redundant information between video frames provides the basis for super-resolution reconstruction and HDR reconstruction; multi-angle imaging can be used for 3D reconstruction of the surface; more flexible imaging parameters setting (exposure time, gain etc.) enables night light imaging and spatial target imaging.

4.1 Moving Target Detection and Tracking

In the staring imaging mode, the video satellite can achieve continuous shooting of the region of interest, which can be used for detecting and continuously tracking of moving targets[4], and traffic flow density estimation[5]. The classic video target tracking methods are mainly divided as follow: algorithm based on moving target detection[6], algorithm based on contrast analysis,algorithm based on kernel method[7], feature matching based algorithm[8]and Bayesian framework-based algorithm[9]. The algorithm based on background subtraction has better precision and a faster processing speed. It is one of the main application methods and consists of foreground target extraction and motion information parameterization. Firstly, the Gaussian mixture model is used to process the video data frame by frame, and the foreground mask is roughly extracted for the initial result.The noise in the mask image is suppressed by the bilateral filtering, and the geometric, velocity, color and other feature constraints are applied to the extraction results to eliminate false positives. Then detection and continuous locking of the moving target are realized, and finally one can obtain the motion information such as the position, speed and direction of the target.

Target tracking method based on spatiotemporal context[10]belongs to the tracking method under Bayesian framework.First mark the target location in the first frame, a spatial context model is established based on the spatial relationship between the target and the background in the local area, and the model is used to update the spatiotemporal context model of the next frame. Combining the spatiotemporal context information, the image is convoluted, a confidence map is obtained, and its maximum likelihood probability position is calculated as the optimal target position. The processing flow is shown in Figure 8.

4.2 Image Super-resolution Using Satellite Video

Super-resolution reconstruction technology is a process of generating high-resolution images by combining complementary information of multi-frame low-resolution images of the same scene, increasing the number of pixels in the image, thereby providing more detailed information for subsequent applications and improving spatial resolution of the remote sensing image. In the staring imaging mode, it is possible to continuously observe changes in the field of view and obtain multi-frame images of the same area, so that a high-resolution image can be reconstructed by using complementary information between multiframe images.

Figure 7 Flow chart of moving target detection and tracking algorithm based on background subtraction

Figure 8 Moving object tracking method based on spatiotemporal context

At present, the image super-resolution method is mainly composed of non-uniform interpolation, the iterative back-projection approach (IBP)[11], adaptive filtering approach (AFA)[12],projection onto convex sets (POCS)[13], maximum likelihood(ML), maximum a posterior (MAP)[14], regularization method[15],Shift and Add[16], sparse expression based method[17]and Markov random field based method (MRF)[18]. Especially with the rapid development of artificial intelligence in remote sensing image processing in recent years, the deep learning method has also been widely used in image super-resolution[19]. Xiao et al.[20]constructed a five-layer end-to-end neural network to achieve super-resolution with the Jilin-1 satellite video image; Luo et al.[21]use the Gaofen 2 satellite imagery data as a training set to train a model specifically for video satellite image super-resolution, and achieved good results with the Jinlin-1 satellite image data. A multi-frame Jilin-1 video satellite image super-resolution example based on the improved POCS method is shown in Figure 9.

Figure 9 Example for multi-frame image super resolution

Figure 10 Example for HDR reconstruction

4.3 Satellite Image HDR Reconstruction

In the staring imaging mode,setting of the imaging parameters is flexible. Multiple images of different exposure times at the same place can be obtained, and a high dynamic range (HDR) remote sensing image of the place can be synthesized by registration and tone mapping of the images. At the same time, the details of the over-bright and over-dark areas in the images are preserved, as shown in Figure 10.

4.4 Digital Surface Model Reconstruction

Figure 11 Example of digital surface model reconstruction

The SAIT satellite has agile maneuverability and can acquire images from multiple angles. Therefore, it is possible to perform 3D reconstruction using stereoscopically oriented satellite video images. Firstly, the satellite video is subjected to absolute orientation processing and image stabilization. Multi-view stereo matching is used to generate dense matching results of corresponding points. The digital surface model is formed by the intersection of images, and the RMS relative elevation error of the DSM is less than 3 m, which is better than the traditional three linear mapping satellites.

4.5 Night-time Imaging Applications

Night-time images in visible bands can reveal human activities, by observing artificial lights on Earth. Traditional night-time imaging, with the resolution of 500 m - 1 km, was used for macrography research such as poverty and Gink coefficient[22],GDP[23], electric power[24], carbon emission estimation[25], urbanization process monitoring[26], humanitarian disaster impact assessment[27], eco-environmental assessment and public health[28]. Associate Professor LI Xi of Wuhan University obtained the historical night-time images of Syria from March 2011 to January 2017 through the intercalibration of DMSP/OLS and VIIRS, and assessed the impact of war on Syria's social economy[29].

Thanks to the flexible imaging parameter selection in the staring imaging mode, a large range of high-resolution nighttime images can be obtained by multi-point imaging of large areas during a single satellite transit time as shown in Figure 12.High-resolution night-time imaging applications such as: fishery monitoring, surveillance of illegal activities, urban planning, commercial site selection, emergency rescue, national defense intelligence and other fields. In addition, high resolution night-time images can show structural details of city lights, which can be used for vacancy rate measurements of communities. Researchers at Zhejiang University use the Jilin-1 satellite high-resolution night light image to classify artificial lights, distinguish high-pressure sodium lamps and LED lamps. The accuracy is better than 83%, with a Kappa coefficient 0.67[30].

Figure 12 Night-time image of Hong Kong taken by Jilin-1 satellite

4.6 Space Object Imaging

The SAIT satellite can realize deep space observation through attitude adjustment. The SAIT satellite image the space target mainly using two imaging modes: Inertial Space Pointing and Inertial Space Scanning, as shown in Figure 13. In the former imaging mode, the satellite aims at the specified target in the inertial space, and the satellite triaxial angular velocity is 0.The latter refers to controlling the satellites triaxial angular velocity in the inertial space according to the tracking needs, and realizing the scanning of the space target. Detailed mission planning should also be carried out to avoid the influence of solar stray light during space target imaging.

Thanks to the easy adjustment of exposure time and gain,we are able to obtain clear space target images. These parameters depend on illumination, size and distance of the target,direction of the sun and can be calculated by model. Figure 14 shows images of Saturn and Jupiter taken by Jilin-1 SAIT satellite.

Figure 13 Imaging modes of SAIT satellite for space target imaging

Figure 14 Images of Saturn and Jupiter taken by Jilin-1 SAIT satellites at different exposure times and gains

Figure 15 ISS image with sea background taken by Jilin-1 satellite

In order to research observation methods of manmade objects' during reentry, in cooperation with the Chinese National Observatory, 6 Jilin-1 video satellites were used to analyze the orbital intersecting frequency with ISS (International Space Station) and orbit recursion was done using ISS's TLE parameters,so ISS images with different backgrounds have been acquired successfully.

5 SATI SATELLITE DEVELOPMENT TREND

The development trend of the SAIT satellite is summarized as follows. First, the SAIT satellite can be compatible with the pushbroom imaging mode, for small-area imaging using the staring imaging mode, the RGB video is captured, and the multi-spectral pushbroom imaging mode is used for a large area. For this purpose, the satellite uses a separate focal plane device for the pushbroom imaging mode.The compatible pushbroom mode is mainly realized by incident light splitting. The incident light is divided into two parts, one for the area array detector imaging and the other for the linear array pushbroom detector imaging. The latest generation of Jilin-1 video satellites (Jilin-1 sp 04 - 08) are compatible with the pushbroom imaging mode, and are equipped with the camera shown in Figure 16. The second is to achieve a higher video imaging resolution. Video imaging is mainly for finding moving targets. Most of these targets are artificial objects, which are small in size and need more image pixels to describe, so a higher resolution is necessary. The third point is that the main value of dynamic video data is high timeliness. With the rapid development of artificial intelligence technology, the fast on-orbit image processing for satellite imagery becomes possible, and the total amount of downlink data can be greatly compressed,further shortening the time required for real-time applications.The last point is that video data as a new type of remote sensing information source has attracted the attention of more and more researchers, and new application fields are expected to develop.

Figure 16 Camera structure diagram of Jilin-1 sp 04 - 08

6 CONCLUSIONS

The SAIT satellite can realize dynamic real-time monitoring of hotspots and targets through video imaging within a certain time range. Since the launch of the first meter-level video satellite in 2013, video micro-satellites have ushered in a period of rapid development. It has become a hot spot in the development of remote sensing satellites. This paper analyzes the main challenges faced by video satellite development and video data processing, introduces the current application of SAIT satellites and looks forward to the development trend of the video micro-satellite. A summary of SAIT applications are as below.

1) Effective preprocessing and fast delivery

The processing of area images is much easier and faster than push-broom images, then, this makes the products' delivery very fast, enabling quasi-realtime applications.

2) Abundant of data products types

Video, moving target information, night-time images and DSM products can be produced, bringing a lot of new applications.

3) Broad application fields

Defense & security, disaster monitoring and rescue, and economy research, etc.