Full Scale-Aware Balanced High-Resolution Network for Multi-Person Pose Estimation

Shaohua Li,Haixiang Zhang,Hanjie Ma,Jie Feng and Mingfeng Jiang

School of Computer Science and Technology,Zhejiang Sci-Tech University,Hangzhou,310018,China

ABSTRACT Scale variation is a major challenge in multi-person pose estimation.In scenes where persons are present at various distances,models tend to perform better on larger-scale persons,while the performance for smaller-scale persons often falls short of expectations.Therefore,effectively balancing the persons of different scales poses a significant challenge.So this paper proposes a new multi-person pose estimation model called FSA Net to improve the model’s performance in complex scenes.Our model utilizes High-Resolution Network(HRNet)as the backbone and feeds the outputs of the last stage’s four branches into the DCB module.The dilated convolution-based(DCB)module employs a parallel structure that incorporates dilated convolutions with different rates to expand the receptive field of each branch.Subsequently,the attention operation-based (AOB) module performs attention operations at both branch and channel levels to enhance high-frequency features and reduce the influence of noise.Finally,predictions are made using the heatmap representation.The model can recognize images with diverse scales and more complex semantic information.Experimental results demonstrate that FSA Net achieves competitive results on the MSCOCO and MPII datasets,validating the effectiveness of our proposed approach.

KEYWORDS Computer vision;high-resolution network;human pose estimation

1 Introduction

Human pose estimation is a hot topic in computer vision,which involves simulating the human body’s joints by predicting key points.In recent years,it has been extensively applied in various fields,including but not limited to autonomous driving [1–3],motion tracking [4–6],and intelligent surveillance[7–9].

In human body key point detection,two mainstream methods currently prevail: top-down and bottom-up.The former involves a multi-stage approach,typically detecting a single-person image using an object detector,followed by single-person pose estimation.The latter is a single-stage method primarily used for multi-person pose estimation,where the standard approach is to directly predict all key points in the image and subsequently perform classification and aggregation.The bottom-up method is more challenging,as larger-scale person features are typically more prominent and more accessible for the network to learn during training.In contrast,smaller-scale person features are inherently more ambiguous and susceptible to surrounding noise.As such,a key focus of research is developing methods to enable the network to perceive better targets of varying scales,which is also the primary focus of our study.

In multi-person pose estimation tasks,the design of the backbone network plays a crucial role due to the prevalent use of single-stage structures in mainstream multi-person pose estimation models.The depth,downsampling rate,and receptive field size of the network significantly impact its performance.Although HRNet[10]is a popular choice as the backbone,its larger network depth enhances its ability to handle large targets but potentially neglects small-scale targets.Moreover,an excessive number of downsampling layers can impair the network’s capability to detect small-scale targets,thus affecting HRNet’s performance and making it difficult to balance the detection of targets of varying scales.

In summary,we propose a bottom-up approach with HRNet as the backbone network,coupled with our meticulously designed post-processing modules,which we refer to as FSA Net.Fig.1 illustrates the rough structure of FSA Net,which encompasses the DCB and AOB modules within the full-scale aware module (FSAM).In contrast to the previous practice of merging the four branches of the backbone network into one branch using a simple upsampling and addition method,our method addresses the semantic differences between them and considers the importance of each feature.Specifically,we input the outputs of the four branches into the DCB module,which comprises four parallel branches with dilated convolutions having different dilation rates of 9,7,5,and 3,respectively,to boost the receptive fields of various branches.After processing with the DCB module,the four feature maps of the branches are cross-stitched to form new feature maps of the four branches.Furthermore,attention operations are performed on the feature maps of each branch from both the branch and channel dimensions to enhance essential features and suppress redundant ones.Finally,we employ the heatmap representation to predict the final key points.The main contributions of this paper are summarized as follows:

? We present FSA Net,a high-resolution human pose estimation network that achieves scaleaware balance by paying attention to the performance of targets at different scales and extracting more complex semantic information during training.In contrast to other networks that tend to ignore the performance of small-scale targets,FSA Net can achieve a full-scale perception balance.

? We propose the DCB module,which covers targets at all scales through parallel structures and controls the receptive fields of different branches through dilated convolutions,thereby better perceiving targets at different scales.

? We propose the AOB module,which performs attention operations on feature maps of different branches after concatenating them.Unlike other models that simply add feature maps of different branches without considering their semantic differences,the AOB module can enhance the fusion ability of multi-scale features by strengthening important features and suppressing noise,thus improving the detection ability of small-scale targets.

? We have evaluated our proposed method on the widely used COCO Validation and testdev datasets and attained remarkable results.Moreover,the feasibility of our approach was validated by conducting ablation experiments.

2 Related Studies

In single-person pose estimation,each person is detected separately,and there is no scale variation problem,making it much simpler than multi-person pose estimation.This chapter focuses on the development history of multi-person pose estimation.This paper first provides an overview of the overall development of multi-person pose estimation,followed by an exploration of the use of highresolution networks in this task.Finally,the paper introduces attention mechanisms in multi-person pose estimation.

Figure 1:Simplified overall framework diagram of FSA Net

2.1 Mutil-Person Pose Estimation

Multi-person pose estimation involves predicting all the key points in an image and grouping them.Hourglass[11]proposed a cascading pyramid network that performs pose estimation by regressing key points directly.In Deepcut[12],the authors used Faster RCNN[13]for human detection,followed by the integer linear program method for pose estimation.Deepercut[14]employed ResNet[15]for body part extraction and Image Conditioned Pairwise Terms for key point prediction,improving accuracy and speed.Personlab[16]detected all the key points in an image using a box-free approach and then used greedy decoding to cluster the key points by combining the predicted offsets.OpenPose[17]used VGG[18]as the backbone network and proposed Part Affinity Fields to connect the key points.PifPaf[19]outperformed previous algorithms in low-resolution and heavily occluded scenes,and Pif predicts body parts,while Paf represents the relationships between them.

2.2 High-Resolution Network

Experimental results have shown that high-resolution feature maps are crucial for achieving superior detection performance in complex tasks such as human pose estimation and semantic segmentation.The proposal of HRNet has caused a huge response,as the authors employed a parallel connection of multiple sub-networks with varying resolutions,maintaining a high-resolution branch and performing repeated multi-scale fusion to enable each high-resolution feature map to receive information from other parallel branches of different resolutions repeatedly,thus generating rich high-resolution representations.Subsequently,the authors proposed HigherHRNet [20],which adopted a bottom-up approach using transpose convolution to expand the size of the last layer output feature map to half of the original image size,leading to improved performance.BalanceHRNet[21]introduced a balanced high-resolution module BHRM was proposed,along with a branch attention module for the branch fusion method to capture the importance of different branches.Multi-Stage HRNet[22]used a bottom-up approach to parallel multiple levels of HRNet and employed cross-stage feature fusion to further refine key point prediction.Wang et al.[23]enhanced the network’s ability to capture global context information using switchable convolution operations.NHRNet[24]optimized HRNet by cutting high-level features in low-resolution branches and adding attention mechanisms in residual blocks.Khan et al.[25] proposed a network to address the challenges of multi-scale and global context feature loss and applied it to flood disaster response in high-resolution images.

2.3 Attention Mechanism

The attention mechanism has become a cornerstone in many computer vision tasks including image classification,object detection,instance segmentation,and pose estimation,owing to its remarkable simplicity and effectiveness.It enables networks to selectively focus on relevant information while ignoring irrelevant noise,thus resulting in improved performance.In recent years,several attentionbased models have been proposed to enhance the feature fusion capability of the HRNet.For instance,SaMr-Net[26]introduced dilated convolutions and attention modules to promote multi-scale feature fusion.HR-ARNet[27]added a refinement attention module to the end of the HRNet for improved feature fusion.Improved HRNet [28] added a dual attention mechanism to parallel sub-networks to minimize interference caused by unrelated information.Meanwhile,Zhang et al.[29] enhanced the feature expression ability of each branch of HRNet by adding attention modules to different branches.X-HRNet [30] employed a one-dimensional spatial self-attention module to generate two one-dimensional heatmaps by projecting the two-dimensional heatmap onto the horizontal and vertical axes for keypoint prediction.

3 Proposed Method

Figure 2:Overall framework diagram of SSA Net

3.1 Dilated Convolution Based Module

Multi-person images are fed into our network,and after feature extraction by the backbone network,the feature maps of four branches are outputted.These feature maps are then individually passed through the DCB module,consisting of four parallel branches with different dilation rates of dilated convolution,as illustrated in Fig.3.Specifically,each branch’s output feature maps undergo bottleneck processing similar to that of ResNet [15],consisting of 1 × 1 convolution,dilated convolution with different dilation rates,and 1 × 1 convolution again.The final bottleneck uses a residual operation instead of an activation function.

In HRNet,limited receptive field and excessive downsampling severely affect the detection ability of small objects.Therefore,we propose the DCB module,which accurately controls the receptive field of each branch through a parallel structure.For the branch with the highest resolution,we use dilated convolution with a dilation rate of 9 to expand the receptive field of each pixel.For the branch with the lowest resolution,we limit its receptive field to a smaller range to better perceive small-scale targets,enhancing the perception ability of targets of various scales.

Figure 3:Schematic diagram of the DCB module

3.2 Attention Operation-Based Module

After undergoing processing by the DCB module,the network can enhance the perception of objects at all scales.However,due to their relatively blurred features,small-scale persons are easily affected by noise.To address this issue,we introduce the AOB module,which performs attention operations on each branch and channel to emphasize important features and suppress noise.This results in significant improvements in detection performance,particularly for small-scale objects.Attention operations on each branch also highlight the semantic differences between branches,enabling the network to better attend to targets of different scales.

Specifically,as shown in Fig.4.The AOB module takes the output feature maps of the DCB module,denoted asw1,w2,w3,w4,as input.These feature maps are first subjected to cross-splicing to generatebranchw1,branchw2,branchw3,branchw4,as shown in Eq.(1):

Whereidenotes theithbranch,andjdenotes the number of used branches.Then the attention operation is performed onbranch wifrom branch and channel dimensions.Firstly,each branch is processed by adaptive average pooling and adaptive max pooling,and then sum the results to obtain the attention score,after which the obtained attention score is applied tobranch wito readjust the weights,and the calculation process is shown in Eq.(2):

WhereAAPstands for adaptive average pooling operation,andAMPstands for adaptive max pooling operation.Then feature fusion is performed on the feature maps after the attention operation to regenerate the four branchs feature maps,and the calculation process is shown in Eq.(3):

whereidenotes theithbranch,andjdenotes the number of branches to be used.

Figure 4:Schematic diagram of the AOB module

3.3 Decoder Module

In the last stage of HRNet,only the branch with the highest resolution is selected as the final output,while the contributions of other branches are ignored.This approach may hurt the performance of targets at other scales.To address this issue,in our decoding stage,we fuse the outputs of all branches by upsampling,generating a heatmap the same size as the original image and with a 1/4 resolution.This heatmap integrates features of all resolutions and can perceive the key points in different scale persons.The predicted coordinates are then decoded,and the L2 loss function is utilized to calculate the loss by comparing the estimated heatmap with the ground truth heatmap.During the loss calculation in the training process,since we have(the number of key points)channels,we compute the loss by calculating the L2 norm of the corresponding key points’indices in each channel.As the labels are generated as heatmaps representing a small region,the loss calculation measures the distance(L2 distance)between the predicted key points and the key points in the labeled region.

4 Experiments

4.1 Experimental Details

In the FSA Net,we adopted a bottom-up approach for multi-person pose estimation.We set the initial learning rate to 0.001 during the experiments and used the Adam optimizer to obtain gradients.The total number of epochs was set to 300,and we trained and evaluated the network on the widely used COCO dataset.Due to the use of heatmap-based methods,a significant amount of memory was required.The experimental hardware and software environment is presented in Table 1.

Table 1:The hardware and software configuration environment for the experiments

4.2 Datasets and Evaluation Metrics

The dataset employed in this experiment is the widely used MSCOCO dataset in human pose estimation.MSCOCO is a large-scale,multi-purpose dataset developed by Microsoft that plays an essential role in mainstream visual tasks.It contains over 200 k images,with 250 k annotated instances of human key points.The MSCOCO training set consists of 118 k images.In contrast,the test set comprises two subsets: COCO Validation,including 5 k images mainly used for simple testing and ablation experiments,and COCO test-dev,including 20 k images mainly used for online testing and fair comparison with state-of-the-art models.The primary evaluation metrics in the COCO dataset are average precision (AP) and average recall (AR),which are calculated according to the following formulas:

wherepis the number of detected human instances,andtis the threshold to refine the evaluation index.

Whentis taken as 0.5 and 0.75,it is noted asAP50andAP75,and OKS is object key point similarity,which is calculated as follows:

whereidenotes theithkey point,didenotes the Euclidean distance between the true value and the predicted value,sis the area of human instance,Viis whether the recognition instance is visible or not,and δ is the regularization parameter of the key point.When 322

In addition,we also constructed the Tiny Validation dataset,which is a subset of the COCO Validation dataset.In this dataset,we labeled images that contain persons with an area smaller than 802as“images with small individuals.”After filtering through the 5000 images,we found that only 361 images met this criterion.We curated these images to form a new dataset.The purpose of using this dataset is to evaluate the network’s detection performance specifically for small individuals.

4.3 Quantitative Experimental Results

In the study of full-scale aware balance,detecting small-scale person poses is the most significant challenge.Traditional pose estimation networks tend to focus more on medium to large-scale persons and may overlook the performance of small-scale persons.Therefore,we first evaluated FSANet’s detection capability for small persons on the Tiny Validation dataset.Table 2 shows that HigherHRNet[20] exhibits significant performance degradation on the Tiny Validation dataset,with an average loss of 20% in AP.This confirms the inference mentioned earlier.Additionally,it can be seen that FSA Net shows a substantial performance improvement,and FSANet-W48 achieves an AP of 60.3%,surpassing HigherHRNet-W48 and other bottom-up models.This demonstrates the significant enhancement of our proposed FSA Net in detecting small-scale persons.

Table 2:Comparison with mainstream bottom-up models on the tiny validation dataset

In Table 3,we present the results of testing FSA Net on the widely used COCO Validation dataset,where we achieved significant improvements across multiple metrics.Specifically,when utilizing HRNet-W32 as the backbone network,FSA Net outperformed HigherHRNet by 1.5% and achieved even greater improvement compared to networks such as the method [15].Furthermore,when the backbone network was HRNet-W48,the network’s AP reached 71.2,which was 1.3% higher than HigherHRNet.Notably,we observed varying degrees of improvement in bothAPMandAPL,underscoring the enhanced performance of our method across different scales.

Table 3:Comparison with mainstream bottom-up models on the COCO Validation dataset,bold is the best result in each column,HG is the hourglass,DLA is deep layer aggregation,W32 is HRNet-W32 and W48 is HRNet-W48

In Table 4,we further evaluated our model on the COCO test-dev dataset to strengthen its persuasiveness.Our experiments show that when HRNetW48 was used as the backbone network,our method outperformed other mainstream bottom-up models,achieving an AP of 70.8.It is worth noting that HigherHRNet and our work share similar ideas of enlarging the feature map area to better perceive small-scale objects by upsampling the last output of HRNet through transpose convolution.However,FSA Net focuses on the performance of each branch,expands the receptive field through parallel-structured dilated convolution,and eliminates noise through branch attention operations.Our experimental results demonstrate the clear superiority of our proposed approach.In addition,since we process all four branches,the parameter count and computational complexity(GFLOPs)of FSANet are slightly higher than HrHRNet but significantly better than other models such as Hourglass.

4.4 Qualitative Experimental Results

To visually demonstrate the effectiveness of our method,we have provided experimental results in Figs.5 and 6.Specifically,we conducted tests on the COCO Validation dataset with HRNetW48 as the backbone network.As shown in Fig.5.Our model can easily handle scenarios with rich scale information.Moreover,as illustrated in Fig.6.The left is the original image,the middle is HigherHRNet,and the right is FSA Net.It can be seen that in complex scenes,such as self-occlusion of people,foreground occlusion,the size of targets is too small,and complex semantic information,our model predicts more accurately and reasonably than other models.

Figure 5:Visualization of the results of FSA Net on the COCO Validation dataset

Figure 6:Qualitative comparison of FSA Net with other state-of-the-art bottom-up models on the COCO Validation dataset

4.5 Ablation Experiments

Our method builds upon the model HRNet by incorporating the DCB and AOB modules.To evaluate the impact of each module,we performed extensive ablation experiments on the COCO Validation dataset in Table 5,adopting the identical training strategy as in previous work.The results reveal that the DCB module boosts AP by 2.5% compared to the baseline model,while the AOB module achieves a significant increase of 3.1% in AP.Both modules have contributed to the overall performance improvement,with the AOB module exhibiting a greater contribution.When the two modules are jointly applied,the overall AP is further boosted by 4.2%,attesting to the remarkable effectiveness of our proposed approach.

Table 5:Ablation experiments on COCO validation,Ba is Baseline

5 Conclusion

This study introduces a novel multi-person high-resolution pose estimation network with fullscale perception balance.Our approach improves upon the HRNet baseline by incorporating DCB and AOB modules.The DCB module expands the receptive field of each branch’s results,while the AOB module performs attention operations to enhance feature representation ability.We evaluate our method on popular datasets and outperform the baseline model and similar networks.Additionally,we visualize the experimental results to provide a more intuitive understanding of the performance improvements of our FSA Net.In our initial attempts,we drew inspiration from[33]and used dilation rates of 4,3,2,and 1 for the dilated convolutions in the DCB module.However,the experimental results were not satisfactory.Upon analysis,we realized that precise localization of key points is crucial in pose estimation.Therefore,we needed to use larger dilation rates to achieve a larger receptive field.Ultimately,we achieved success with dilation rates of 9,7,5,and 3.Although FSA Net has achieved impressive results,it still faces some challenges.As shown in Table 2,FSA Net can be considered to have achieved a scale-awareness balance compared to other bottom-up networks,but it has not fully achieved a scale-aware balance.There is still significant room for improvement in the performance on small-scale targets,which will be the focus of our future research.

Acknowledgement:Thanks for the full guidance and supervision from Associate Professor Haixiang Zhang,and the financial support from Professor Mingfeng Jiang,and we would like to express our gratitude to the esteemed editors and reviewers for their invaluable contributions to this article.

Funding Statement:This work is supported in part by the National Natural Science Foundation of China 61672466 62011530130,Joint Fund of Zhejiang Provincial Natural Science Foundation LSZ19F010001.

Author Contributions:The authors confirm contribution to the paper as follows: study conception and design:Shaohua Li;data collection:Shaohua Li;analysis and interpretation of results:Haixiang Zhang,Hanjie Ma,Jie Feng;draft manuscript preparation:Mingfeng Jiang.All authors reviewed the results and approved the final version of the manuscript.

Availability of Data and Materials:This experiment uses the public COCO dataset,which is directly available,and since there are subsequent algorithm improvements,the code can be obtained from the corresponding author under reasonable circumstances.

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present research.