IEEE Transactions on Circuits and Systems I Regular Papers · 2023 · 24 citations · 44 references
EngineeringComputer EngineeringComputer ScienceComputer VisionField-programmable Gate ArraysReconfigurable ComputingAccelerator ArchitectureDeep LearningField-programmable Gate Array ArchitectureFpga ArchitectureHardware AccelerationReconfigurable AcceleratorsAcceleratorsComputer Vision AccelerationFeature AlignmentField-programmable Gate Array AccelerationField-programmable Gate Array AcceleratorsResource-constrained DevicesSuper-resolutionVideo Super-resolution
Deformable convolutional networks (DCNs) have shown outstanding potential in video super-resolution with their powerful inter-frame feature alignment. However, deploying DCNs on resource-limited devices is challenging, due to their high computational complexity and irregular memory accesses. In this work, an algorithm-hardware co-optimization framework is proposed to accelerate the DCNs on field-programmable gate array (FPGA). Firstly, at the algorithm level, an anchor-based lightweight deformable network (ALDNet) is proposed to extract spatio-temporal information from the aligned features, boosting the visual effects with low model complexity. Secondly, to reduce intensive multiplications, an innovative shift-based deformable 3D convolution is developed using low-cost bit shifts and additions, maintaining comparable reconstruction quality. Thirdly, at the hardware level, a dedicated critical processing core, together with a block-level interleaving storage scheme, is presented to avoid dynamic and irregular memory accesses caused by the deformable convolutions. Finally, an overall architecture is designed to accelerate the ALDNet and implemented on an Intel Stratix 10GX platform. Experimental results demonstrate that the proposed design can provide significantly better visual perception than other FPGA-based super-resolution implementations. Meanwhile, compared with the prior hardware accelerators, our design can achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.75\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.63\times $ </tex-math></inline-formula> improvements in terms of throughput and energy efficiency, respectively.
44
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh et al. · IEEE Transactions on Image Processing · 2004 · 56K citations
Deformable Convolutional Networks
Jifeng Dai, Haozhi Qi, Yuwen Xiong et al. · 2017 · 6.9K citations