高级搜索

多级跳线连接的深度残差网络超分辨率重建

赵小强 宋昭漾

引用本文: 赵小强, 宋昭漾. 多级跳线连接的深度残差网络超分辨率重建[J]. 电子与信息学报, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu
Citation:  Xiaoqiang ZHAO, Zhaoyang SONG. Super-Resolution Reconstruction of Deep Residual Network with Multi-Level Skip Connections[J]. Journal of Electronics and Information Technology, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu

多级跳线连接的深度残差网络超分辨率重建

    作者简介: 赵小强: 男,1969年生,博士生导师,教授,主要研究方向为故障诊断,图像处理,生产调度等;
    宋昭漾: 男,1995年生,硕士生,研究方向为图像处理
    通讯作者: 赵小强,xqzhao@lut.cn
  • 基金项目: 国家科学自然基金(61763029, 61873116)

摘要: 由于快速的卷积神经网络超分辨率重建算法(FSRCNN)卷积层数少、相邻卷积层的特征信息之间缺乏关联性,因此难以提取到图像深层信息导致图像超分辨率重建效果不佳。针对此问题,该文提出多级跳线连接的深度残差网络超分辨率重建方法。首先,该方法设计了多级跳线连接的残差块,在多级跳线连接的残差块基础上构造了多级跳线连接的深度残差网络,解决相邻卷积层的特性信息缺乏关联性的问题;然后,使用随机梯度下降法(SGD)以可调节的学习率策略对多级跳线连接的深度残差网络进行训练,得到该网络超分辨率重建模型;最后,将低分辨率图像输入到多级跳线连接的深度残差网络超分辨率重建模型中,通过多级跳线连接的残差块得到预测的残差特征值,再将残差图像和低分辨率图像组合在一起转化为高分辨率图像。该文方法与bicubic, A+, SRCNN, FSRCNN和ESPCN算法在Set5和Set14测试集上进行了对比测试,在视觉效果和评价指标数值上该方法都优于其它对比算法。

English

    1. [1]

      THORNTON M W, ATKINSON P M, and HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473–491. doi: 10.1080/01431160500207088

    2. [2]

      PELED S and YESHURUN Y. Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29–35. doi: 10.1002/1522-2594(200101)45

    3. [3]

      ZOU W W W and YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327–340. doi: 10.1109/TIP.2011.2162423

    4. [4]

      LU Huimin, LI Yujie, CHEN Min, et al. Brain intelligence: Go beyond artificial intelligence[J]. Mobile Networks and Applications, 2018, 23(2): 368–375. doi: 10.1007/s11036-017-0932-8

    5. [5]

      KOCH M. Artificial intelligence is becoming natural[J]. Cell, 2018, 173(3): 531–533. doi: 10.1016/j.cell.2018.04.007

    6. [6]

      LEO M, MEDIONI G, TRIVEDI M, et al. Computer vision for assistive technologies[J]. Computer Vision and Image Understanding, 2017, 154: 1–15. doi: 10.1016/j.cviu.2016.09.001

    7. [7]

      ZHU Hong, TANG Xinming, XIE Junfeng, et al. Spatio-temporal super-resolution reconstruction of remote-sensing images based on adaptive multi-scale detail enhancement[J]. Sensors, 2018, 18(2): 498. doi: 10.3390/s18020498

    8. [8]

      SHI Jun, LIU Qingping, WANG Chaofeng, et al. Super-resolution reconstruction of MR image with a novel residual learning network algorithm[J]. Physics in Medicine & Biology, 2018, 63(8): 085011. doi: 10.1088/1361-6560/aab9e9

    9. [9]

      SU Heng, ZHOU Jie, and ZHANG Zhihao. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202–1213. doi: 10.3724/SP.J.1004.2013.01202

    10. [10]

      GRIBBON K T and BAILEY D G. A novel approach to real-time bilinear interpolation[C]. The 2nd IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 2004: 126–131. doi: 10.1109/DELTA.2004.10055.

    11. [11]

      FRITSCH F N and CARLSON R E. Monotone piecewise cubic interpolation[J]. SIAM Journal on Numerical Analysis, 1980, 17(2): 238–246. doi: 10.1137/0717021

    12. [12]

      STARK H and OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715

    13. [13]

      PATANAVIJIT V and JITAPUNKUL S. An iterative super-resolution reconstruction of image sequences using fast affine block-based registration with BTV regularization[C]. Proceedings of 2006 IEEE Asia Pacific Conference on Circuits and Systems, Singapore, 2006: 1717–1720. doi: 10.1109/APCCAS.2006.342128.

    14. [14]

      ZHOU Fei, YANG Wenming, and LIAO Qingmin. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312–3318. doi: 10.1109/TIP.2012.2189576

    15. [15]

      LIN Zhouchen and SHUM H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83–97. doi: 10.1109/TPAMI.2004.1261081

    16. [16]

      YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55–75. doi: 10.1109/MCI.2018.2840738

    17. [17]

      KERMANY D S, GOLDBAUM M, CAI Wenjia, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning[J]. Cell, 2018, 172(5): 1122–1131.e9. doi: 10.1016/j.cell.2018.02.010

    18. [18]

      DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281

    19. [19]

      DONG Chao, LOY C C, and TANG Xiaoou. Accelerating the super-resolution convolutional neural network[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 391–407. doi: 10.1007/978-3-319-46475-6_25.

    20. [20]

      KIM J, LEE J K, and LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1637–1645. doi: 10.1109/CVPR.2016.181.

    21. [21]

      SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874–1883. doi: 10.1109/CVPR.2016.207.

    22. [22]

      LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4681–4690. doi: 10.1109/CVPR.2017.19.

    23. [23]

      HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.

    24. [24]

      KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1646–1654. doi: 10.1109/CVPR.2016.182.

    25. [25]

      YUAN Fei, HUANG Lianfen, and YAO Yan. An improved PSNR algorithm for objective video quality evaluation[C]. 2007 Chinese Control Conference, Hunan, China, 2007: 376–380. doi: 10.1109/CHICC.2006.4347144.

    26. [26]

      WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861

    27. [27]

      GAO Shengkui and GRUEV V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters[J]. Optics Express, 2011, 19(27): 26161–26173. doi: 10.1364/OE.19.026161

    28. [28]

      TIMOFTE R, DE SMET V, and VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]. The 12th Asian Conference on Computer Vision, Singapore, 2014: 111–126. doi: 10.1007/978-3-319-16817-3_8.

    29. [29]

      ZEYDE R, ELAD M, and PROTTER M. On single image scale-up using sparse-representations[C]. The 7th International Conference on Curves and Surfaces, Avignon, France, 2010: 711–730. doi: 10.1007/978-3-642-27413-8_47.

    1. [1]

      袁庆军, 张勋成, 高杨, 王永娟. 轻量级分组密码PUFFIN的差分故障攻击. 电子与信息学报, 2020, 42(6): 1519-1525.

    2. [2]

      游凌, 李伟浩, 张文林, 王科人. 基于深度神经网络的Morse码自动译码算法. 电子与信息学报, 2020, 41(0): 1-6.

    3. [3]

      申铉京, 沈哲, 黄永平, 王玉. 基于非局部操作的深度卷积神经网络车位占用检测算法. 电子与信息学报, 2020, 41(0): 1-8.

    4. [4]

      姚敏立, 王旭健, 张峰干, 戴定成. 基于动态参数差分进化算法的多约束稀布矩形面阵优化. 电子与信息学报, 2020, 42(5): 1281-1287.

    5. [5]

      马彬, 王梦雪, 谢显中. 超密集异构无线网络中基于位置预测的切换算法. 电子与信息学报, 2020, 42(0): 1-9.

    6. [6]

      张文明, 姚振飞, 高雅昆, 李海滨. 一种平衡准确性以及高效性的显著性目标检测深度卷积网络模型. 电子与信息学报, 2020, 42(5): 1201-1208.

    7. [7]

      缪祥华, 单小撤. 基于密集连接卷积神经网络的入侵检测技术研究. 电子与信息学报, 2020, 41(0): 1-7.

    8. [8]

      贺利芳, 吴雪霜, 张天骐. 正交多用户短参考差分混沌移位键控通信系统性能分析. 电子与信息学报, 2020, 42(0): 1-9.

    9. [9]

      贺利芳, 陈俊, 张天骐. 短参考多用户差分混沌移位键控通信系统性能分析. 电子与信息学报, 2020, 42(0): 1-8.

    10. [10]

      褚征, 于炯. 基于随机森林的流处理检查点性能预测. 电子与信息学报, 2020, 42(6): 1452-1459.

    11. [11]

      陈家祯, 吴为民, 郑子华, 叶锋, 连桂仁, 许力. 基于虚拟光学的视觉显著目标可控放大重建. 电子与信息学报, 2020, 42(5): 1209-1215.

    12. [12]

      王年, 胡旭阳, 朱凡, 唐俊. 基于视图感知的单视图三维重建算法. 电子与信息学报, 2020, 42(0): 1-8.

    13. [13]

      唐伦, 曹睿, 廖皓, 王兆堃. 基于深度强化学习的服务功能链可靠部署算法. 电子与信息学报, 2020, 42(0): 1-8.

    14. [14]

      陈前斌, 管令进, 李子煜, 王兆堃, 杨恒, 唐伦. 基于深度强化学习的异构云无线接入网自适应无线资源分配算法. 电子与信息学报, 2020, 42(6): 1468-1477.

    15. [15]

      张斌, 吴浩明. 一种面向连接的快速多维包分类算法. 电子与信息学报, 2020, 42(6): 1526-1533.

    16. [16]

      唐伦, 魏延南, 谭颀, 唐睿, 陈前斌. H-CRAN网络下联合拥塞控制和资源分配的网络切片动态资源调度策略. 电子与信息学报, 2020, 42(5): 1244-1252.

    17. [17]

      向敏, 饶华阳, 张进进, 陈梦鑫. 基于GCN的软件定义电力通信网络路由控制策略. 电子与信息学报, 2020, 42(0): 1-8.

    18. [18]

      李劲松, 彭建华, 刘树新, 季新生. 一种基于线性规划的有向网络链路预测方法. 电子与信息学报, 2020, 41(0): 1-9.

    19. [19]

      归伟夏, 陆倩, 苏美力. 关于系统级故障诊断的烟花-反向传播神经网络算法. 电子与信息学报, 2020, 42(5): 1102-1109.

    20. [20]

      杨书新, 梁文, 朱凯丽. 基于三级邻居的复杂网络节点影响力度量方法. 电子与信息学报, 2020, 42(5): 1140-1148.

  • 图 1  残差块结构图

    图 2  多级跳线连接的残差块结构图

    图 3  相邻两个多级跳线连接的残差块结构图

    图 4  多级跳线连接的深度残差网络结构图

    图 5  不同跳线系数测得的峰值信噪比(PSNR)曲线

    图 6  Set5 测试集中的baby_GT重建对比图

    表 1  在Set5测试集上的测得的PSNR(dB)/SSIM值

    放大因子Bicubic[27]A+[28]SRCNN[18]FSRCNN[19]ESPCN[21]本文方法
    233.66/0.929936.54/0.954436.66/0.954237.00/0.955837.06/0.955937.35/0.9573
    330.39/0.868232.58/0.908832.75/0.909033.16/0.910433.13/0.913533.45/0.9162
    428.42/0.810430.28/0.860330.48/0.862830.71/0.865730.90/0.867331.07/0.8751
    下载: 导出CSV

    表 2  在Set14测试集上的测得的PSNR(dB)/ SSIM值

    放大因子BicubicA+SRCNNFSRCNNESPCN本文方法
    230.24/0.868832.28/0.905632.42/0.906332.63/0.908832.75/0.909833.34/0.9143
    327.55/0.774229.13/0.818829.28/0.820929.43/0.824229.49/0.827130.09/0.8512
    426.00/0.702727.32/0.749127.49/0.750327.59/0.753527.73/0.763728.26/0.7893
    下载: 导出CSV
  • 加载中
图(6)表(2)
计量
  • PDF下载量:  51
  • 文章访问数:  1429
  • HTML全文浏览量:  703
文章相关
  • 通讯作者:  赵小强, xqzhao@lut.cn
  • 收稿日期:  2019-01-15
  • 录用日期:  2019-06-30
  • 网络出版日期:  2019-07-19
  • 刊出日期:  2019-10-01
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

/

返回文章