高级搜索

多级跳线连接的深度残差网络超分辨率重建

赵小强 宋昭漾

引用本文: 赵小强, 宋昭漾. 多级跳线连接的深度残差网络超分辨率重建[J]. 电子与信息学报, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu
Citation:  Xiaoqiang ZHAO, Zhaoyang SONG. Super-Resolution Reconstruction of Deep Residual Network with Multi-Level Skip Connections[J]. Journal of Electronics and Information Technology, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu

多级跳线连接的深度残差网络超分辨率重建

    作者简介: 赵小强: 男,1969年生,博士生导师,教授,主要研究方向为故障诊断,图像处理,生产调度等;
    宋昭漾: 男,1995年生,硕士生,研究方向为图像处理
    通讯作者: 赵小强,xqzhao@lut.cn
  • 基金项目: 国家科学自然基金(61763029, 61873116)

摘要: 由于快速的卷积神经网络超分辨率重建算法(FSRCNN)卷积层数少、相邻卷积层的特征信息之间缺乏关联性,因此难以提取到图像深层信息导致图像超分辨率重建效果不佳。针对此问题,该文提出多级跳线连接的深度残差网络超分辨率重建方法。首先,该方法设计了多级跳线连接的残差块,在多级跳线连接的残差块基础上构造了多级跳线连接的深度残差网络,解决相邻卷积层的特性信息缺乏关联性的问题;然后,使用随机梯度下降法(SGD)以可调节的学习率策略对多级跳线连接的深度残差网络进行训练,得到该网络超分辨率重建模型;最后,将低分辨率图像输入到多级跳线连接的深度残差网络超分辨率重建模型中,通过多级跳线连接的残差块得到预测的残差特征值,再将残差图像和低分辨率图像组合在一起转化为高分辨率图像。该文方法与bicubic, A+, SRCNN, FSRCNN和ESPCN算法在Set5和Set14测试集上进行了对比测试,在视觉效果和评价指标数值上该方法都优于其它对比算法。

English

    1. [1]

      THORNTON M W, ATKINSON P M, and HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473–491. doi: 10.1080/01431160500207088

    2. [2]

      PELED S and YESHURUN Y. Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29–35. doi: 10.1002/1522-2594(200101)45

    3. [3]

      ZOU W W W and YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327–340. doi: 10.1109/TIP.2011.2162423

    4. [4]

      LU Huimin, LI Yujie, CHEN Min, et al. Brain intelligence: Go beyond artificial intelligence[J]. Mobile Networks and Applications, 2018, 23(2): 368–375. doi: 10.1007/s11036-017-0932-8

    5. [5]

      KOCH M. Artificial intelligence is becoming natural[J]. Cell, 2018, 173(3): 531–533. doi: 10.1016/j.cell.2018.04.007

    6. [6]

      LEO M, MEDIONI G, TRIVEDI M, et al. Computer vision for assistive technologies[J]. Computer Vision and Image Understanding, 2017, 154: 1–15. doi: 10.1016/j.cviu.2016.09.001

    7. [7]

      ZHU Hong, TANG Xinming, XIE Junfeng, et al. Spatio-temporal super-resolution reconstruction of remote-sensing images based on adaptive multi-scale detail enhancement[J]. Sensors, 2018, 18(2): 498. doi: 10.3390/s18020498

    8. [8]

      SHI Jun, LIU Qingping, WANG Chaofeng, et al. Super-resolution reconstruction of MR image with a novel residual learning network algorithm[J]. Physics in Medicine & Biology, 2018, 63(8): 085011. doi: 10.1088/1361-6560/aab9e9

    9. [9]

      SU Heng, ZHOU Jie, and ZHANG Zhihao. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202–1213. doi: 10.3724/SP.J.1004.2013.01202

    10. [10]

      GRIBBON K T and BAILEY D G. A novel approach to real-time bilinear interpolation[C]. The 2nd IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 2004: 126–131. doi: 10.1109/DELTA.2004.10055.

    11. [11]

      FRITSCH F N and CARLSON R E. Monotone piecewise cubic interpolation[J]. SIAM Journal on Numerical Analysis, 1980, 17(2): 238–246. doi: 10.1137/0717021

    12. [12]

      STARK H and OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715

    13. [13]

      PATANAVIJIT V and JITAPUNKUL S. An iterative super-resolution reconstruction of image sequences using fast affine block-based registration with BTV regularization[C]. Proceedings of 2006 IEEE Asia Pacific Conference on Circuits and Systems, Singapore, 2006: 1717–1720. doi: 10.1109/APCCAS.2006.342128.

    14. [14]

      ZHOU Fei, YANG Wenming, and LIAO Qingmin. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312–3318. doi: 10.1109/TIP.2012.2189576

    15. [15]

      LIN Zhouchen and SHUM H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83–97. doi: 10.1109/TPAMI.2004.1261081

    16. [16]

      YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55–75. doi: 10.1109/MCI.2018.2840738

    17. [17]

      KERMANY D S, GOLDBAUM M, CAI Wenjia, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning[J]. Cell, 2018, 172(5): 1122–1131.e9. doi: 10.1016/j.cell.2018.02.010

    18. [18]

      DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281

    19. [19]

      DONG Chao, LOY C C, and TANG Xiaoou. Accelerating the super-resolution convolutional neural network[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 391–407. doi: 10.1007/978-3-319-46475-6_25.

    20. [20]

      KIM J, LEE J K, and LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1637–1645. doi: 10.1109/CVPR.2016.181.

    21. [21]

      SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874–1883. doi: 10.1109/CVPR.2016.207.

    22. [22]

      LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4681–4690. doi: 10.1109/CVPR.2017.19.

    23. [23]

      HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.

    24. [24]

      KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1646–1654. doi: 10.1109/CVPR.2016.182.

    25. [25]

      YUAN Fei, HUANG Lianfen, and YAO Yan. An improved PSNR algorithm for objective video quality evaluation[C]. 2007 Chinese Control Conference, Hunan, China, 2007: 376–380. doi: 10.1109/CHICC.2006.4347144.

    26. [26]

      WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861

    27. [27]

      GAO Shengkui and GRUEV V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters[J]. Optics Express, 2011, 19(27): 26161–26173. doi: 10.1364/OE.19.026161

    28. [28]

      TIMOFTE R, DE SMET V, and VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]. The 12th Asian Conference on Computer Vision, Singapore, 2014: 111–126. doi: 10.1007/978-3-319-16817-3_8.

    29. [29]

      ZEYDE R, ELAD M, and PROTTER M. On single image scale-up using sparse-representations[C]. The 7th International Conference on Curves and Surfaces, Avignon, France, 2010: 711–730. doi: 10.1007/978-3-642-27413-8_47.

    1. [1]

      董哲康, 杜晨杰, 林辉品, 赖俊昇, 胡小方, 段书凯. 基于多通道忆阻脉冲耦合神经网络的多帧图像超分辨率重建算法. 电子与信息学报, 2020, 42(4): 835-843.

    2. [2]

      罗会兰, 卢飞, 孔繁胜. 基于区域与深度残差网络的图像语义分割. 电子与信息学报, 2019, 41(11): 2777-2786.

    3. [3]

      潘一苇, 杨司韩, 彭华, 李天昀, 王文雅. 基于矢量图的特定辐射源识别方法. 电子与信息学报, 2020, 42(4): 941-949.

    4. [4]

      韩玉兵, 吴乐南, 张冬青. 基于正则化处理的超分辨率重建. 电子与信息学报, 2007, 29(7): 1713-1716.

    5. [5]

      李民, 李世华, 李小文, 乐翔. 非局部联合稀疏近似的超分辨率重建算法. 电子与信息学报, 2011, 33(6): 1407-1412.

    6. [6]

      杨浩, 高建坡, 吴镇扬. 一种新的图像配准和超分辨率重建算法. 电子与信息学报, 2008, 30(1): 168-171.

    7. [7]

      乔建苹, 刘琚, 闫华, 孙建德. 基于Log-WT的人脸图像超分辨率重建. 电子与信息学报, 2008, 30(6): 1276-1280.

    8. [8]

      宋海英, 何小海, 吴媛媛, 卿粼波. 一种稳健的多视频时空超分辨率重建算法. 电子与信息学报, 2011, 33(9): 2253-2257.

    9. [9]

      徐忠强, 朱秀昌. 基于噪声分布特性的压缩视频超分辨率重建. 电子与信息学报, 2008, 30(3): 752-755.

    10. [10]

      李小燕, 和红杰, 尹忠科, 陈帆. 基于加权Boosting的核偏最小二乘图像超分辨率重建. 电子与信息学报, 2012, 34(7): 1525-1530.

    11. [11]

      程倩倩, 范新南, 李庆武. 基于自类推的NSCT域单幅图像超分辨率重建. 电子与信息学报, 2011, 33(12): 2881-2887.

    12. [12]

      韩玉兵, 束锋, 吴乐南. 基于加权最小二乘滤波的视频序列超分辨率重建. 电子与信息学报, 2009, 31(1): 120-123.

    13. [13]

      张宇, 徐伟强, 陈积明, 孙优贤. 基于最速下降法的车载网络功率控制研究. 电子与信息学报, 2010, 32(10): 2536-2540.

    14. [14]

      孙伟峰, 戴永寿. 采用多级残差滤波的非局部均值图像去噪方法. 电子与信息学报, 2016, 38(8): 1999-2006.

    15. [15]

      陈书贞, 张祎俊, 练秋生. 基于多尺度稠密残差网络的JPEG压缩伪迹去除方法. 电子与信息学报, 2019, 41(10): 2479-2486.

    16. [16]

      谢湘, 张立强, 王晶. 残差网络在婴幼儿哭声识别中的应用. 电子与信息学报, 2019, 41(1): 233-239.

    17. [17]

      曹明明, 干宗良, 崔子冠, 李然, 朱秀昌. 基于2D-PCA特征描述的非负权重邻域嵌入人脸超分辨率重建算法. 电子与信息学报, 2015, 37(4): 777-783.

    18. [18]

      周辉林, 欧阳韬, 刘健. 基于随机平均梯度下降和对比源反演的非线性逆散射算法研究. 电子与信息学报, 2020, 42(0): 1-6.

    19. [19]

      徐忠强, 朱秀昌. 压缩视频超分辨率重建技术. 电子与信息学报, 2007, 29(2): 499-505.

    20. [20]

      刘哲, 杨静, 陈路. 基于非局部稀疏编码的超分辨率图像复原. 电子与信息学报, 2015, 37(3): 522-528.

  • 图 1  残差块结构图

    图 2  多级跳线连接的残差块结构图

    图 3  相邻两个多级跳线连接的残差块结构图

    图 4  多级跳线连接的深度残差网络结构图

    图 5  不同跳线系数测得的峰值信噪比(PSNR)曲线

    图 6  Set5 测试集中的baby_GT重建对比图

    表 1  在Set5测试集上的测得的PSNR(dB)/SSIM值

    放大因子Bicubic[27]A+[28]SRCNN[18]FSRCNN[19]ESPCN[21]本文方法
    233.66/0.929936.54/0.954436.66/0.954237.00/0.955837.06/0.955937.35/0.9573
    330.39/0.868232.58/0.908832.75/0.909033.16/0.910433.13/0.913533.45/0.9162
    428.42/0.810430.28/0.860330.48/0.862830.71/0.865730.90/0.867331.07/0.8751
    下载: 导出CSV

    表 2  在Set14测试集上的测得的PSNR(dB)/ SSIM值

    放大因子BicubicA+SRCNNFSRCNNESPCN本文方法
    230.24/0.868832.28/0.905632.42/0.906332.63/0.908832.75/0.909833.34/0.9143
    327.55/0.774229.13/0.818829.28/0.820929.43/0.824229.49/0.827130.09/0.8512
    426.00/0.702727.32/0.749127.49/0.750327.59/0.753527.73/0.763728.26/0.7893
    下载: 导出CSV
  • 加载中
图(6)表(2)
计量
  • PDF下载量:  46
  • 文章访问数:  1377
  • HTML全文浏览量:  679
文章相关
  • 通讯作者:  赵小强, xqzhao@lut.cn
  • 收稿日期:  2019-01-15
  • 录用日期:  2019-06-30
  • 网络出版日期:  2019-07-19
  • 刊出日期:  2019-10-01
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

/

返回文章