高级搜索

多级跳线连接的深度残差网络超分辨率重建

赵小强 宋昭漾

引用本文: 赵小强, 宋昭漾. 多级跳线连接的深度残差网络超分辨率重建[J]. 电子与信息学报, doi: 10.11999/JEIT190036 shu
Citation:  Xiaoqiang ZHAO, Zhaoyang SONG. Super-Resolution Reconstruction of Deep Residual Network with Multi-Level Skip Connections[J]. Journal of Electronics and Information Technology, doi: 10.11999/JEIT190036 shu

多级跳线连接的深度残差网络超分辨率重建

    作者简介: 赵小强: 男,1969年生,博士生导师,教授,主要研究方向为故障诊断,图像处理,生产调度等;
    宋昭漾: 男,1995年生,硕士生,研究方向为图像处理;
    通讯作者: 赵小强, xqzhao@lut.cn
  • 基金项目: 国家科学自然基金(61763029, 61873116)

摘要: 由于快速的卷积神经网络超分辨率重建算法(FSRCNN)卷积层数少、相邻卷积层的特征信息之间缺乏关联性,因此难以提取到图像深层信息导致图像超分辨率重建效果不佳。针对此问题,该文提出多级跳线连接的深度残差网络超分辨率重建方法。首先,该方法设计了多级跳线连接的残差块,在多级跳线连接的残差块基础上构造了多级跳线连接的深度残差网络,解决相邻卷积层的特性信息缺乏关联性的问题;然后,使用随机梯度下降法(SGD)以可调节的学习率策略对多级跳线连接的深度残差网络进行训练,得到该网络超分辨率重建模型;最后,将低分辨率图像输入到多级跳线连接的深度残差网络超分辨率重建模型中,通过多级跳线连接的残差块得到预测的残差特征值,再将残差图像和低分辨率图像组合在一起转化为高分辨率图像。该文方法与bicubic, A+, SRCNN, FSRCNN和ESPCN算法在Set5和Set14测试集上进行了对比测试,在视觉效果和评价指标数值上该方法都优于其它对比算法。

English

    1. [1]

      THORNTON M W, ATKINSON P M, and HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473–491. doi: 10.1080/01431160500207088

    2. [2]

      PELED S and YESHURUN Y. Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29–35. doi: 10.1002/1522-2594(200101)45

    3. [3]

      ZOU W W W and YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327–340. doi: 10.1109/TIP.2011.2162423

    4. [4]

      LU Huimin, LI Yujie, CHEN Min, et al. Brain intelligence: Go beyond artificial intelligence[J]. Mobile Networks and Applications, 2018, 23(2): 368–375. doi: 10.1007/s11036-017-0932-8

    5. [5]

      KOCH M. Artificial intelligence is becoming natural[J]. Cell, 2018, 173(3): 531–533. doi: 10.1016/j.cell.2018.04.007

    6. [6]

      LEO M, MEDIONI G, TRIVEDI M, et al. Computer vision for assistive technologies[J]. Computer Vision and Image Understanding, 2017, 154: 1–15. doi: 10.1016/j.cviu.2016.09.001

    7. [7]

      ZHU Hong, TANG Xinming, XIE Junfeng, et al. Spatio-temporal super-resolution reconstruction of remote-sensing images based on adaptive multi-scale detail enhancement[J]. Sensors, 2018, 18(2): 498. doi: 10.3390/s18020498

    8. [8]

      SHI Jun, LIU Qingping, WANG Chaofeng, et al. Super-resolution reconstruction of MR image with a novel residual learning network algorithm[J]. Physics in Medicine & Biology, 2018, 63(8): 085011. doi: 10.1088/1361-6560/aab9e9

    9. [9]

      SU Heng, ZHOU Jie, and ZHANG Zhihao. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202–1213. doi: 10.3724/SP.J.1004.2013.01202

    10. [10]

      GRIBBON K T and BAILEY D G. A novel approach to real-time bilinear interpolation[C]. Proceedings of the 2nd IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 2004: 126–131. doi: 10.1109/DELTA.2004.10055.

    11. [11]

      FRITSCH F N and CARLSON R E. Monotone piecewise cubic interpolation[J]. SIAM Journal on Numerical Analysis, 1980, 17(2): 238–246. doi: 10.1137/0717021

    12. [12]

      STARK H and OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715

    13. [13]

      PATANAVIJIT V and JITAPUNKUL S. An iterative super-resolution reconstruction of image sequences using fast affine block-based registration with BTV regularization[C]. Proceedings of 2006 IEEE Asia Pacific Conference on Circuits and Systems, Singapore, 2006: 1717–1720. doi: 10.1109/APCCAS.2006.342128.

    14. [14]

      ZHOU Fei, YANG Wenming, and LIAO Qingmin. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312–3318. doi: 10.1109/TIP.2012.2189576

    15. [15]

      LIN Zhouchen and SHUM H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83–97. doi: 10.1109/TPAMI.2004.1261081

    16. [16]

      YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55–75. doi: 10.1109/MCI.2018.2840738

    17. [17]

      KERMANY D S, GOLDBAUM M, CAI Wenjia, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning[J]. Cell, 2018, 172(5): 1122–1131.e9. doi: 10.1016/j.cell.2018.02.010

    18. [18]

      DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281

    19. [19]

      DONG Chao, LOY C C, and TANG Xiaoou. Accelerating the super-resolution convolutional neural network[C]. Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 391–407. doi: 10.1007/978-3-319-46475-6_25.

    20. [20]

      KIM J, LEE J K, and LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1637–1645. doi: 10.1109/CVPR.2016.181.

    21. [21]

      SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874–1883. doi: 10.1109/CVPR.2016.207.

    22. [22]

      LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4681–4690. doi: 10.1109/CVPR.2017.19.

    23. [23]

      HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.

    24. [24]

      KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1646–1654. doi: 10.1109/CVPR.2016.182.

    25. [25]

      YUAN Fei, HUANG Lianfen, and YAO Yan. An improved PSNR algorithm for objective video quality evaluation[C]. Proceedings of 2007 Chinese Control Conference, Hunan, China, 2007: 376–380. doi: 10.1109/CHICC.2006.4347144.

    26. [26]

      WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861

    27. [27]

      GAO Shengkui and GRUEV V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters[J]. Optics Express, 2011, 19(27): 26161–26173. doi: 10.1364/OE.19.026161

    28. [28]

      TIMOFTE R, DE SMET V, and VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]. Proceedings of the 12th Asian Conference on Computer Vision, Singapore, 2014: 111–126. doi: 10.1007/978-3-319-16817-3_8.

    29. [29]

      ZEYDE R, ELAD M, and PROTTER M. On single image scale-up using sparse-representations[C]. Proceedings of the 7th International Conference on Curves and Surfaces, Avignon, France, 2010: 711–730. doi: 10.1007/978-3-642-27413-8_47.

    1. [1]

      陈书贞张祎俊练秋生. 基于多尺度稠密残差网络的JPEG压缩伪迹去除方法. 电子与信息学报, doi: 10.11999/JEIT180963

    2. [2]

      葛芸马琳江顺亮叶发茂. 基于高层特征图组合及池化的高分辨率遥感图像检索. 电子与信息学报, doi: 10.11999/JEIT190017

    3. [3]

      秦宁宁金磊许健徐帆杨乐. 邻近信息约束下的随机异构无线传感器网络节点调度算法. 电子与信息学报, doi: 10.11999/JEIT190094

    4. [4]

      王练张贺张昭张勋杨. 基于自适应随机线性网络编码的优先级调度方案. 电子与信息学报, doi: 10.11999/JEIT180885

    5. [5]

      杨宏宇王峰岩. 基于深度卷积神经网络的气象雷达噪声图像语义分割方法. 电子与信息学报, doi: 10.11999/JEIT190098

    6. [6]

      兰巨龙于倡和胡宇翔李子勇. 基于深度增强学习的软件定义网络路由优化机制. 电子与信息学报, doi: 10.11999/JEIT180870

    7. [7]

      盖杉鲍中运. 基于改进深度卷积神经网络的纸币识别研究. 电子与信息学报, doi: 10.11999/JEIT181097

    8. [8]

      周武杰潘婷顾鹏笠翟治年. 基于金字塔池化网络的道路场景深度估计方法. 电子与信息学报, doi: 10.11999/JEIT180957

    9. [9]

      韦永壮史佳利李灵琛. LiCi分组密码算法的不可能差分分析. 电子与信息学报, doi: 10.11999/JEIT180729

    10. [10]

      张刚赵畅畅张天骐. 短参考正交多用户差分混沌键控方案的性能分析. 电子与信息学报, doi: 10.11999/JEIT181038

    11. [11]

      罗钧杨永松侍宝玉. 基于改进的自适应差分演化算法的二维Otsu多阈值图像分割. 电子与信息学报, doi: 10.11999/JEIT180949

    12. [12]

      佟星元李茂董嗣万. 一种快速响应无片外电容低压差线性稳压器. 电子与信息学报, doi: 10.11999/JEIT181060

    13. [13]

      刘广凯全厚德孙慧贤崔佩璋池阔姚少林. 极低信噪比下对偶序列跳频信号的随机共振检测方法. 电子与信息学报, doi: 10.11999/JEIT190157

    14. [14]

      唐敏齐栋刘成城赵拥军. 基于多级阻塞的稳健相干自适应波束形成. 电子与信息学报, doi: 10.11999/JEIT180332

    15. [15]

      魏嘉琪张磊刘宏伟盛佳恋. 曲线交叠外推的微动多目标宽带分辨算法. 电子与信息学报, doi: 10.11999/JEIT190033

    16. [16]

      于洪涛丁悦航刘树新黄瑞阳谷允捷. 一种基于超节点理论的本体关系消冗算法. 电子与信息学报, doi: 10.11999/JEIT180793

    17. [17]

      余映吴青龙邵凯旋康迂星杨鉴. 基于超复数域小波变换的显著性检测. 电子与信息学报, doi: 10.11999/JEIT180738

    18. [18]

      王斐吴仕超刘少林张亚徽魏颖. 基于脑电信号深度迁移学习的驾驶疲劳检测. 电子与信息学报, doi: 10.11999/JEIT180900

    19. [19]

      李扬张伟涛楼顺天. 基于联合对角化的声信号深度卷积混合盲分离方法. 电子与信息学报, doi: 10.11999/JEIT190067

    20. [20]

      周洋吴佳忆陆宇殷海兵. 面向三维高效视频编码的深度图错误隐藏. 电子与信息学报, doi: 10.11999/JEIT180926

  • 图 1  残差块结构图

    图 2  多级跳线连接的残差块结构图

    图 3  相邻两个多级跳线连接的残差块结构图

    图 4  多级跳线连接的深度残差网络结构图

    图 5  不同跳线系数测得的峰值信噪比(PSNR)曲线

    图 6  Set5 测试集中的baby_GT重建对比图

    表 1  在Set5测试集上的测得的PSNR(dB)/SSIM值

    放大因子Bicubic[27]A+[28]SRCNN[18]FSRCNN[19]ESPCN[21]本文方法
    233.66/0.929936.54/0.954436.66/0.954237.00/0.955837.06/0.955937.35/0.9573
    330.39/0.868232.58/0.908832.75/0.909033.16/0.910433.13/0.913533.45/0.9162
    428.42/0.810430.28/0.860330.48/0.862830.71/0.865730.90/0.867331.07/0.8751
    下载: 导出CSV

    表 2  在Set14测试集上的测得的PSNR(dB)/ SSIM值

    放大因子BicubicA+SRCNNFSRCNNESPCN本文方法
    230.24/0.868832.28/0.905632.42/0.906332.63/0.908832.75/0.909833.34/0.9143
    327.55/0.774229.13/0.818829.28/0.820929.43/0.824229.49/0.827130.09/0.8512
    426.00/0.702727.32/0.749127.49/0.750327.59/0.753527.73/0.763728.26/0.7893
    下载: 导出CSV
  • 加载中
图(6)表(2)
计量
  • PDF下载量:  8
  • 文章访问数:  136
  • HTML全文浏览量:  63
  • 引证文献数: 0
文章相关
  • 通讯作者:  赵小强, xqzhao@lut.cn
  • 收稿日期:  2019-01-15
  • 录用日期:  2019-06-30
  • 网络出版日期:  2019-07-19
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

/

返回文章