高级搜索

多级跳线连接的深度残差网络超分辨率重建

赵小强 宋昭漾

引用本文: 赵小强, 宋昭漾. 多级跳线连接的深度残差网络超分辨率重建[J]. 电子与信息学报, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu
Citation:  Xiaoqiang ZHAO, Zhaoyang SONG. Super-Resolution Reconstruction of Deep Residual Network with Multi-Level Skip Connections[J]. Journal of Electronics and Information Technology, 2019, 41(10): 2501-2508. doi: 10.11999/JEIT190036 shu

多级跳线连接的深度残差网络超分辨率重建

    作者简介: 赵小强: 男,1969年生,博士生导师,教授,主要研究方向为故障诊断,图像处理,生产调度等;
    宋昭漾: 男,1995年生,硕士生,研究方向为图像处理
    通讯作者: 赵小强,xqzhao@lut.cn
  • 基金项目: 国家科学自然基金(61763029, 61873116)

摘要: 由于快速的卷积神经网络超分辨率重建算法(FSRCNN)卷积层数少、相邻卷积层的特征信息之间缺乏关联性,因此难以提取到图像深层信息导致图像超分辨率重建效果不佳。针对此问题,该文提出多级跳线连接的深度残差网络超分辨率重建方法。首先,该方法设计了多级跳线连接的残差块,在多级跳线连接的残差块基础上构造了多级跳线连接的深度残差网络,解决相邻卷积层的特性信息缺乏关联性的问题;然后,使用随机梯度下降法(SGD)以可调节的学习率策略对多级跳线连接的深度残差网络进行训练,得到该网络超分辨率重建模型;最后,将低分辨率图像输入到多级跳线连接的深度残差网络超分辨率重建模型中,通过多级跳线连接的残差块得到预测的残差特征值,再将残差图像和低分辨率图像组合在一起转化为高分辨率图像。该文方法与bicubic, A+, SRCNN, FSRCNN和ESPCN算法在Set5和Set14测试集上进行了对比测试,在视觉效果和评价指标数值上该方法都优于其它对比算法。

English

    1. [1]

      THORNTON M W, ATKINSON P M, and HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473–491. doi: 10.1080/01431160500207088

    2. [2]

      PELED S and YESHURUN Y. Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29–35. doi: 10.1002/1522-2594(200101)45

    3. [3]

      ZOU W W W and YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327–340. doi: 10.1109/TIP.2011.2162423

    4. [4]

      LU Huimin, LI Yujie, CHEN Min, et al. Brain intelligence: Go beyond artificial intelligence[J]. Mobile Networks and Applications, 2018, 23(2): 368–375. doi: 10.1007/s11036-017-0932-8

    5. [5]

      KOCH M. Artificial intelligence is becoming natural[J]. Cell, 2018, 173(3): 531–533. doi: 10.1016/j.cell.2018.04.007

    6. [6]

      LEO M, MEDIONI G, TRIVEDI M, et al. Computer vision for assistive technologies[J]. Computer Vision and Image Understanding, 2017, 154: 1–15. doi: 10.1016/j.cviu.2016.09.001

    7. [7]

      ZHU Hong, TANG Xinming, XIE Junfeng, et al. Spatio-temporal super-resolution reconstruction of remote-sensing images based on adaptive multi-scale detail enhancement[J]. Sensors, 2018, 18(2): 498. doi: 10.3390/s18020498

    8. [8]

      SHI Jun, LIU Qingping, WANG Chaofeng, et al. Super-resolution reconstruction of MR image with a novel residual learning network algorithm[J]. Physics in Medicine & Biology, 2018, 63(8): 085011. doi: 10.1088/1361-6560/aab9e9

    9. [9]

      SU Heng, ZHOU Jie, and ZHANG Zhihao. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202–1213. doi: 10.3724/SP.J.1004.2013.01202

    10. [10]

      GRIBBON K T and BAILEY D G. A novel approach to real-time bilinear interpolation[C]. The 2nd IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 2004: 126–131. doi: 10.1109/DELTA.2004.10055.

    11. [11]

      FRITSCH F N and CARLSON R E. Monotone piecewise cubic interpolation[J]. SIAM Journal on Numerical Analysis, 1980, 17(2): 238–246. doi: 10.1137/0717021

    12. [12]

      STARK H and OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715

    13. [13]

      PATANAVIJIT V and JITAPUNKUL S. An iterative super-resolution reconstruction of image sequences using fast affine block-based registration with BTV regularization[C]. Proceedings of 2006 IEEE Asia Pacific Conference on Circuits and Systems, Singapore, 2006: 1717–1720. doi: 10.1109/APCCAS.2006.342128.

    14. [14]

      ZHOU Fei, YANG Wenming, and LIAO Qingmin. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312–3318. doi: 10.1109/TIP.2012.2189576

    15. [15]

      LIN Zhouchen and SHUM H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83–97. doi: 10.1109/TPAMI.2004.1261081

    16. [16]

      YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55–75. doi: 10.1109/MCI.2018.2840738

    17. [17]

      KERMANY D S, GOLDBAUM M, CAI Wenjia, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning[J]. Cell, 2018, 172(5): 1122–1131.e9. doi: 10.1016/j.cell.2018.02.010

    18. [18]

      DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281

    19. [19]

      DONG Chao, LOY C C, and TANG Xiaoou. Accelerating the super-resolution convolutional neural network[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 391–407. doi: 10.1007/978-3-319-46475-6_25.

    20. [20]

      KIM J, LEE J K, and LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1637–1645. doi: 10.1109/CVPR.2016.181.

    21. [21]

      SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874–1883. doi: 10.1109/CVPR.2016.207.

    22. [22]

      LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4681–4690. doi: 10.1109/CVPR.2017.19.

    23. [23]

      HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.

    24. [24]

      KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1646–1654. doi: 10.1109/CVPR.2016.182.

    25. [25]

      YUAN Fei, HUANG Lianfen, and YAO Yan. An improved PSNR algorithm for objective video quality evaluation[C]. 2007 Chinese Control Conference, Hunan, China, 2007: 376–380. doi: 10.1109/CHICC.2006.4347144.

    26. [26]

      WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861

    27. [27]

      GAO Shengkui and GRUEV V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters[J]. Optics Express, 2011, 19(27): 26161–26173. doi: 10.1364/OE.19.026161

    28. [28]

      TIMOFTE R, DE SMET V, and VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]. The 12th Asian Conference on Computer Vision, Singapore, 2014: 111–126. doi: 10.1007/978-3-319-16817-3_8.

    29. [29]

      ZEYDE R, ELAD M, and PROTTER M. On single image scale-up using sparse-representations[C]. The 7th International Conference on Curves and Surfaces, Avignon, France, 2010: 711–730. doi: 10.1007/978-3-642-27413-8_47.

    1. [1]

      潘一苇, 杨司韩, 彭华, 李天昀, 王文雅. 基于矢量图的特定辐射源识别方法. 电子与信息学报, 2019, 41(0): 1-9.

    2. [2]

      何灏, 易卫东, 陈永锐, 王喆. 基于无迹卡尔曼滤波估计的无线传感器网络时钟分辨率优化. 电子与信息学报, 2019, 41(3): 687-693.

    3. [3]

      谢湘, 张立强, 王晶. 残差网络在婴幼儿哭声识别中的应用. 电子与信息学报, 2019, 41(1): 233-239.

    4. [4]

      陈书贞, 张祎俊, 练秋生. 基于多尺度稠密残差网络的JPEG压缩伪迹去除方法. 电子与信息学报, 2019, 41(10): 2479-2486.

    5. [5]

      胡炎, 单子力, 高峰. 一种面向多星多分辨率的SAR图像舰船候选区域提取方法. 电子与信息学报, 2019, 41(4): 770-778.

    6. [6]

      王超, 王岩飞, 王琦, 詹学丽. 基于回波序列最小二乘拟合的高分辨率SAR运动目标速度估计. 电子与信息学报, 2019, 41(5): 1055-1062.

    7. [7]

      葛芸, 马琳, 江顺亮, 叶发茂. 基于高层特征图组合及池化的高分辨率遥感图像检索. 电子与信息学报, 2019, 41(10): 2487-2494.

    8. [8]

      徐少平, 张贵珍, 李崇禧, 刘婷云, 唐祎玲. 基于深度置信网络的随机脉冲噪声快速检测算法. 电子与信息学报, 2019, 41(5): 1130-1136.

    9. [9]

      刘嘉铭, 邢孟道, 符吉祥, 徐丹. 基于模型重建的深度卷积网络权值可视化方法. 电子与信息学报, 2019, 41(9): 2194-2200.

    10. [10]

      袁野, 贾克斌, 刘鹏宇. 基于深度卷积神经网络的多元医学信号多级上下文自编码器. 电子与信息学报, 2019, 41(0): 1-8.

    11. [11]

      高丽江, 杨海钢, 张超. 低热梯度导向的三维FPGA互连通道网络架构研究. 电子与信息学报, 2019, 41(10): 2389-2395.

    12. [12]

      汪荣贵, 韩梦雅, 杨娟, 薛丽霞, 胡敏. 多级注意力特征网络的小样本学习. 电子与信息学报, 2019, 41(0): 1-7.

    13. [13]

      王粉花, 谢斌, 王华涛. 基于FPGA的快速差频测量系统设计. 电子与信息学报, 2019, 41(1): 187-194.

    14. [14]

      王练, 张贺, 张昭, 张勋杨. 基于自适应随机线性网络编码的优先级调度方案. 电子与信息学报, 2019, 41(8): 1861-1868.

    15. [15]

      秦宁宁, 金磊, 许健, 徐帆, 杨乐. 邻近信息约束下的随机异构无线传感器网络节点调度算法. 电子与信息学报, 2019, 41(10): 2310-2317.

    16. [16]

      高敏娟, 党宏社, 魏立力, 张选德. 基于非局部梯度的图像质量评价算法. 电子与信息学报, 2019, 41(5): 1122-1129.

    17. [17]

      盖杉, 鲍中运. 基于改进深度卷积神经网络的纸币识别研究. 电子与信息学报, 2019, 41(8): 1992-2000.

    18. [18]

      兰巨龙, 于倡和, 胡宇翔, 李子勇. 基于深度增强学习的软件定义网络路由优化机制. 电子与信息学报, 2019, 41(0): 1-6.

    19. [19]

      王鑫, 李可, 宁晨, 黄凤辰. 基于深度卷积神经网络和多核学习的遥感图像分类方法. 电子与信息学报, 2019, 41(5): 1098-1105.

    20. [20]

      郭晨, 简涛, 徐从安, 何友, 孙顺. 基于深度多尺度一维卷积神经网络的雷达舰船目标识别. 电子与信息学报, 2019, 41(6): 1302-1309.

  • 图 1  残差块结构图

    图 2  多级跳线连接的残差块结构图

    图 3  相邻两个多级跳线连接的残差块结构图

    图 4  多级跳线连接的深度残差网络结构图

    图 5  不同跳线系数测得的峰值信噪比(PSNR)曲线

    图 6  Set5 测试集中的baby_GT重建对比图

    表 1  在Set5测试集上的测得的PSNR(dB)/SSIM值

    放大因子Bicubic[27]A+[28]SRCNN[18]FSRCNN[19]ESPCN[21]本文方法
    233.66/0.929936.54/0.954436.66/0.954237.00/0.955837.06/0.955937.35/0.9573
    330.39/0.868232.58/0.908832.75/0.909033.16/0.910433.13/0.913533.45/0.9162
    428.42/0.810430.28/0.860330.48/0.862830.71/0.865730.90/0.867331.07/0.8751
    下载: 导出CSV

    表 2  在Set14测试集上的测得的PSNR(dB)/ SSIM值

    放大因子BicubicA+SRCNNFSRCNNESPCN本文方法
    230.24/0.868832.28/0.905632.42/0.906332.63/0.908832.75/0.909833.34/0.9143
    327.55/0.774229.13/0.818829.28/0.820929.43/0.824229.49/0.827130.09/0.8512
    426.00/0.702727.32/0.749127.49/0.750327.59/0.753527.73/0.763728.26/0.7893
    下载: 导出CSV
  • 加载中
图(6)表(2)
计量
  • PDF下载量:  17
  • 文章访问数:  561
  • HTML全文浏览量:  256
文章相关
  • 通讯作者:  赵小强, xqzhao@lut.cn
  • 收稿日期:  2019-01-15
  • 录用日期:  2019-06-30
  • 网络出版日期:  2019-07-19
  • 刊出日期:  2019-10-01
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

/

返回文章