高级搜索

基于Q-Learning算法的毫微微小区功率控制算法

李云 唐英 刘涵霄

引用本文: 李云, 唐英, 刘涵霄. 基于Q-Learning算法的毫微微小区功率控制算法[J]. 电子与信息学报, doi: 10.11999/JEIT181191 shu
Citation:  Yun LI, Ying TANG, Hanxiao LIU. Power Control Algorithm Based on Q-Learning in Femtocell[J]. Journal of Electronics and Information Technology, doi: 10.11999/JEIT181191 shu

基于Q-Learning算法的毫微微小区功率控制算法

    作者简介: 李云: 男,1974年生,教授,博士生导师,主要研究领域为无线移动通信;
    唐英: 女,1993年生,硕士生,研究方向为异构蜂窝无线网络;
    刘涵霄: 男,1994年生,硕士生,研究方向为异构蜂窝无线网络
    通讯作者: 唐英,17749963914@163.com
  • 基金项目: 国家自然科学基金(61671096),重庆市研究生科研创新项目(CYS17220),重庆市“科技创新领军人才支持计划”(CSTCCXLJRC201710),重庆市基础科学与前沿技术研究项目(cstc2017jcyjBX0005),重庆市留学人员创业创新支持计划

摘要: 该文研究macro-femto异构蜂窝网络中移动用户的功率控制问题,首先建立了以最小接收信号信干噪比为约束条件,最大化毫微微小区的总能效为目标的优化模型;然后提出了基于Q-Learning算法的毫微微小区集中式功率控制(PCQL)算法,该算法基于强化学习,能在没有准确信道状态信息的情况下,实现对小区内所有用户终端的发射功率统一调整。仿真结果表明该算法能实现对用户终端的功率有效控制,提升系统能效。

English

    1. [1]

      LÓPEZ-PÉREZ D, DING M, CLAUSSEN H, et al. Towards 1 Gbps/UE in cellular systems: understanding ultra-dense small cell deployments[J]. IEEE Communications Surveys & Tutorials, 2015, 17(4): 2078–2101. doi: 10.1109/COMST.2015.2439636

    2. [2]

      YUNAS S F, VALKAMA M, and NIEMELÄ J. Spectral and energy efficiency of ultra-dense networks under different deployment strategies[J]. IEEE Communications Magazine, 2015, 53(1): 90–100. doi: 10.1109/MCOM.2015.7010521

    3. [3]

      MARTOLIA D, SATHYA V, RANGISETTI A K, et al. Enhancing performance of victim macro users via joint ABSF and dynamic power control in LTE HetNets[C]. The Twenty-third National Conference on Communications, Chennai, India, 2017: 1–6.

    4. [4]

      SHIN D and CHOI S. Dynamic power control for balanced data traffic with coverage in femtocell networks[C]. The 8th International Wireless Communications and Mobile Computing Conference, Limassol, Cyprus, 2012: 648–653.

    5. [5]

      ZHANG Jinzhu, HONG Peilin, XUE Kaiping, et al. A novel power control scheme for femtocell in heterogeneous networks[C]. 2012 IEEE Consumer Communications and Networking Conference, Las Vegas, USA, 2012: 802–806.

    6. [6]

      PAN Zhenni, MEGUMI, SAITOU, et al. Neuron control-based power adjustment scheme for sleep two-tier cellular networks[C]. 2014 IEEE Wireless Communications and Networking Conference, Istanbul, Turkey, 2014: 3201–3206.

    7. [7]

      ZHOU Tianqing, LIU Zunxiong, ZHAO Junhui, et al. Joint user association and power control for load balancing in downlink heterogeneous cellular networks[J]. IEEE Transactions on Vehicular Technology, 2018, 67(3): 2582–2593. doi: 10.1109/TVT.2017.2768574

    8. [8]

      MARTIN-VEGA F J, GOMEZ G, AGUAYO-TORRES M C, et al. Analytical modeling of interference aware power control for the uplink of heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2016, 15(10): 6742–6757. doi: 10.1109/TWC.2016.2588469

    9. [9]

      ZHANG Jing, LIAO Yan, and XIN Yili. Uplink power control for heterogeneous small cell networks[C]. 2016 IEEE 83rd Vehicular Technology Conference, Nanjing, China, 2016: 1–5.

    10. [10]

      WANG Min, GAO Hui, and LV Tiejun. Energy-efficient user association and power control in the heterogeneous network[J]. IEEE Access, 2017, 5: 5059–5068. doi: 10.1109/ACCESS.2017.2690305

    11. [11]

      ZHANG Jing, XIANG Lin, NG D W K, et al. Energy efficiency evaluation of multi-tier cellular uplink transmission under maximum power constraint[J]. IEEE Transactions on Wireless Communications, 2017, 16(11): 7092–7107. doi: 10.1109/TWC.2017.2739142

    12. [12]

      PAN Zhenni and SHIMAMOTO S. Cell sizing based energy optimization in joint macro-femto deployments via sleep activation[C]. 2013 IEEE Wireless Communications and Networking Conference, Shanghai, China, 2013: 4765–4770.

    13. [13]

      SHIFAT A S M Z, CHOWDHURY M Z, and JANG Y M. Game-based approach for QoS provisioning and interference management in heterogeneous networks[J]. IEEE Access, 2018, 6: 10208–10220. doi: 10.1109/ACCESS.2017.2704094

    14. [14]

      MISHRA S and MURTHY C S R. Increasing energy efficiency via transmit power spreading in dense femto cell networks[J]. IEEE Systems Journal, 2018, 12(1): 971–980. doi: 10.1109/JSYST.2016.2573845

    15. [15]

      GURUACHARYA S, NIYATO D, KIM D I, et al. Hierarchical competition for downlink power allocation in OFDMA femtocell networks[J]. IEEE Transactions on Wireless Communications, 2013, 12(4): 1543–1553. doi: 10.1109/TWC.2013.022213.120016

    16. [16]

      WANG Haining, WANG Jiaheng, and DING Zhi. Distributed power control in a two-tier heterogeneous network[J]. IEEE Transactions on Wireless Communications, 2015, 14(12): 6509–6523. doi: 10.1109/TWC.2015.2456055

    17. [17]

      MAO Tingli, FENG Gang, LIANG Liang, et al. Energy-efficient power control for macro-femto networks[C]. The 22nd Wireless and Optical Communication Conference, Chongqing, China, 2013: 122–125.

    18. [18]

      MAO Tingli, FENG Gang, LIANG Liang, et al. Distributed energy-efficient power control for macro-femto networks[J]. IEEE Transactions on Vehicular Technology, 2016, 65(2): 718–731. doi: 10.1109/TVT.2015.2402618

    19. [19]

      LAI Weisheng, CHANG T H, and LEE T S. Joint power and admission control for spectral and energy efficiency maximization in heterogeneous OFDMA networks[J]. IEEE Transactions on Wireless Communications, 2016, 15(5): 3531–3547. doi: 10.1109/TWC.2016.2522958

    20. [20]

      LOODARICHEH R A, MALLICK S, BHARGAVA V K. Energy-efficient resource allocation for OFDMA cellular networks with user cooperation and QoS provisioning[J]. IEEE Transactions on Wireless Communications, 2014, 13(11): 6132–6146. doi: 10.1109/TWC.2014.2329877

    21. [21]

      GHADIMI E, CALABRESE F D, PETERS G, et al. A reinforcement learning approach to power control and rate adaptation in cellular networks[C]. 2017 IEEE International Conference on Communications, Paris, French, 2017: 1–7.

    22. [22]

      周志华. 机器学习[M]. 北京: 清华大学出版社, 2016: 372–390.
      ZHOU Zhihua. Machine Learning[M]. Beijing: Tsinghua University Press, 2016: 372–390.

    1. [1]

      张达敏, 张绘娟, 闫威, 陈忠云, 辛梓芸. 异构网络中基于能效优化的D2D资源分配机制. 电子与信息学报,

    2. [2]

      孙远, 李春国, 黄永明, 杨绿溪. 基于带缓存的云接入网络最优能效设计. 电子与信息学报,

    3. [3]

      刘焕淋, 方菲, 陈勇, 向敏, 马跃. 基于无色无向无冲突可重构光分插复用器节点的全光IP组播能效调度. 电子与信息学报,

    4. [4]

      唐伦, 马润琳, 刘云龙, 王耀玮, 陈前斌. 接入与回传一体化小基站的接入控制与资源分配联合优化算法. 电子与信息学报,

    5. [5]

      李丹, 兰巨龙, 王鹏, 胡宇翔. 基于最长有效功能序列的服务功能链部署算法. 电子与信息学报,

    6. [6]

      钱志鸿, 田春生, 王鑫, 王雪. D2D网络中信道选择与功率控制策略研究. 电子与信息学报,

    7. [7]

      黄容兰, 刘云, 李啟尚, 唐文. 基于非正交多址接入中继通信系统的功率优化. 电子与信息学报,

    8. [8]

      胡宇翔, 李子勇, 胡宗魁, 胡涛. 基于流量工程的软件定义网络控制资源优化机制. 电子与信息学报,

    9. [9]

      黄盛. 两用户非正交多址接入的最优时延均衡和功率控制方法. 电子与信息学报,

    10. [10]

      杨善超, 田康生, 刘仁争, 郑玉军. 基于价值优化的相控阵雷达任务调度算法. 电子与信息学报,

    11. [11]

      唐伦, 马润琳, 杨恒, 陈前斌. 基于非正交多址接入的网络切片联合用户关联和功率分配算法. 电子与信息学报,

    12. [12]

      吕增威, 魏振春, 韩江洪, 孙仁浩, 夏成凯. 基于多目标优化的无线传感器网络移动充电及数据收集算法. 电子与信息学报,

    13. [13]

      卢昱, 刘益岑, 李玺, 陈兴凯, 乔文欣, 陈立云. 面向软件定义网络的服务功能链优化部署算法研究. 电子与信息学报,

    14. [14]

      曾芳玲, 欧阳晓凤, 徐浩, 吕大千. 基于时频融合的长码直接捕获优化算法研究. 电子与信息学报,

    15. [15]

      史久根, 徐皓, 张径, 王继. 软件定义网络中基于效率区间的负载均衡在线优化算法. 电子与信息学报,

    16. [16]

      冯维, 徐永鑫, 刘浩, 许晓荣, 姚英彪. 无线多跳网络快速跨层资源优化分配算法. 电子与信息学报,

    17. [17]

      梁晓萍, 郭振军, 朱昌洪. 基于头脑风暴优化算法的BP神经网络模糊图像复原. 电子与信息学报,

    18. [18]

      钱亚冠, 卢红波, 纪守领, 周武杰, 吴淑慧, 云本胜, 陶祥兴, 雷景生. 基于粒子群优化的对抗样本生成算法. 电子与信息学报,

    19. [19]

      刘小龙. 改进多元宇宙算法求解大规模实值优化问题. 电子与信息学报,

    20. [20]

      刘浩然, 张力悦, 范瑞星, 王海羽, 张春兰. 基于改进鲸鱼优化策略的贝叶斯网络结构学习算法. 电子与信息学报,

  • 图 1  异构蜂窝网络模型

    图 2  代理自主学习过程

    图 3  小区用户数为4时,系统能效对比

    图 4  小区用户数为4时,系统吞吐量对比

    图 5  系统能效与用户数的关系

    图 6  系统吞吐量与用户数的关系

    图 7  信道状态信息存在估计误差时,系统能效与用户数的关系

    图 8  信道状态信息存在估计误差时,系统吞吐量与用户数的关系

    图 9  能效优化的算法运行时间对比

    图 10  吞吐量优化的算法运行时间对比

    表 1  基于Q-Learning算法的毫微微小区功率控制算法(PCQL)

     输入:W, ${n_0}$, $P_{b,\mu }^{\rm{c}} $, ${\rm{SINR}}_{b,\mu }^{\min }$, $p_{b,\mu }^{{\rm{max}}}$, $\gamma $, $\alpha $, $T\;$, $\varepsilon $,动作空间${A_b}$;
     输出:${{\text{π}}^ * }$, $p_{b,\mu }^*$($\mu \in {U_b}$);
     定义:${\text{k}}$表示代理选取的动作;${\rm{SINR}}_{b,\mu }^{{\rm{real}}}$表示${u_{b,\mu }}$与基站$b$通信时  的实际信干噪比;
      $Q\left( {{{\text{s}}_b},{{\text{a}}_b}} \right) = 0$, ${\text{π}}\left( {{{\text{s}}_b},{{\text{a}}_b}} \right) = \frac{1}{{\left| {{A_b}\left( {{{\text{s}}_b}} \right)} \right|}}$, $\text{s}_b^t = \text{s}_b^0$;
      for $t = 0,1, ·\!·\!· ,T\;$ do
      若rand()<$\varepsilon $,从${A_b}$中随机选动作${\text{k}}$;否则${\text{k}} \!=\! \mathop {\arg \max }\limits_{{\text{a}}_b^t} \!Q\left( {{\text{s}}_b^t,{\text{a}}_b^t} \right)$;
      根据式(1)确定${\rm{SINR}}_{b,\mu }^{{\rm{real}}}$;
      for $\mu = 1,2, ·\!·\!· ,{N_b}$ do
      若${\rm{SINR}}_{b,\mu }^{{\mathop{\rm real}\nolimits} } \ge {\rm{SINR}}_{b,\mu }^{\min }$,那么${\lambda _{b,\mu }} = 1$;否则${\lambda _{b,\mu }} = 0$;
      end for;
      根据式(7)计算采取动作${\text{a}}_b^t = {\text{k}}$所带来的奖赏值${\Re _b}\left( {{\text{s}}_b^t,{\text{a}}_b^t} \right)$;
      ${\text{a}}_b^{t + 1} = {\text{π}}\left( {{\text{s}}_b^{t + 1}} \right)$;
      ${\rm Q}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) \leftarrow {\rm Q}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) + \alpha ( {\Re _b}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) \!+\! \gamma \mathop {\max}\limits_{ {\rm{a} }_b^{t + 1} } \left( { {\rm Q}\left( { { {\text{s} } }_b^{t + 1},{ {\text{a} } }_b^{t + 1} } \right)} \right)$  $\left.- {{\rm Q}\left( {{{\text{s}}}_b^t,{{\text{a}}}_b^t} \right)} \right)$;
      ${\text{s}}_b^t \leftarrow {\text{s}}_b^{t + 1}$;
      end for;
      ${{\text{π}}^ * }\left( {{{\text{s}}_b}} \right) = \mathop {\arg \max }\limits_{{{\text{a}}_b}} Q\left( {{{\text{s}}_b},{{\text{a}}_b}} \right),\forall {{\text{s}}_b} \in S$.
    下载: 导出CSV

    表 2  主要的仿真参数

    参数名称参数值
    MBS/FBS1个/4个
    MUE/FUE最大的发射功率37 dBm/30 dBm
    MBS/FBS覆盖范围半径250 m/50 m
    ${{\rm{SINR}} _{b,\mu }}^{\min }$–9 dB
    固定的电路功耗100 mW
    信道带宽10 MHz
    高斯白噪声的功率谱密度${10^{ - 11}}$ W/Hz
    下载: 导出CSV
  • 加载中
图(10)表(2)
计量
  • PDF下载量:  23
  • 文章访问数:  382
  • HTML全文浏览量:  232
文章相关
  • 通讯作者:  唐英, 17749963914@163.com
  • 收稿日期:  2018-12-28
  • 录用日期:  2019-04-10
  • 网络出版日期:  2019-05-21
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

/

返回文章