從天貓雙11成交額2684億看RDMA網絡

HUASHUO2019TIANMAOSHUANG11CHENGJIAOE2684YI!ZAICISHUAXINSHIJIEJILU,DINGDANCHUANGXINFENGZHIDADAO54.4WANBI/MIAO,DANRISHUJUCHULILIANGDADAO970PB。

快3开奖结果查询KANZHESHUJUSHIBUSHIJUEDEHENLIHAI!970PB/TIANSHIJIEJILIULIANGHONGFENG!

快3开奖结果查询HEIHEI,BUYONGNIJUEDE,ALIZIJIRENYESHUO“BUSHIRENHEYIDUOYUNDOUNENGCHENGZHUZHEGELIULIANG。ZHONGGUOYOULIANGDUOYUN,YIDUOSHIALIYUN,YIDUOJIAOQITAYUN。”

人家憑啥這么自信呢?阿里巴巴集團CTO張建鋒介紹,即使是在最為繁忙的零點前后,消費者也沒有感受到任何抖動,能夠順暢購物,是因為阿里云在以下四個方面做了核心突破:
第一, 在核心虛擬機系統上自研神龍架構,用自研的服務器做虛擬化,壓力再大也是線性輸出;
第二, 自研云原生數據庫,Ocean Base和Polar DB兩個數據庫表現優越;
第三, 計算與存儲分離,數據從遠端存取,方便擴容;
快3开奖结果查询第四, 采用RDMA網絡,阿里是全球第一個大規模做RDMA網絡的公司。

前三點就不談了,不是本文的主角。今天我們只談RDMA網絡。
注:讀者請注意本文不是阿里的軟文

RDMA的誕生背景

快3开奖结果查询CHUANTONGDETCP/IPWANGLUOTONGXIN,SHUJUXUYAOTONGGUOYONGHUKONGJIANFASONGDAOYUANCHENGJIQIDEYONGHUKONGJIAN。SHUJUFASONGFANGXUYAOJIANGSHUJUCONGYONGHUYINGYONGKONGJIANBufferFUZHIDAONEIHEKONGJIANDESocket BufferZHONG。RANHOUKernelKONGJIANZHONGTIANJIASHUJUBAOTOU,JINXINGSHUJUFENGZHUANG。TONGGUOYIXILIEDUOCENGWANGLUOXIEYI(TCP、UDP、IPDENG)DESHUJUBAOCHULIGONGZUO,SHUJUCAIBEIPushDAONICWANGKAZHONGDEBufferJINXINGWANGLUOCHUANSHU。

快3开奖结果查询XIAOXIJIESHOUFANGJIESHOUCONGYUANCHENGJIQIFASONGDESHUJUBAOHOU,YAOJIANGSHUJUBAOCONGNIC bufferZHONGFUZHISHUJUDAOSocket Buffer。RANHOUJINGGUOYIXIELIEDEDUOCENGWANGLUOXIEYIJINXINGSHUJUBAODEJIEXIGONGZUO。JIEXIHOUDESHUJUBEIFUZHIDAOXIANGYINGWEIZHIDEYONGHUYINGYONGKONGJIANBuffer。ZHEGESHIHOUZAIJINXINGXITONGSHANGXIAWENQIEHUAN,YONGHUYINGYONGCHENGXUCAIBEIDIAOYONG。XIATUZHANSHILECHUANTONGDETCP/IPXIEYICENGDEGONGZUOYUANLI。

本圖來源CSDN博客

BENZHISHANGLAISHUOCHUANTONGDETCP/IPWANGLUOTONGXINSHITONGGUONEIHEFASONGXIAOXI。ERMIANDUIGAOXINGNENGJISUAN、DASHUJUFENXIHELANGYONGXINGIOGAOBINGFA、DISHIYANYINGYONG,XIANYOUTCP/IPRUANYINGJIANJIAGOUHEYINGYONG、GAOCPUXIAOHAODEJISHUTEZHENGGENBENBUNENGMANZUYINGYONGDEXUQIU。DUOCINEICUNKAOBEI、ZHONGDUANCHULI,SHANGXIAWENQIEHUAN、FUZADETCP/IPXIEYICHULI、CUNCHUZHUANFAMOSHIHEDIUBAODAOZHIEWAIYANSHICHAOCHANG。

RDMA是個啥

快3开奖结果查询JIANYUCHUANTONGWANGLUOJIAGOUDEBIDUANHEDANGJINWANGLUODEFAZHANXIANZHUANG,GONGCHENGSHIWUSHIBUKEZAIGANTAN: WEISHENMEFASONGSHUJUYAOHUAZHEMECHANGSHIJIAN?WEISHENMEJISUANNENGLIZHEMENIUDEFUWUQILAOSHIYAOCHANGSHIJIANDENGDAIWANGLUOCHUANSHUWANCHENG?YOUMEIYOUYIZHONGWANGLUOKEYIBUJINGGUOCPUJIUNENGFASONGSHUJU?

快3开奖结果查询DAANSHIYOUDE。TAJIUSHIRDMA (Remote Direct Memory Access)。ZAITANTAZHIQIAN,WOMENXIANRENSHIYIXIADMA。 

DMA(Direct Memory Access)SHIZHUJINEICUNHEWAIBUSHEBEIZHIJIANCHUANSHUSHUJUDEYIZHONGFANGSHI,XITONGJIANGNEICUNZUOWANXUNIDIZHIHEWULIDIZHIYINGSHEZHIHOU,JIUJIANGSHUJUCHUANSHUDEKONGZHIQUANJIAOGEILEWAIBUSHEBEIDEDMAKONGZHIQI,RANHOUSUOYOUDESHUJUCHUANSHUCAOZUODOUYOUWAIBUSHEBEILAIWANCHENG。ZHEZHONGFANGSHIKEYIJIESHENGCPUZIYUAN。

RDMAKEYIKANCHENGSHIYUANCHENGDEDMAJISHU,WEILEJIEJUEWANGLUOCHUANSHUZHONGFUWUQIDUANSHUJUCHULIDEYANCHIERCHANSHENGDE。RDMAYUNXUYONGHUTAIDEYINGYONGCHENGXUZHIJIEDUQUHUOXIERUYUANCHENGNEICUN,ERWUNEIHEGANYUHENEICUNKAOBEIFASHENG。QICHU,ZHIYINGYONGZAIGAOXINGNENGJISUANLINGYU,ZUIJIN,YOUYUZAIDAGUIMOFENBUSHIXITONGHESHUJUZHONGXINZHONGWANGLUOPINGJINGYUELAIYUETUCHU,RDMAZHUJIANZOUJINRENMENDESHIYE。

圖片來源知乎

從上圖的RDMA和傳統TCP/IP網絡對比中可以看出,RDMA的性能優勢主要體現在以下幾個方面:

  • Zero Copy:減少數據拷貝次數。由于沒有將數據拷貝到內核態并處理數據包頭部的過程,傳輸延遲會顯著減小。
  • Kernel Bypass和Protocol Offload:不需要內核參與。數據通路中沒有繁瑣的處理報頭邏輯,不僅會使延遲降低,而且也大大節省了CPU的資源。

RDMA網絡分類

MUQIAN,DAZHIYOUSANLEIRDMAWANGLUO,FENBIESHIInfiniband、RoCE、iWARP。

快3开奖结果查询1.InfinibandSHIYIZHONGZHUANWEIRDMASHEJIDEWANGLUO,CONGYINGJIANJIBIEBAOZHENGKEKAOCHUANSHU。InfiniBandJISHU GUIGEJIBIAOZHUNGUIFANYU1999NIANKAISHIQICAO,2000NIANZHENGSHIFABIAO,DANFAZHANSUDUBUJIRapid I/O、PCI-X、PCI-EHEFC。JIASHANGEthernetCONG1GbpsJINZHANZHI10Gbps,SUOYIZHIDAO2005NIANZHIHOU,InfiniBand Architecture(IBA)CAIZAIJIQUNSHICHAOJIJISUANJISHANGGUANGFANYINGYONG。QUANQIUHPCGAOSUANXITONGTOP500DAXIAONENGDECHAOJIJISUANJIZHONGYOUXIANGDANGDUOTAOXITONGDOUSHIYONGSHANGIBA。ZUIZAOZUOInfiniBandDECHANGSHANGSHIIBMHEHP,XIANZAIZHUYAOSHIMellanox。InfiniBandCONGL2DAOL4DOUXUYAOZIJIDEZHUANYOUYINGJIAN,SUOYICHENGBENFEICHANGGAO!

快3开奖结果查询2.RoCE JIRDMA over Converged Ethernet,TASHIYIGEYUNXUZAIYITAIWANGSHANGZHIXINGRDMADEWANGLUOXIEYI。QIJIAODIDEWANGLUOBIAOTOUSHIYITAIWANGBIAOTOU,QIJIAOGAODEWANGLUOBIAOTOU(BAOKUOSHUJU)SHIInfiniBandBIAOTOU。ZHEZHICHIZAIBIAOZHUNYITAIWANGJICHUSHESHI(JIAOHUANJI)SHANGSHIYONGRDMA。YOUWANGYOURENWEIRoCESHIJISHANGSHIMellanoxJIANYUIBGUOYUANGGUIERTUICHUDEYIZHONGDICHENGBENDECHANPIN,HEXINJIUSHIBAIBDEBAOJIAZAITONGYONGEthernetSHANGFACHUQU,YINCIDUIYURoCE,SHIJISHANGTADEERCENGBAOTOUYIJINGSHIPUTONGYITAIWANGDEBAOTOU。

3.iWARP SHIYIGEYUNXUZAITCPSHANGZHIXINGRDMADEWANGLUOXIEYI。IBHERoCEZHONGCUNZAIDEGONGNENGZAIiWARPZHONGBUSHOUZHICHI。iWARPZHIJIEJIANGRDMASHIXIANZAILETCPSHANG,YOUDIANSHICHENGBENZUIDI,ZHIXUYAOCAIGOUZHICHIiWARPDENICJIKESHIYONGRDMA,QUEDIANSHIXINGNENGBUHAO,YINWEITCPBENSHENXIEYIZHANGUOYUZHONGLIANGJI,JIBIANSHIANZHAOYIBANiWARPCHANGSHANGDEZUOFAJIANGTCP offloadDAOYINGJIANSHANGSHIXIAN,YEHENNANZHUISHANGIBHERoCEDEXINGNENG。MUQIANZAIZUOiWARPDEZHUYAOSHIIntelHEChelsioLIANGJIA,CONGXIANZAIDEGUANCHAChelsioLVESHENGYICHOU,ERIntelZEYOUDIANXIANGFANGQIiWARPZHUANTOURoCEZHISHI。

XUYAOSHUOMINGDEYIDIAN,QISHIBUGUANSHIiWARPHUANSHIRoCE,SHIJISHANGBINGBUSHIZIJIZHONGXINFAMINGLERDMA,ERSHILIYONGLEIBDESHANGCENGJIEKOUXIUGAILEXIACENGDESHIXIAN。CIWAI,SANZHONGSHIXIANFANGSHISHIYONGDEuser-space apiDOUSHIlibibverbs,ZHEYUANBENJIUSHIGEIIBYONGDE,XIANGDANGYUIBDEsocket。

RDMA應用場景

快3开奖结果查询RDMAZUOWEIYUANSHENGYUInfinibandDEJISHU,ZAOQIGUANGFANYONGYUGAOXINGNENGJISUAN(High Performance Computing)JIQUN。DANSHIYOUYUInfinibandHEJUEDADUOSHUHULIANWANGGONGSISHUJUZHONGXINLIMIANBENLAIDEEthernetGOUJIABUPIPEI,ERQIESHISHIQILAICHENGBENYEGAO,YOUQISHISHEBEICHENGBENHEYUNWEICHENGBEN,SUOYIGANGKAISHIBINGBUHUO。

快3开奖结果查询RDMAZUIJINZAIHULIANWANGQUANHUOQILAIYINGDEYIYUROCE(RDMA over Converged Ethernet)DECHUXIAN。DANROCEYEBUSHINALIDOUHAO,YINWEITABUSHIYIGEFANGZHISIHAIERJIEZHUNDETONGXUNBIAOZHUN。ROCESUIRANZUODAOLEJIANRONGYITAIWANG,DANSHIMUQIANDERDMAHUANSHIBUNENGZHIJIEJIANRONGYIYOUYINGYONG。ERQIERDMAQUESHAOLELEISIYUTCPDEsocketFENGZHUANGJIZHI,HENDUOSHIQINGBIANDEFUZAHUALE。BIRUNISHENZHIXUYAOKAOLVRDMAWANGKAHUANCUNRONGLIANGBUGOUDEWENTI。SUOYISHIXIANYIGE“TONGYONGDE”RDMAKUOZHANQISHIBINGBURONGYI。

DANGRAN,RUGUORUOHUA “TONGYONGDE”ZHEGEYAOQIU,NEIMESHIXIANQILAIHUANSHIXIANGDUIRONGYIDE。XIANGWEIRUAN、AWSZHEXIEZAISHUJUZHONGXINZHUIQIUGAOXINGNENGJISUANDEHyperscaleGONGSIZAIJINIANQIANJIUYIJINGYOULERDMABUSHU。YIXIEBUCHAQIANDEXIANGMU(RUAIXUNLIAN),YIJIZHENZHENGDUISHIYANYOUJIZHIYAOQIUDEYUNJUTOUMENYEKAISHIGUANGFANSHIYONG。

RDMA+GPU

快3开奖结果查询WEIRUANSHIMUQIANZAISHUJUZHONGXINDAGUIMOBUSHURDMADEDIYIJIAHyperScaleGONGSI。WEIRUANZAI2015/6/7/8NIANDESigcomm YOUDALIANGDELUNWENLAIJIANGRDMAZAISHUJUZHONGXINDEBUSHU。SUIHOUAWSYEBUSHIRUOBUSHULERDMA,TADEZUOFAHEWEIRUANYOUHENDADEBUTONG。WEIRUANWEILEAzureDEYUNHUANJINGDEDIYANSHIWANGLUO,CONG40GKAISHIGUIMOBUSHURDMA。WEIRUANWEIDAGUIMOBUSHURoCEv2SUOZUODEGEZHONGLIUKONGSUANFAYIJIYINGYONGDEYOUHUADUIYUZHENGGEYEJIEDOUSHIFEICHANGYOUYONGDE,DANSHITAMENZHUYAOTINGLIUZAICHUANTONGDEWANGLUOSHANGMIAN。

AWSZEBUTONG。ZAI2014NIANZHIQIAN,DIYANSHIWANGLUODEDABUFENCHANGJINGDOUSHISDS,HENDUODECUNCHUstartupGONGSI,SHIYONGPCIE FlashHERDMA WANGKALAIGOUJIANZIJIDEFENBUSHICUNCHUXITONG,BIJIAODIANXINGJIUSHIEMCSHOUGOUDEScaleI/O。2014NIANZHIHOU,YOUYUALexnetDECHUSEBIAOXIAN,RMDAHENvidiaYOULESHENDUHEZUO,LIYONGGPU Direct,ZAIGPUJIQUNZHONGKUAISUCHUANDISHUJUCHENGLERDMADELINGYIGEDAZHONGYINGYONG。

SUOWEIGPUDirect RDMA,JIUSHIJISUANJI1DEGPUKEYIZHIJIEFANGWENJISUANJI2DEGPUNEICUN。ERZAIMEIYOUZHEXIANGJISHUZHIQIAN,GPUXUYAOXIANJIANGSHUJUCONGGPUNEICUNBANYIDAOXITONGNEICUN,RANHOUZAILIYONGRDMACHUANSHUDAOJISUANJI2,JISUANJI2DEGPUHUANYAOZUOYICISHUJUCONGXITONGNEICUNDAOGPUNEICUNDEBANYIDONGZUO。GPUDirect RDMAJISHUSHIDEJINYIBUJIANSHAOLEGPUTONGXINDESHUJUFUZHICISHU,TONGXINYANCHIJINYIBUJIANGDI。

快3开奖结果查询NVIDIA DEGPUDirectYOUDIANDUIDIANHERDMAZUJIANZUCHENG,JIANRONGDUOZHONGTONGXINHULIANCHANPINYUAPI,BAOKUONVIDIA NVLink?HENCCLYIJIOpenMPIHEUCX。

YOUYUGPUDirect RDMADECHUXIANJINYIBUJIANGDILETONGXINDEYANCHI,ZAI re:Invent 2018 SHANG,AWSFABULE Elastic Fabric Adapter (EFA),TASHIYIZHONGYONGYU Amazon EC2 SHILIDEWANGLUOJIEKOU,RANGKEHUNENGGOUZAI AWS SHANGDAGUIMOYUNXINGXUYAOGAOJIBIESHILIJIANTONGXINDE HPC YINGYONGCHENGXU,LIRUJISUANLIUTIDONGLIXUE、TIANQIJIANMOHEYOUZANGMONI。JIEZHUEFA,XIAOXICHUANDIJIEKOU(MPI)DEGAOXINGNENGJISUAN(HPC)YINGYONGCHENGXUHENVIDIA Collective Communications Library(NCCL)DEJIQIXUEXI(ML)YINGYONGCHENGXUKEYIKUOZHANDAOSHUQIANGECPUHUOGPU。EFADINGWEIHENQINGCHU,HPCHEFENBUSHIJIQIXUEXIDEXUNLIAN。CHUANTONGDENitroXINPIANZAISHENGJILE25GDESerdesZHIHOU,LIYONGlibfabricDESHENGTAISHIXIANLEBUFENDERDMADEGONGNENG,KEYIMANZUAWSSHANGDEHPCHEMLDEYEWUXUQIU。ZHEGEHEGUONEIDEDACHANGDEXIANGFALEISI。

快3开奖结果查询CONG2016NIANKAISHI,ALIBABAJIUTOURUZHUANXIANGYANJIU,YIGAIZAORDMA、TIGAOCHUANSHUXINGNENG。CONGWANGKADICENGKAISHISHEJIMANZUDAGUIMOYINGYONGDEWANGLUO,BINGJIEHEALIZIYANJIAOHUANJISHIXIANXINGNENGZUIDAHUA,ZUIZHONGJIANCHENGQUANQIUZUIDAGUIMOSHUJUZHONGXINNEIDE“GAOSUWANG”,SHIDEJIQUNJIDADITUPOLECHUANSHUSUDUPINGJING,BINGJIANGSHIYANXIANZHUJIANGDI90%。YI2019NIANTIANMAOSHUANG11WEILI,JIYURDMAWANGLUOJISHUDEYUNCUNCHUHEDIANSHANGSHUJUKUFUWUQIKEYICONGRONGDIYINGDUIFENGZHILIULIANGKAOYAN。

RDMA+AI

ZAIZUIJINHUOREDEAIXUNLIANKUANGJIAZHONGRDMAYETEBIECHANGJIAN。BAIDU、TENGXUN、JINGDONG、XINGYUNDENGGONGSIDOULIYONGRDMAJISHUZAITensorflow、Paddle、AmberDENGPINGTAISHANGZUOLEERCIKAIFA。

快3开奖结果查询BAIDUGONGSIZAI2014NIANQIANHOUKAISHIYINRURDMAWANGLUO,XIANHOUBUSHULElnfinbandJIQUNHERoCEv1JIQUN。2015NIAN,BAIDUFENBIEZAISZWGJIFANGHEYQ01JIFANGDAGUIMOBUSHULERoCEv2JIQUN,FENBIECHENGZAILESHENDUXUEXI,YUYINSHIBIEHEZIRANYUYANCHULIDENGXIANGGUANDEJIQIXUEXIRENWU。MUQIANRDMAJIQUNZONGTIGUIMOWEI600TAIZUOYOU,ZHESHIGUONEIZUIDADEYIGERoCEv2WANGLUO。

快3开奖结果查询QIZHONGYUYINSHIBIEXUNLIANHENLPJIQIFANYI,SHIYONGOpenMPIZUOWEIJICHUKU,40G RDMATONGXINXINGNENGXIANGDUIYU10GTCPWANGLUOTISHENGLE1GESHULIANGJI。PaddleTUXIANGXUNLIANSHIYONGLEISocketKU,YEYOUBUCUODEXINGNENGSHOUYI。WEILEGENGJIAZHIGUANDELIJIE,DUIYUOpenMPI BenchMark,40GRDMAXIANGDUIYU10GTCPNENGGOUTISU1-2GESHULIANGJI。

YOUTENGXUNWEIXIN-XIANGGANGKEJIDAXUERENGONGZHINENGLIANHESHIYANSHI (WHAT Lab) YANFADEAmber,TASHIYIZHONGSHENDUXUEXIJISUANKUANGJIA,BINGZHICHIDAGUIMOFENBUSHIBINGXINGSHENDUXUEXIJISUAN。QISHUJUCHUANSHUMOKUAIZHONGDE RDMA GONGNENG,YOUXIANGGANGKEJIDAXUECHENKAIJIAOSHOUKAIFADE RoCE TIGONGJISHUZHICHI,SHIDE Amber NENGGOUCHONGFENSHIYONGYINGJIAN RDMA XINGNENG,SHIXIANDIYANCHI、GAOTUNTULIANGDEMOXINGCHUANSHU。

JINGDONGRENGONGZHINENGYANFATUANDUIZAIFENBUSHIDEMOXINGXUNLIANCHANGJINGZHONG,YESHIYONGLERDMAJISHU,ZHENDUIMOXINGWENJIANDEGAOXINGNENGCHUANSHU,MANZULEFENBUSHIXUNLIANDEXUQIU。

JINGDONGITZIYUANFUWUBUDEYINGJIANXITONGBUZAICESHIRDMASHIYEKANDAO,ZAIVMDEFUWUZHONG,ZAITONGYANGWULIFUWUQIDEQINGKUANGXIA,LIYONGRDMA LAIZUOSHUJUCHUANSHU,KEYIJIANGJIHU100%DECPU ZIYUANDOUTIGONGGEIJISUAN。JIANGDILECPU ZAITONGXUNZHONGDEZHANYONGLV,YONGHUJIUKEYILIYONGZHEXIECPU ZIYUANLAIZUOGENGDUOJISUANHUOTIGONGQITADEFUWU,ZHEYEXIANGDANGYUZENGJIALEVMDESHULIANG。

HUANYOUXINGYUNClustarPINGTAI,LIYONGRDMAWANGLUOJIEJUELETensorFlowDENGXUNLIANKUANGJIAZAIZHICHIFENBUSHIXUNLIANFANGMIANDEXINGNENGPINGJING。TONGGUORDMAXIAOCHUDUOGPUKUAJIEDIANTONGXINDEWANGLUOPINGJING,XIANZHUJIANGDILEXUNLIANRENWUZHENGGEZHOUQIZHONGDETONGXINHAOSHIZHANBI,TIGAOLEGPUJIQUNJISUANZIYUANLIYONGLVHEXUNLIANXIAOLV,YEWEIJIQUNHENGXIANGKUOZHANDAOGENGDAGUIMOSHIDEXIANXINGJIASUBITIGONGLEBAOZHENG。SHEJUNQUANXIANSHENGDUICIZUOLEFEICHANGSHENGDONGDEMIAOSHU:“YINGYONGJIUXIANGSHIQICHE,YINGYONGDERUANJIANYOUHUAKEYITISHENGFADONGJIDEXINGNENG,RDMAJIUSHIJIANGYUANLAIDESICHEDAOBIANCHENGLEBACHEDAO,WEIYINGYONGXINGNENGDETISHENGCHUANGZAOLEGENGDADEKONGJIAN。”

RDMA設備與k8s集成

YAOZAIKubernetesZHONGSHIYONGrdmaSHEBEI,SHOUXIANSHITONGGUOdevice pluginCHAJIANJIZHI,RANGk8sNENGGOUSHIBIEHESHIYONGrdmaSHEBEI。QICI,rdmaWANGKAJIBENSHANGDOUZHICHISR-IOVJISHU,LIYONGJIYUGAIJISHUDEKubernetes CNICHAJIAN,KEYIGOUJIANCHUWANZHENGDERONGQIWANGLUO。

DANGQIANZHICHIRDMADEWANGKAZHUYAOSHIintelYUmellanoxLIANGGECHANGJIA,GEZIJUNTIGONGYOUDUIYINGk8s device pluginYUSR-IOVWANGLUOcniCHAJIAN。

ZAIHENCHANGYIDUANSHIJIAN,WANGLUODOUBUSHISHUJUZHONGXINXINGNENGDEPINGJING,JIYU”DADAIKUAN”DEWANGLUOSHEJIKEYIMANZUYEWUYINGYONGXUQIU。DANJINXIENIANLAI,FUWUQIJISHUDEKUAISUFAZHANDAIDONGLESHUJUZHONGXINJISUAN、CUNCHUNENGLIDEFEISUTIGAO,ERRoCE、NVME over FabricDENGRDMAJISHUJIANGSHUJUZHONGXINDEXINGNENGPINGJINGZHUANYIDAOLEWANGLUOCE。TEBIESHIDUIYUHPC、FENBUSHICUNCHU、GPUYUN、CHAORONGHEJIAGOUZHEXIEJIYURDMADEXINXINGYINGYONGLAISHUO,WANGLUOSHIYANYIJINGCHENGWEIZHIYUEXINGNENGDEZHUYAOYINSU。YINCI,KEYIYUJIANDAO,WEILAISHUJUZHONGXINDESHEJIMUBIAOHUIZHUBUCONGDAIKUANQUDONGZOUXIANGSHIYANQUDONG,RUHEGOUJIANDISHIYAN、WUSUNDEDAXINGYITAIWANGSHUJUZHONGXIN,BINGJIANLIWANZHENGDEHUANCUN、SHIYANJIANKONGJIZHI,HUISHIYEJIECHANGQITANSUODEMUBIAO。

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.


  • 本站原創文章僅代表作者觀點,不代表SDNLAB立場。所有原創內容版權均屬SDNLAB,歡迎大家轉發分享。但未經授權,嚴禁任何媒體(平面媒體、網絡媒體、自媒體等)以及微信公眾號復制、轉載、摘編或以其他方式進行使用,轉載須注明來自 SDNLAB并附上本文鏈接。 本站中所有編譯類文章僅用于學習和交流目的,編譯工作遵照 CC 協議,如果有侵犯到您權益的地方,請及時聯系我們。
  • 本文鏈接http://cslinyu.com/23769.html
分享到:
相關文章
條評論

登錄后才可以評論

SDNLAB君 發表于19-12-11
0