Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to 云网锦绣 SDN实战研讨会
Similar to 云网锦绣 SDN实战研讨会 (20)
More from Hardway Hou
More from Hardway Hou (20)
云网锦绣 SDN实战研讨会
- 1. 云网锦绣 CONNECTING CLOUDS WITH BROCADE SDN实战研讨会 Cloud Connect 2016
- 2. 云网锦绣 连接云的力量 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 4. 还记着的三件事儿…. 1. 好多的Cloud！ Connect呢？ 2. 近距离接触 AWS 3. Hybrid Cloud, the First Contact… Cloud Connect 2010
- 5. 今日头条… 之一
- 6. 云中有网 • 云内 (Fabric, NV/NFV/SDN) • 云间 (DCI, SD-WAN, Hybrid Cloud, Multi- Cloud/Broker, Cloud Direct/Exchange) 网中有云 • NFV需要VIM • Distributed (Mini-, Micro-) Cloud, MEC (Fog Computing) 云网合一 两类资源，一样理念 最佳效益，最佳体验 2016
- 7. 锦 1995
- 8. 2016 一个新型的网络公司 交换路由 软件网络 虚拟化 控制/自动化 分析 移动网络 无线 新边缘数据中心 Fabrics 存储网络 IP Fabrics Campus Products vRouter vADC SDN Controller Network Automation Machine Learning NVA AMP Security vEPC MEC WLAN Small-cell Cloud- RAN CORD FOG IOT LWA MEC Small Cell vCPE
- 9. Brocade 网络创新平台框架 网络虚拟化 • BGP-EVPN • VTEP • Service Chaining 高性能硬件基础设施 • DC SAN • DC Ethernet and IP Fabrics • DC Routing • Campus Switching • Wireless & Small Cell • SD-WAN, SDN-WAN SDN 应用 / 自动化 • Brocade SDN Controller, Orchestrator • Network, Security, Analytics, Mobility Apps • DevOps, BWC, StackStorm 虚拟网络服务 （NFV） • vRouter, vNAT, vADC, vFW, vCPE, vEPC… • Vyatta Network OS, MEC 虚拟服 务 网络 Brocade Virtual Services Platform Brocade App Platform/SDN Controller/Orchestration Open Interfaces vFW vRTR vNAT vVPN vIPS vADC vTAP Analytic s Security Network Mobile 应用 Brocade Data Fabric 数据矩阵服务 • Proxy Data Src, Realtime Streams, Pub/Sub 3rd Party SIEMs LAN/WAN/Cloud 控制/编排 /分析 Machin e Learnin g
- 11. 生态 Symbiosis
- 12. Collaboration With your Team ... but also with your Competitors 竞合 Coopetition
- 13. Connecting Clouds with Brocade SDN NFV Cloud Networking
- 14. SDN实战团 衷心感谢！
- 15. 议程 • 开场: 云网锦绣 - 连接云的力量 （张宇峰，Brocade大中华区CTO） • “大云“之端 – 中国移动OpenStack公有云虚拟应用交付的集成编排（李睿，Brocade Sr. SE) • 开放创新 – 中国银联对下一代金融云平台中SDN/NFV架构及应用探索 （周雍恺 中国银联研究院博士后研究院，张泽鹏 Brocade Sr. SE） • 应用为王 – 互联港湾混合云/多云环境下的虚拟应用交付部署及自动扩展 （任志远 互联港湾CEO，余欣 Brocade Sr. SWN SE） • vEPC & NVA网络可视分析 – 慧岭科技在SDN/NFV使能移动虚拟运营商的初步实践 （智勇 慧岭科技 CTO） • 敏捷之魂 – LinkedIn和Netflix钟爱的事件驱动云数据中心工作流级自动化 （储子瑞 Brocade资深软件网络工程师，梁博 九州云资深架构师及培训负责人） • MANO驱动的NFV/SDN服务编排 – 运营商云网合一开放生态实战方案分享 （张宇峰， 龚永生 九州云资深架构师） • Panel：DevOps – 颠覆文化，开放边界，蜕变产业，演进生态 （任志远，余欣，智勇，梁博，龚永生，方辉 SDNLAB联合创始人, 张宇峰） • 尾声: 回到未来 – 软件，机器，赛跑，学习（张宇峰）
- 16. 李睿，Brocade Sr. SE Sept 20th, 2016 大云之端 中国移动公有云LBaaS的集成编排 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 17. $- $500 $1,000 $1,500 $2,000 $2,500 2014 2015 2016 2017 2018 2019 ADC Market Disruption (Billions $) The ADC Market is Changing • Rapidly changing market place – Shift from propriety hardware to virtualized Intel servers + software – Driven by private / public cloud movement – Telco Service Providers pushing shift to software through NFV – Large savings in CAPEX and OPEX through migration from hardware to virtual • Existing hardware ADC incumbents not well suited to new market – Trying to protect hardware revenue while moving to software – Legacy software designed for ASICs must be “translated” to run on Intel servers – Huge code size a barrier to virtualization 17 35% vADC CAGR -3.7% HW CAGR Source: IDC
- 20. 云中的LBaaS • 公有云 以租户自服务为主，Portal提供配置模板。 建设初期提供简单功能，后续提供逐步增加各功能。 LBaaS/LBaaSv2 功能/性能/可靠性/安全性…… 更依赖于一个优秀的编排架构 • 私有云 需针对企业常见应用提供优化的解决方案（Exchange/Web logic/IAG/Sharepoint…….） 可由维护人员代维 20
- 21. 开源的vADC • 没有购买成本 • 开源社区支持 • 某些功能不够完备（如缺少HA和许多高级功能，优化、带宽管理、安全等） • 需要做面向商业化的定制开发 系统更新，bug修复。 随着业务的发展，可能需要持续的对功能和性能进行开发和优化。 维护人员的技术水平，尤其是传统IDC向云IDC转换时。
- 22. 商业化vADC • 成熟度高，功能完善。 基本不需要再做功能级别的开发，便于今后业务拓展。 运行稳定 • 性能已优化 采用DPDK等技术，可达120Gbps吞吐。 • 完善的服务保障体系 Tech中心/本地support/厂商的DevOps • 面向云编排提供解决方案 vADC Plugin MANO
- 23. 移动公有云中LBaaS的技术要求 • 采用Plugin方式和Openstack集成 • 初步要求具备Openstack neutron LBaaSv2上的功能 • LBaaS不能有单点故障 • 性能要求(500个计算节点规模情况下） 支持50个用户的并发创建/配置/修改/删除LBaaS（创建并删除的内容包括 loadbalancer/listener/pool/member/healthmonitor） • 可被管理和监控 • 进一步的DevOps能力
- 24. 博科vADC在移动公有云 24 • 支持Neutron LBaaSv2 • 功能完备（SLB/GLB/Cache/WAF……) • 高性能（可达120Gbps吞吐） • 高可用（Cluster/Multi-Site） • 扩展能力 Scale-out/Scale-in License动态分配 • 定制化开发
- 27. 中国银联电子支付研究院 Sept 20th, 2016 开放创新 中国银联对下一代金融云平台 SDN/NFV架构及应用的探索 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 30. 产业/企业 政府监管部门 • 鼓励金融企业上云，建云 • 金融企业和互联网企业都在探索金融云 金融云发展动态 11月15日国务院常务会议明确提出“对云计算 关键技术进行突破，加快云计算的应用与研发的 总体要求”，支持云计算关键技术研发和重大项 目建设。 2011 2012 2013 201420152016 阿里云在京正式宣布对外提供金融 云计算服务,该项服务也被称之为“聚 宝盆”,计划面向2000多家区域银行 以及证券、保险、基金等金融机构提 供云计算服务。 银联《基于云计算的电子支付和电子商务综 合服务平台》项目得到发改委正式批复，列 入2011年中央财政战略性新兴产业发展专项 资金补助计划（国家云计算示范工程项目） 国际上PayPal、英国巴克莱银行开始将 自己的金融生产应用迁移到云平台上运行， 比表示云平台有效降低了金融成本，以及 加速了业务敏捷性 国务院印发《关于积极推进“互联网+”行动 的指导意见》，明确指出“探索互联网企业 构建互联网金融云服务平台”，“支持银行、 证券、保险企业稳妥实施系统架构转型，鼓 励探索利用云服务平台开展金融核心业务”。 银监会印发《中国银行业信息科技 “十三五”发展规划监管指导意见》 指出，银行业金融机构要“积极开展 云计算架构规划”，制定云计算标准， 联合建立行业云平台，主动实施架构 转型“。
- 31. 互联网+金 融 理财 支付 征信 保 险 手机银行 手机钱包 证券 手机支付 移动受理终端 未来 金融科技 互联 网金 融 移 动 金 融 过去数年经验 当前与未来 • 正在快速发展的“互联网+金融”，以及未来“车联 网金融”等金融创新新形态，对金融云技术发展也在 不断提出新的要求 • 云计算新技术应用确实支撑起了“互联网金融”、 “移动金融”等创新应用的发展，对金融云“落地下 雨”可行性有了明确判断 金融云是金融创新的基石，金融创新推动金融云发展
- 32. 金融云的理解 金融属性 • 合规性：符合金融主管部门的合规性要求，如等级保 护要求、IT审计等 • 安全性：符合金融行业国家金融安全的要求，具备完 备、严格的安全防护能力 • 可用性：满足金融业务国家重大民生服务的业务可用 性要求，具备高等级的业务连续支撑能力 • 服务能力：相比传统架构可以有效支撑金融创新，具 备适应业务发展的敏捷性 • 金融架构特征：必须满足“两地三中心”等金融行业 特殊的金融架构 +
- 34. 银联云的发展历程与定位 2009 2011 2015 预研启动，启动对云计算的跟 踪研究 国家立项， 10 月 国家 云计算示范工程项目获 得批准，全国 15 家入选 单位中，唯一的国有企 业，唯一的金融企业 2014 2012 2013 原型试用，自主研发的生产云基础平台完成原 型建设，当年12月第一个云应用生产上线。 试点应用，核心关 键技术取得突破， 服务能力较为完善， 公司范围内各部门 开始推进云业务试 点。 规模应用，公司级重点项目与产品应用系 统均选择在云计算平台上部署落地。私有 云已完成建设，突破千台级物理服务器规 模能力。 深入研究，下一代基 于SDN技术的云平台， 对主流商业与开源方 案进行大规模测试。 2016 建立了金融创新实验云服务平台，加深 对行业合作伙伴的云计算合作，探索行 业云服务模式，助力金融行业区块链、 生物识别等创新应用的发展
- 36. 银联云SDN网络能力与模型研究 交换机 （SPINE） 交换机 （SPINE） 交换机 （LEAF） Server Leaf Service Leaf 防 火 墙 负 载 均 衡 SDN控 制器 虚拟交换机 VM VM 5. 网络功 能标准化集 成 4. 数据平 面性能加 速 3. SDN网络控制 （包含服务链， 网络运维调试等） 1. Underlay 二层组网架构 六大关键能力，三大场景模型 A. 金融大区组网 管理模型 B. 以应用为中心的 网络编排模型 C. 虚拟两地三中心 多租户管理模型 Border Leaf 2. Overlay 虚拟化组网 VM VM 6. 跨数据中心组网 广域网连接 DCI链 路
- 37. 一、网络功能的集成测试 # 方案 详细说明 1 软件模式－开源路由方案 OpenStack Neutron L3 2 软件模式－商业路由方案 Vyatta 5600 (非SR-IOV模式) 3 混合模式－商业路由方案＋硬件加速 Vyatta 5600 (SR-IOV模式) 4 硬件路由 硬件交换机
- 38. 网络功能的集成测试——功能对比 方案 A方案(OpenStack原生纯开源) B方案（B厂商纯软，紧密集成） B’方案（B厂商纯软，松散集成） 分类 功能项对比 通过 Neutron 调 用 L3 和 开源 VNF （FWaaS/VPNaaS/LBaaS） 通过Neutron调用VNF 并配置，不 使用VNF自带配置界面进行配置 不 通 Neutron 而 是 通 过 Nova 调 用 VNF 虚 拟 机 ， 通 过 B 厂 商 自 带配 VNF置界面进行配置 三层路由 网关功能 √ √ √ 路由功能 √ √ √ OSPF支持 x x √ PolicyBasedRouting x x √ QoS √ √ √ IGMP x x √ MLD x x √ PPP-Based Encapsulations x x √ PIM x x √ RIP x x √ RIPng x x √ Multicast Routing x x √ iBGP x x √ eBGP x x √ 主备高可用 √ √［注1］ √［注1］ 主主高可用 x √［注1］ √［注1］ 防 火 墙 四 层防护 无状态防火墙 √ √ √ 状态防火墙 √ √ √ 双向地址转换NAT √ √ √ 目标地址转换DNAT x x √ 源地址转换SNAT x x √ 高可用 AS x x √ 高可用 AA x x √ VPN IPSec √ √ √ Site-2-Site VPN功能 √ √ √ OpenVPN支持 x x √ Remote Access VPN x x √ 高可用AS x √［注1］ √［注1］ 高可用 AA x √［注1］ √［注1］
- 39. 网络功能的集成测试——性能对比 三层路由性能压测数据 延时(MS) 租户网 络 每个网络模拟虚 拟机个数 帧大小 (Bytes) 模拟负载 (%) 1 纯开源方案 (Neutron I3) 2 商业软件路由方案 （Vyatta，无 SRIOV加速） 3 商业软件路由方案 （Vyatta，有SR－ IOV硬件加速） 4 硬件路由方案 （硬件路由BNT G8124-E） 2 10 64 30 25.3 12.5 0.55 0.00185 2 10 64 50 16.2 11.5 0.537 0.00185 2 10 64 100 18.2 12.3 0.998 0.00186 2 10 1024 30 52.2 8.9 0.103 0.00189 2 10 1024 50 40.9 10.4 0.136 0.00189 2 10 1024 100 40.5 10.7 0.53 0.0019 测试总结 • 相对于硬件方案（方案4），基于X86的开源方案（方案1）性能上相对硬件有较大的性 能差距，方案2（商业）在性能上有一定改进，主要原因还是由于二层还是通过了OVS 开源模块造成的瓶颈。 • 我们的改进方案是混合模式（方案3），通过 SR-IOV模式，让承载虚拟路由的虚拟机环 境直接访问网卡硬件，将软件方案提升，但是和纯粹硬件方案相比还是有一定性能差距 。 • 后期我们将继续测试通过调整CPU／内存参数，再加入Intel DPDK来提高加速效果。
- 40. 二、云网络环境下的网络监控系统• NPM （Network Performance Monitor） – 将网络中传输的报文进行镜像，辅助网管人员在后台针 对网络流向、吞吐性能、故障排错以及安全异常等进行 分析 – 一套优秀的流量监控分析系统是网络运维的重要支撑 三级设备构成： 1. 汇聚设备； 2. NetScout探针设备； 3. NetScout控制分析服务器； 部署规模： • n台汇聚设备+8台探针设备+1台 控制分析服务器 监控范围： • 转接核心系统、 • 外网出口流量 • 部分的内网流量与云计算系统中的 流量； • 后端的报文存储容量约为十天左右 传统网络监控系统
- 41. 云网络环境下的网络监 控系统 云网监 控 • 镜像能力全网覆盖 • 按需细粒度镜像 • 云网信息关联 • 流量大数据分析 虚拟化的租户流量镜像 • 成本高：云网络流量大，不可能为每个 租户、每条链路都配置探针设备 • 无法适应云网络的动态性：云中的租户 虚拟机是可以全网迁移的，传统吉祥无 法进行镜像链路追踪 • 无法识别云网络的流量：带VxLan报头 的租户流量 • 与云平台中的元数据信息无法关联 传统NPM在云场景下的局限性
- 42. 云网监控的组成部分 网络数据 提取 • 基于OpenStack的网络流量统计 数据提取； • 基于SDN流量镜像的网络端口捕 获； 报文存储 分析 • 报文处理存储； • 报文数据分析； • 分析结果展示； 云网关联 分析 • 云平台元数据的对应关联； • Underlay网络关联分析；
- 43. 整体架构
- 44. 研究点1.1：云网络镜像 • 基于SDN的流量镜像 匹配域 动作集合 正常转 发 镜像复 制+ 细粒度匹配 OpenFlow流 表项 • 建立镜像隧道 • 添加ERSPAN 报头 • 打上时间戳 • 复制转发 • 能够镜像全网的 Overlay流量，复用 原有的Fabric，无需 布置额外的镜像线路 • 细粒度镜像，可以精 细到某一租户/某一 应用/某一网络流 • 租户按需镜像 特点 • 镜像流量以ERSPAN 的方式进行导出，会 影响到正常转发流量 需要作相应的优化研 究，使得其对正常流量 的影响降到最低。 此外未来的万兆网络 以及40G上联也为此提 供了带宽保证 可能的问题 • 镜像报文流的控制 调低镜像报文的优先 级 选择空闲的物理链路 传输 • 后端接收的DPDK加 速 • 租户报文的精细化导 出 技术攻关点
- 45. 研究点1.2：统计流量提取 • 技术背景 – 研究点1.1中的“云网络镜像”采用的仍是较为传统的重量级镜像方 案 – 在某些分析场景中，仅仅需要一些针对报文元数据的统计信息即可 反映出出网络的状态 – SDN网络中，存在更加轻量级的方法，也即基于“网络流”的报文 统计数据提取 • 实现原理 – SDN的数据层面是基于“流表”进行转发的，对于每个“网络流”， SDN交换机会进行针对性的统计。SDN控制器只需要定期轮询交换 机的流表统计信息即可 • 优点 – 减轻镜像开销，比较适合虚拟交换机的场景 – 直接从报文源头获取数据并实时统计，更好地感知网络状态 – 与1.1的流量镜像形成高低搭配
- 47. 研究点2.2：报文数据分析展示 基于研究点2.1所形成的报文数据的公共处理平台，可以根据后台分析的需求，定制 化开发相应的分析功能 • 网络流吞吐量、时延、 抖动等指标的统计与 分析 • 与生产网络的业务端 策略进行关联，分析 链路的质量状况 • 网络侦错 网络质量分析 • 识别金融行业类特有 的交易支付报文 • 分析统计各笔交易的 时延，以及交易频度 • 对于每笔交易所经过 的所有应用子系统的 链路进行回溯与关联 查询 应用质量分析 • 通过深度学习算法识 别出具有攻击嫌疑的 网络报文 • 与控制器端的网络策 略进行关联，发现异 常的流量 报文安全分析
- 48. 研究点3.1 云平台数据关联分析 • 与云平台的应用定义高度整合，镜像出的网络流可以关联对应 到某个租户的某个应用 • 自动为租户起Wireshark虚机对某条链路进行监控流量的显示 • 链路流量经过网络设备前后的比对 应用连 接定义 链路 抓包
- 49. 研究点3.2 Underlay网络关联排错 • Overlay的 网络流量与 Underlay网 络经过的链 路进行关联 • 辅助网管人 员进行排错 分析 Overlay 网络 Underla y链路 硬件网络监控 云网SDN监控
- 54. 金融SDN评测指标选取 网络基本性质 • 国内标准 • 国际标准 SDN网络特性 • 控制平面集中管 理 • 开放接口 • 网络虚拟化 金融行业特殊需求 • 合规性要求高 • 业务多分区 • 多中心的灾备模 式 • 可靠性要求高 目前已经整理出360个左右的评测指标 指标选取思路
- 56. 谢谢！
- 57. 任志远 互联港湾CEO 余欣 Brocade Sr. SWN SE Sept 20th, 2016 应用为王 互联港湾混合云/多云环境下的 虚拟应用交付部署及自动扩展 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 61. SDN化-实现流量的伸缩控制
- 63. 业 务 需 求 的 演 变 传统ADC控制平面单点化，资源固化（扩展性差），运维有效性差、成本过高 以VADC灵活、自动化的特性及全局调度的能力构建全新混合云业务调度架构 业务 跨平台 （多云+ 多IDC） 部署 成本 降低/ 可靠性 提高 可扩展+ 负载分担 +故障快 速恢复 产生 混合云资 源调配+ 流量等联 动需求 业务的发展越来越依赖于ADC 越来越多的应用业务由原来的单节点部署发展为扩平台业务部署
- 64. VADC应用交付需求 1 2 4 云化管理，跨云管理 部署简便，高可用性 全局监控、预先策略 维护简便，降低成本 交付及时有效、扩展性强 降低运维成本，提高部署效率. 提升用户业务流的性能稳定性、 持续性与安全性。 3
- 65. 访问者 访问 者 访问 者 ①访问压力触动阈值时，VADC自动 依据设置在每个云节点生成不同数量 的新虚拟主机（视节点资源情况）， 业务访问流量也可根据标签调配引流 到不同的业务节点 ②访问高峰期过后， VADC自动释放多余资 源， 确保资源不浪费 访问者访问 者 访问 者 访问者 访问者 访问者 互联港湾cloud 公有云服务商B公有云服务商A VADC VADC的高可扩展性功能 IDC数据中心 Router
- 67. Beijing Internet Harbor Technology Co., Ltd. Beijing Office: F11, Building 12, Wanda Plaza, No. 93 Jianguo Road, Chaoyang District, Beijing ，P.R.C Tel : (86)10-58203300 Fax : (86)10-58205536 Shanghai Office : Room 809, No.336 Central Tibet Road , Huangpu District ,Shanghai , P.R.C Tel : (86)21-60938033 Fax : (86)21-60932938 http : //www.idccun.com 24*7 Hotline : 400-670-8875
- 68. 传统业务部署 • 网络搭建基础网络 ？天 • SA安装系统 ？天 • PE部署应用 ？天 • 网络部署VIP ？天 69 SA安 装系统 PE部 署应用 网络部 署VIP 基础 网络 扩容 缩容 扩容 缩容 扩容 缩容 APP 交付 扩容 缩容
- 70. Private NetworkPublic Virtual IP 应用于应用交付系统 Select * from passwords BASH: Rm –r / ACCESS GRANTED ?? ? JPEG CSS JS HTML www www APPS APPS DB DB Web + Application Servers CSS JSJPEG JPEG CSS JSHTML JPEG CSS JS
- 72. © 2016 年博科通讯系统公司版权所有。 73 DEMO
- 78. 79 vADC 配置 1.在global settings 里面设置EC2（aws）的可用区和登陆信息: 2. global settings 里面设置EC2 的密码 ec2!secret_access_key:不是密码是单独的key. 在用户管理中有。但是， 用户必须先设置一个密码。
- 79. 80 可用区URL获取: https://docs.aws.amazon.com/general/latest/gr/rande.html Region Name Region Endpoint Protocol US East (N. Virginia) us-east-1 ec2.us-east-1.amazonaws.com HTTPS US West (N. California) us-west-1 ec2.us-west-1.amazonaws.com HTTPS US West (Oregon) us-west-2 ec2.us-west-2.amazonaws.com HTTPS Asia Pacific (Mumbai) ap-south-1 ec2.ap-south- 1.amazonaws.com HTTPS Asia Pacific (Seoul) ap-northeast-2 ec2.ap-northeast- 2.amazonaws.com HTTPS Asia Pacific (Singapore) ap-southeast-1 ec2.ap-southeast- 1.amazonaws.com HTTPS Asia Pacific (Sydney) ap-southeast-2 ec2.ap-southeast- 2.amazonaws.com HTTPS Asia Pacific (Tokyo) ap-northeast-1 ec2.ap-northeast- 1.amazonaws.com HTTPS EU (Frankfurt) eu-central-1 ec2.eu-central- 1.amazonaws.com HTTPS EU (Ireland) eu-west-1 ec2.eu-west-1.amazonaws.com HTTPS South America (São Paulo) sa-east-1 ec2.sa-east-1.amazonaws.com HTTPS
- 80. 81 debug 设置(可选) system->global setting ->logging
- 81. 82 配置Cloud Credentials 配置位置: (catalogs cloud credentials) 配置文件位置: /usr/local/zeus/zxtm/conf/cloudcredentials 这里的access key id and secret access key:和前面的一样。
- 82. 83 配置pool 中的 autoscaling 信息. 配置为: services->Pools->autoscaling 配置文件位置: /usr/local/zeus/zxtm/conf/pools
- 84. 智勇 慧岭科技 CTO vEPC & NVA网络可视分析 慧岭科技在SDN/NFV使能 移动虚拟运营商的初步实践
- 86. 关于慧岭科技
- 87. Virtual EPC市场预测 预计到2018年，接近80%的EPC市场将虚拟化，约合$1.5B • EPC系统：移动网络中具有战略意义的系统 • 当前有约11家EPC系统供应商，没有单个供应商的份额超过 25% • 供应商Top Five: Cisco, Ericsson, Samsung, Huawei, ZTE EPC Total Available Market
- 88. 移动网络可视化的巨大需求 部署LTE所面临的最大挑战 • 缺乏可扩展的分析系统 • 现有体系无法满足带宽的增长 • LTE业务回报模型不明确 • 难以提供端到端的服务检测 • 服务能力和客户体验下降 • 难以为创建和部署新服务提 供有效的数据依据
- 89. Brocade VCM Architecture 97 • 每个节点分别处理部分工作 • 刚性的系统划分导致规划、扩展和业务监测方面的挑战 • 系统边缘重复的封装和解封装导致效率的下降 接口封装和解封装 Protocol handling Event loop Session context Mobility context Forwarding functions State machine 控制平面 会话状态 数据平面 MME HSS PGWSGW Brocade VCM架构 Session context Mobility context Protocol handling Event loop Session context State machine Forwarding functions Protocol handling Event loop Session context State machine Protocol handling 打破系统之间的边界节点之间的封装和解封装不再需要 只留下各个功能模块 去除重复的功能模块 压缩通用的功能模块 按照功能组，如控制、数据转发和状态等， 进行模块重组 接口
- 90. Brocade VCM Architecture Any Hardware Any Hypervisor VCM Platform External Interfaces Control Call Handlers IP Flows Data Processors Linux State SessionsSubscrip tion Device Provisioning Appliance Management Configuration Network Management CLI/GUI Network Visibility Mgmt./APIs 数据库 层面 服务逻 辑 层面 接口 层面 基础设施 管理层面 物理服务器 • VCM的计算节点提供了完整的移动网功能，包括控制平面，会话平面和数据转发平面 • 能够将移动核心网的相关负载（包括信令、会话和数据）直接映射到对应的节点上 • 节点的集群能够根据运负荷情况和行效率独立实现性能扩展和资源回收
- 92. 大数据实验室VCM + NVA 整体设计 FemtoCell PDN (Internet, carrier service, etc.) SGi UE S1 All components in Guiyang Open Lab S5 vEPC + vHSS Hostname: XXX Hostname: XXX Hostname: XXX vPGW SGi SGi MLXe S5 (mirrored) SGi (mirrored) SME vIPE-Gi vACE vIPE-GTP Dashboard (Map) Mirrored Port S5 SGi Vyatta vRouter
- 93. VCM VM规划 Switch1 VDX Switch2 VDX Switch3 DELL Switch4 DELL MLXe MGMT VCM vEPC & vHSS 1 Server No.6 vEPC & vHSS 2 Server No.7 vPGW 1 Server No.8 vPGW 2 Server No.9 Server No.10 Not used now To BSC Eth 4/13 Eth 3/12 10.61.12.25/24 Eth 32 Eth 4/15 Eth 4/16 S5 mirror, eth33 femto Eth35 ens6f1 ens6f1 ens6f1 ens6f1
- 94. NVA VM规划
- 95. NVA Packet Broker • L2/L3/L4 filtering • Aggregation • Replication • Basic load-balancing (LAG) • Jumbo frame support • VLAN tagging • VLAN tag stripping • Header Stripping • Advanced load-balancing (GTP- TEID) • Packet slicing • Flex match • Time-stamping • GTP-IP inner IP filtering • TCP packet length filtering • Source port labeling • Packet fragmentation • De-duplication • Packet masking • Extended flex match • Extended packet slicing • PTP time stamping Basic Advanced Expert
- 96. UE接入测试
- 98. SAND Dashboard
- 99. 端到端服务质量监测
- 100. KPI分析和趋势分析
- 102. 储子瑞 Brocade资深软件网络工程师 梁博 九州云资深架构师及培训负责人 Sept 20th, 2016 敏捷之魂 LinkedIn和Netflix钟爱的 事件驱动云数据中心工作流级自动化 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 103. DIGITAL TRANSFORMATION © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 104. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 112
- 105. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 113 Digital transformation and agility are top-of- mind with CIOs IT infrastructure is only as agile as its least agile component
- 106. It’s Time to Talk about the Network 114VANSON BOURNE, BROCADE GLOBAL CIO SURVEY 2015, MAY 2015 75% of CIOs say the network is an issue in achieving their organization’s goals © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 107. Automation Is Our Hallmark © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 115 1995 Fibre Channel Fabric 70%+ Market Share 2009 VCS Fabric 4,100+ Installations February 2016 IP Fabric Market Expansion
- 108. Brocade Workflow Composer © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 116 Brocade Workflow Composer is a network automation platform that automates the entire network life cycle and integrates with cross-domain workflows to improve business agility
- 109. Open at all layers Cross-domain Workflow- centric DevOps inspired Event-driven, closed-loop automation Extensible Brocade Workflow Composer Design Principles © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 117 Flexible
- 110. Brocade Workflow Composer Automation your way © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 118 Turnkey Customizable Do-It-Yourself
- 111. Brocade Workflow Composer Workflow-centric, cross-domain, event-driven automation © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 120 Rules Inbound integration plugins that watch for events from external systems Outbound integration plugins that request changes to external systems ActionsSensors Brocade Workflow Composer Network Domain Compute Domain Storage Domain Ops Support Services Cloud Infrastructure IT Domains Multivendor Workflow Powered by StackStorm
- 112. Rules Engine Credential / Authorization Device Inventory Workflow Engine Data Store REST API Brocade Workflow Composer Architecture Powered by StackStorm 121 Audit Workflows Composable Services Sensors / Actions User Interfaces Sensor Design UI Admin UI CLI Chat Ops Message Bus Sensor Action Action Systems Devices Tools © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 113. Benefits © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 114. DATA CENTER Software-Driven Lifecycle Workflows Infrastructure examples: Enabling data center fabric © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 123 Remediate Provision Validate Troubleshoot IP connectivity BGP EVPN IP Fabric SLA (packet loss) OS rolling upgrades BGP Flaps Connectivity loss VCS Fabric DC tenant
- 115. Automating in Silos Is Not Enough Cross-functional handoffs create delays © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 124 • Oracle - Understanding Workflows: (https://docs.oracle.com/cd/E17984_01/doc.898/e14729/understand_workflow.htm) Total Time Delay “90% of the total time used to complete business tasks is classified as transfer delay time and 10% on actual task execution.” Workflow Management Coalition* End-to-EndWorkflow Delay Delay Server Team Application Team Network Team Security Team
- 116. Automating Across Domains Is the Solution Cross-domain automation eliminates delays, enables business agility 125 Total Time Delay “90% of the total time used to complete business tasks is classified as transfer delay time and 10% on actual task execution.” Workflow Management Coalition End-to-EndWorkflow Delay Delay Server Team Application Team Network Team Security Team Brocade Workflow Composer cross- domain workflows eliminate transfer delay and reduce workflow execution from weeks or days to just seconds
- 117. Use Cases © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 118. Infrastructure Service Provisioning Data center fabrics © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 127 1. ZTP process registers the switch to the inventory service through the registration sensor. 2. Registration triggers the “IP Fabric Leaf” workflow. 3. Workflow engine models the IP fabric. 5. Configuration is executed through the appropriate action. 6. IP Fabric provision is complete. 4. Brocade Workflow Composer walks through the steps in the workflow. 1 5 2 6 IP fabric 7. Validation workflow begins. Rules Engine Credential / Authorization Device Inventory Workflow Engine Data Store REST API Audit Registration Sensor Visual Flow Web GUI CLI Chat Ops Message Bus Configuration Action VDX (NOS) VDX (NOS) VDX (NOS) VDX (NOS) DAD (ZTP) 3 4 Leaf/Spine validation7
- 119. Example: Validation Data center fabrics © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 128 1. Configure new BGP Peer on Leaf Switch 4 • Validate BGP is running on switch • Alert on failure 2. Configure new BGP Peer on Spine Switch • Validate BGP is running on switch • Alert on failure 3. Validate that BGP Peer state is established on both switches • Alert on failure New Peer Config 1. Configure new BGP Advertisement on Leaf Switch 2 • Validate BGP is advertising route • Alert on failure 2. Validate that Leaf Switch 4 receives new route: • Validate that route exists • Validate the route is learned via BGP • Validate correct number of next-hops for ECMP New Subnet Config Spine Leaf eBGP Private ASN 10.10.10.0/24
- 120. Elapsed Time Event Troubleshooting & Remediation Example: Turning 2am calls into 10am follow-ups © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 129 Spine Leaf eBGP Private ASN SYSLOG Brocade Workflow Composer Slack PagerDuty Help Desk Switch Failure (BGP) Switch link goes down Switch sends SYSLOG message BWC sensor triggers BGP to troubleshoot workflow BWC SYSLOG sensor matches error message (e.g., REGEX) BWC action extracts information: • Switch IP, switch peer IP address, egress interface BWC action logs into switch, performs the following actions: • Execute workflow to determine service and interface state • If interface state is down, attempt to restart interface BWC action creates help desk ticket with outputs from above BWC initiates an alert workflow if interface cannot be restarted • Include help desk ID and URL • Post message to Slack for network ops team visibility • Launch incident to PagerDuty to notify operator Seconds Seconds Seconds Seconds Seconds <1 Minute <1 Minute Total elapsed time: <5 minutes
- 121. Cassandra Ring Status: Operational Health: HealthyHealth: Degraded Monitorin g Server Brocade Workflow Composer Cassandra Node Failure Example: Remediation Cassandra Database © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 130 1. Cassandra node in ring dies 2. Monitoring services detects node failure 3. Monitoring server triggers Cassandra “Replace Host” workflow 4. Remediation workflow: a. Detect seed host state b. Spin up a New VM c. Install Cassandra d. Join to existing Cassandra ring e. Ensure Cassandra node is online For more information on this use case, read our blog post: https://stackstorm.com/2015/09/22/auto-remediating-bad-hosts-in-cassandra-cluster-with-stackstorm/ New VM
- 122. What Customers Are Saying © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 131 “Event Driven Automation is how I sleep soundly during my on-call. It allows my team to turn 2am problems into 10am problems. Issues that would wake up an engineer are now a ticket to follow-up on the next morning.” Ben Goldsbury, Site Reliability Engineer, Well-Known Social Media Site for Professionals “Event-driven provisioning and network automation within the cloud stack is a crucial piece of our service offering, to meet the complex demands of enterprise cloud systems.” Travis Mims, Director of Engineering at Cloud Automation Solutions
- 123. Enable end-to-end IT workflow automation Brocade Workflow Composer Summary © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 132 Software-Driven Lifecycle Management Cross-Domain Integration Flexible Workflow Automation and No Vendor Lock-In Devops Inspired Automate the entire network lifecycle with event-driven automation Unleash IT agility with open turnkey, customizable, or DIY workflow automation Leverage the power of DevOps methodologies, open source technologies, and a thriving technical community An open network automation platform that enables cross-domain workflows to improve business agility.
- 124. How Do You Get Brocade Workflow Composer? © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 125. StackStorm Open source project © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 134 • Open Source Project: – Brocade Workflow Composer/ST2 Platform – Web UI – Secure Data Store – Brocade Workflow Composer Exchange* to manage suites and packs from Community • Community Support • StackStorm open source project is available via GitHub for community use, and accepts pull requests for community contributed features. * Availability TBD StackStorm StackStorm Open Source Project ExchangeWeb UI Secure Data Store
- 126. Brocade Workflow Composer © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 135 • Brocade Workflow Composer: – StackStorm platform – Web UI – Secure Data Store – Designer UI: Graphical Interface to design workflows – Role-Based Access Control and LDAP Integration – Brocade Workflow Composer Exchange* to manage packs from Community and Subscriptions • Brocade Technical Support • Brocade Workflow Composer support includes break/fix support and updates Brocade Workflow Composer StackStorm Open Source Project Flow UI RBAC LDAP Pack Exchange Web UI Secure Data Store BrocadeSupport * Availability TBD
- 127. Automation Suites © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 136 Automation suite(s): A collection of workflows, services, and documentation to enable specific use case(s). Includes workflow definitions, required actions, sensors, and integrations Types of suites 1. Brocade-provided automation suites: Published and supported by Brocade and available through a subscription 2. Community-provided automation suites: Developed, contributed, and supported by the Stackstorm community Workflow SensorsActions Automation Suite
- 128. MANO 驱动的 NFV/SDN 服务编排 服务商 云网合一 开放生态实战方案分享
- 129. MANO 架构 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. COMPANY ALL RIGHTS RESERVED
- 131. NFV Orchestrator (NFVO) 140 Network Service Orchestration using a collection of VNFs and Forwarding Graphs Templatize an end-to-end Network Service using decomposed VNFs Resource Checks and Resource Allocation VNFs connected using Forwarding Graphs o Described in a VNF Forwarding Graph Descriptor o Render VNF Forwarding Graphs using SDN Controller or a SFC API Ability to orchestrate VNFs across Multiple VIMs PNF connectivity and configuration in end to end network service orchestration
- 132. VNF Manager (VNFM) © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. COMPANY ALL RIGHTS RESERVED 141 VNF Catalog – repository of VNF descriptors (VNFDs) in a database VNF Instantiation and Termination VNF Monitoring - Health and Performance Indicators Self Healing and Auto Scaling VNF Configuration injection during instantiation VNF Image management Support both simple and complex VNFs Enforce placement policy ensure efficient placement of VNFs (NUMA, CPU Pinning)
- 133. Virtual Infra Manager (VIM) 142 Responsible for controlling and managing the NFVI compute, storage and network resources Usually within one operator’s Infra Domain (i.e. NFVI-POP) Exposes Northbound APIs (i.e. used by VNFM) OpenStack is an example of a VIM
- 134. TACKER
- 135. API (WSGI, extension /plugin framework) NFV Catalog VNFDNSD FFGD VIM Site 1 VIM Site 3 VNFM Management Driver Framework Monitoring Driver Framework TOSCA Workflow VNF Instances TOSCA Template Validation NFVO Multi-Site VNF Fwd Graph TOSCA Workflow Network Service Instances VIM Instances Horizon CLI Alarm Monitor Event Audit Log Infra Driver (Heat, Keystone) Tacker Architecture Modified on slides from Sridhar Ramaswamy, Tacker PTL
- 136. OpenStack Heat Translator TOSCA parser NFV Descriptors NSD VNFD VNFFGD ETSI NFV OpenStack Tacker (NFVO / VNFM/ VIM) TOSCA Data model 145 heat-translator Modified on slides from Sridhar Ramaswamy, Tacker PTL
- 137. Audit Log / Event Framework Tacker DB VIM VNF FFG NSD External Event Stream (websocket/zm q) Lifecycle Events and Audit Log API access / client API Modified on slides from Sridhar Ramaswamy, Tacker PTL
- 138. VNF Tacker Manual Scale Request Alarm Monitor (Ceilometer, Monasca, Custom) VNF VNF scale VNF Scaling + Alarm Monitoring monito r trigger Modified on slides from Sridhar Ramaswamy, Tacker PTL
- 139. VNFFG API Tacker Compute Node 1 OVS SDN Controller OVSDB Compute Node 2 OVS ... VNF vRouter VNF DPI Templates Neutron (networking-sfc) DB Workflow: 1) Onboard NSD, FGD, VNFD to Catalog 2) Instantiate 2 or more VNFs from Catalog 3) Invoke Tacker VNFFGD API to render the graph VNF FFG D Heat VNFFG Copy from Sridhar Ramaswamy, Tacker PTL
- 140. G-VNFM + NFVO Tacker VNFM Tacker VNFM Tacker NFVO NFVO (ManageIQ, Cloudify/Aria , Proprietary) Modified on slides from Sridhar Ramaswamy, Tacker PTL
- 141. TOSCA Topology and Orchestration Specification for Cloud Applications
- 142. What is TOSCA NFV? • Topology and Orchestration Specification for Cloud Applications (TOSCA) comes from the OASIS open standards consortium and has been used in the applications ecosystem since 2014 • TOSCA NFV is a data modeling and templating language for NFV services allowing you to define: Topology, Composition, Lifecycle, Portability • Is very different to Yang, Yang is good for network device configuration not good for above complex relationships • Some Open Source Implementations: Senlin, Tacker, Parser, Alien4cloud, Cloudify, Aria TOSCA NFV Specification: http://docs.oasis-open.org/tosca/tosca-nfv/v1.0/csd02/tosca-nfv-v1.0-csd02.pdf Source: TOSCA NFV Specification
- 143. TOSCA NFV templates describes the service in vendor abstract terms A simple p2p circuit just uses the WAN infrastructure Input parameters specify the endpoints of the service 152
- 144. TOSCA NFV is useful as it can describe both VNF and Carrier SDN-WAN 153
- 145. Bringing SDN/NFV together Open Modular architecture allows the platform to evolve over time (no monolithic proprietary lock-ins) Service API provides a simple programmatic interface for all network services SDN WAN for proven scalability and reliability Open Source projects with broad, active communities for components that enable interoperability 155 WAN PCE Application leverages open source ODL SDN Controller platform
- 146. 服务商部署实例 156
- 147. AT&T SDN创新 按需网络 愿景落地 按需广域网服务 Brocade vRouter 局域/领域控制器： • NSX • Nuage • Contrail • …. Domain 2.0 The First Major Case 服务编排 云编排 策略 分析 Brocade SDN 控制器
- 148. Latest Commercial Case – MANO Driven Large Service Provider looking to provider core offload via SDN 158 Develop a software-defined network and NFV platform that: Transports and processes non-critical traffic, such as tap/sensor streams, for monitoring and analytics Supports low cost network business case to migrate traffic off the premium customer MPLS core network Provides a national network, that is programmable and able to insert VNF instances into network paths for traffic processing That is developed in an agile manner, to provide the SP the ability to adapt throughout the development process Delivers the project to educate SP’s engineers and set-up software practices, process and tools which can be re-used Provides SP options for operate and transfer following build Project Objectives
- 149. Modular Clear, well defined and open interfaces Simplified integration points with existing systems Proven, fully supported products Bespoke integration Open Source projects with broad, active communities for components that enable interoperability Customer Use Case 159
- 150. Brocade & Partner Products & NetDev Integrations Brocade Products: 1. Brocade SDN Controller 2. Brocade Flow Manager 3. Brocade VNF Manager (Tacker) Brocade NetDev Integrations: a. SP TOSCA NFV Service Templates (NSD) b. SP TOSCA VNF Definition (VNFD) c. Service Orchestrator Plugin – BFM d. Service Orchestrator Plugin – BVM (Tacker) e. SDN Traffic Engineering: BFM & Brite-box f. Service Assurance integration with existing NMS and remediation into the orchestrator 160 1 2 3 1 2 3 a b c d e c d a e f b f Partner Products: 1. Red Hat RHEL/OSP 2. Gigaspaces Cloudify 3. Brite-box switches Supermicro Servers 4 5 6 7 4 7 6 5
- 151. Open interfaces allow components to be interchangeable Flow Manager interface used to bring provision virtual links (paths) across the SD-WAN Tacker interface provisions VNFs, passes VNF parameters and optionally config Heat interface used by tacker to bring up VMs with Openstack TOSCA NFV based “Service API” used to deploy end to end network services Openstack used to provision OS provider networks for the VNF 161
- 152. Unique capability of solution is ability to scale out • Service Orchestrator integrates the end to end network service (multiple Openstack regions supported out of the box, can also support Hybrid clouds) • SDN-WAN efficiently scales out as the network grows (supports TE in the WAN) • Not an overlay solution • Overlay technologies (like VXLan) across the WAN at scale is not proven – designed for the DC • Overlay technologies between multiple VIMs still needs something to co-ordinate between them • SFC Technologies like NSH still evolving, our solution will make use of these as they become more widely accepted and standardised (i.e. SFC support in Tacker) 162
- 153. SP can take control from day-one in creating new services with templates • Created once per service type (i.e. p2p, p2p with FW+LB, l2vpn, etc) • Is simple enough to be done by Telstra, although Brocade can provide support if needed • Defined using the TOSCA NFV model and written in either YAML or JSON • YAML is a very easy modelling language with many tools and editors, Cloudify does have a blueprint editor available • Once created the service template (or Blueprint) is loaded into the service orchestrator where it is validated before being added to the service catalogue 163
- 154. Demonstration topology 164 1a. Service Orchestration 1b. VNF Orchestration 1c. Instances 1d. Networks of:11 of:21 of:12 of:22 of:31 31513101 3102 3103 3153 OVS SDN Controller BFM City#1 POD City#2 POD Service Orchestrator OS Controller Tacker OF:100 OF:200of:32 3152 vRouter vRouter Openflow Horizon 1 2 3 4 n Screens used in demonstrations
- 155. Sw itch Sw itch Sw itch Edge Transit Edge Match Match Demo: Bi-directional Point to Point Path Path traverses one or more switches End-user defines match criteria and two edge switches Path computed based on standard least-hops algorithm and programmed in switches in real-time Back-up provided as standard, path recomputed and reprogrammed following any service impacting topology changes Demo: Bi-directional Point to Point Path, with Way Point(s) Path traverses two or more switches End-user defines match criteria, two edge switches and one or more transit switches as way points Path computed based on least-hops algorithm between way points and programmed in real-time Back-up provided as standard, path recomputed and reprogrammed following any service impacting topology changes Sw itch Sw itch Sw itch Edge Way Point(s) - Transit Edge Match Match Demos: End-to-end Paths 165
- 156. Sw itch Sw itch Sw itch Edge Transit Edge Match Match C om pute VNFDemo: Bi-directional Point to Point Path with VNF Path traverses one or more switches End-user defines match criteria, two edge switches and VNF details Path computation and programming as before VNF inputs specified by end-user: location, VM, VNF image and config VNF defined at edge switch location Two VNF defined on different compute pods Sw itch Sw itch Sw itch Edge Transit Edge Match Match C om pute VNF C om pute VNF Demos: Paths + VNF Services 166
- 157. Demo Time 167
- 158. Another Possibility: 混合组网方案 - OpenFlow MPLS LSP互连 • 通过Openflow将不同客户/业务流映射进不同的LSP中 • BSC (Brocade ODL 控制器) 通过Netconf Yang Model配置LSP • 利用Openflow 1.3的meter、mirror等功能实现计费、镜像等功能 • 利用MPLS骨干网实现面向关键业务的最高服务可用性, 传统与SDN兼容演进支持, 规模部署, 及 TE流量工程 (e.g. FRR路径保护等) • 通过基于BSC的BFO(Flow Optimizer)应用实现DDoS的防护和流量优化 ShippingShippingConcept 168
- 159. 张宇峰， 任志远，余欣，智勇，梁博 龚永生，方辉 SDNLAB联合创始人 PANEL 论坛 DevOps 颠覆文化，开放边界 变产业，演进生态
- 160. Back to The Future 软件 机器 赛跑 学习 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.
- 161. © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 2011
- 162. 1995 “Steve Jobs: The Lost Interview” On the importance of software to the computer industry from 1995 on: "There hasn't really been a revolution in how we created software, at least not in the last 20 years. [But] software is infiltrating everything we do these days. Software is going to be a major enabler in our society."
- 163. The Race against Machines… 和机器的赛跑… 开始了吗？ 1968 1999 1984 2015 The Rise of The Machines
- 165. 1995 Mr. Jobs on importance of computer programming to cognitive ability: "It didn't have to do with using them for anything practical; we used it as a way to mirror our thought process. I think everyone should learn how to program a computer, because it teaches you how to think. I view computer science as a liberal art, something everyone should learn to do."
- 166. Up the Stack, Deeper into Data, Playing with Machines…学习 • 网络作为 Data Fabric & Sensors • 网络可视分析与大数据平台 • 网络大数据，SDN，机器学习 • 基于云平台的IT运维全局分析 & benchmark 云网交界处的数据湖
- 167. 聚焦产业实战的生态平台 微信+线下 @中国&世界学习 SDN实战团
- 168. 未来网络学院 SDN实战集训营 第1期 北京 2016.4.20~24 30名学员 第2期 北京 2016.7.18~22 33名学员 第3,4期 上海，广州，2016 Q4 第5期 成都 2017 Q1 学习
- 169. Thank You 感谢聆听 © 2016 BROCADE COMMUNICATIONS SYSTEMS, INC.