FCoE Update – The Good News and The Bad News

FCoE (Fibre Channel over Ethernet) is a relatively new industry effort designed to combine the lossless features of FC with the ubiquity of Ethernet. FCoE is essentially Fibre Channel (FC) frames encapsulated in Ethernet packets using Ethernet links instead of Fibre Channel links. Nonetheless, at the upper layers it is still Fibre Channel which allows for the preservation of existing FC infrastructures – a major design goal. FCoE allows storage and network traffic to be converged onto one set of cables, switches and adapters thereby reducing cables, energy consumption and heat generation. Storage management using an FCoE interface has the same look and feel as storage management with traditional FC interfaces. Nonetheless, FCoE is Layer 2 only and this fact greatly impacts its capabilities. This industry standards effort depends on the coordinated work of three standards bodies:

• IEEE for Ethernet extensions
• INCITS/ANSI T11committee for the Fibre Channel protocols
• IETF for routing

The Good News

Industry Standards are Moving Along

Fibre Channel Backbone 5 (INCITS/ANSI T11 FC-BB-5) was approved by the T11 committee in June, 2009. This is the big one. It defines a network, and its associated resources and services that are used to connect one or more Fibre Channel entities over non-Fibre Channel protocol infrastructures. While the standard covers many use cases, FCoE, the focus here, is part of FC-BB-5 and is technically known as FC-BB_E. The FC-BB_E model defines the means by which Fibre Channel frames are transported over a Lossless Ethernet network.

Lossless Ethernet — This form of Ethernet includes new enhancements that make it a viable transport for storage traffic and storage fabrics without requiring TCP/IP overheads. Converged Enhanced Ethernet is currently a 10Gbps full duplex lossless Ethernet with dynamic prioritization/bandwidth allocation and some other goodies. This form is known variously as

• CEE: Converged Enhanced Ethernet – a generic term used by many vendors including HP, IBM, Dell, Brocade, and others to describe enhanced Ethernet.
• DCE: Data Center Ethernet – was a term originally coined and trademarked by Cisco. DCE referred to Enhanced Ethernet based on the Data Center Bridging standards. DCE is CEE plus additional functionality including Congestion Notification
• EEDC: Enhanced Ethernet for Data Center – A term coined by Intel that also includes deterministic latency for High-Performance Computing and Communications traffic as well as seamless integration with classical Ethernet networks.
• DCB: Data Center Bridging – Work underway in an IEEE task group described as an architectural collection of Ethernet extensions designed to improved Ethernet networking and management in the data center.

All four of these terms are pretty much interchangeable in the context of this document. The Lossless Ethernet enhancements are defined by the following IEEE specifications:

• 802.1Qbb: Priority Flow Control (PFC)
  • Ability to control a flow (pause) based on a priority
  • Allows lossless FCoE traffic without affecting classical Ethernet traffic
  • Establishes priority groups using 802.1Q tags
• 802.1Qaz: Enhanced Transmission Selection (ETS)
  • Allows bandwidth allocation based on Priority Groups
  • Allows Strict Priority for low bandwidth / low latency traffic
  • DCBX (DCB Exchange Notification) is part of 802.1Qaz and leverages LLDP (Link Layer Discovery Protocol, 802.1AB)
• 802.1Qau: Congestion Notification (CN)
  • Allows for throttling of traffic at the edge of the network when congestion occurs within the network

These specifications are not final or published yet, but are presently considered “technically stable”. This basically means they have been reviewed by a whole lot of different people, have had sufficient working demonstrations, no technical changes are expected and it is safe enough for vendors to implement it. Possible changes, if any, will be minor and with little effect on implementations. Actual approval is expected sometime in 2010 and depends on who you ask.

Proven Infrastructure Becoming Available

Test Drive Results Favorable – 14 companies participated in at test drive in June, 2009 with quite favorable results.

Demo at SNW Fall 2009 – QLogic, Emulex, Intel, Brocade, PMC-Sierra, Cisco, NetApp, EMC and LSI Enginio successfully demonstrated FCoE and FC switches, Converged Network Adapters, FC HBAs and storage targets.

Another Plug Fest is scheduled for this November, 2009.

Software Stacks – An Open FCoE stack has been accepted Into the Linux Kernel v2.6.29.  OpenSolaris has both target and initiator stacks. Chelsio has announced its software stacks and Microsoft is developing its own.

Storage Targets – Many disk subsystem vendors now offer FCoE support and most will within 12 months

Servers and blades – Dell and IBM among others are offering servers with FCoE support and IBM just added support on its blade offering.

Protocol Analyzers are Becoming Available – from the likes of JDSU and Wireshark

Multi-hop available from Cisco –  The Nexus 5000 can be a DCB-capable switch and Cisco has supported FC-BB-5 since September 2009 including FIP (FCoE Initialization Protocol), but Cisco does not recommend it – likely for cost reasons

The Bad News

No Multi-hop with Brocade– This really technical discussion explores why  multi-hop FCoE switches were not initially available . The comments are especially enlightening. It boils down to while FC Backbone 5 has been approved, one part of it, the FIP, is not fully supported by Brocade. Indeed, very recent Brocade documents specifically refer to a “Pre-FIP version of the protocol”.

No Multipath – Since Ethernet is a Layer 2 protocol, it is not routable, but the IETF is working on it with a specification called TRILL (Transparent Interconnection of Lots of Links). TRILL will provide a solution for shortest-path frame routing in multi-hop CEE networks with arbitrary topologies. TRILL closely resembles Cisco’s L2MP (Layer 2 Multipathing) protocol. Solutions will be able to have multipathing only after TRILL is fully approved and implemented in products. The result is less than optimal resource utilization and no load balancing.

Shared Media Issues – While native FC protocols and FC switches are designed to deal only with point-to-point connectivity, FCoE pass-through switches introduce shared-media (point-multipoint) semantics. In other words, a FCoE pass through switch is invisible to the Fibre Channel stack, but it acts as a concentrator of flows from multiple servers into the same port of a dual-stack switch, and thus it creates a shared-medium hanging off the dual-stack switch port. A typical Fibre Channel stack is not prepared to address shared-media links, since they do not exist in native Fibre Channel; these new semantics must therefore be handled by FCoE without affecting the Fibre Channel stack.

FC-BB-6 – This project is starting up within the T11 committee and while the charter and timeline are not finalized, it will be discussing how to enrich FCoE with larger FCoE configurations such as allowing for creation of a routable single CEE cloud, point-to-point CEE configurations with no switches, investigate improvements in support for high BER Ethernet transmission media, e.g., 10GBASE-T and any other item as deemed necessary during the development. In other words, one Layer 2 for everything in the data center.

New infrastructure required from limited suppliers – Not really a new issue as the market for native FC switches consolidated down to Brocade, Cisco and QLogic. And, despite demonstrated interoperability at a base level, users consciously get locked into one switch vendor as usual, choosing one for value-added or price, for example. The same applies for adapter cards.

It’s not really 10-Gbit – While storage traffic starts out on 10-gigabit CEE, it gets dropped onto an 8Gb native FC SAN (or 4Gb). This not really a big deal as it roughly balances the encapsulation overhead of FCoE.

FC and FCoE are diverging – At the recent Storage Networking World conference held this October, 2009, the Fibre Channel Industry Association (FCIA) showed a roadmap for FC that goes 4->8->16->32 gigabit. Whereas the roadmap for FCoE is 10->40 gigabit. Fortunately, all but 40Gb-FCoE use the same typical data center optical and copper assemblies, i.e., OM2, OM3, OM4 and TwinAx with the same SFP+ module connection.

Summary

The number one reason to adopt FCoE today is to reduce heat, energy, cable plant and space. Data center networks are converging, but are not converged. Nonetheless, FCoE today can achieve more than 80% of the involved economies. Protocols like TRILL and FC BB 6 become more relevant when the entire data center infrastructure is converged over Ethernet, and this will bring the other 20% – someday.