Internet Engineering Task Force                               I. Hussain
Internet-Draft                                               R. Valiveti
Intended status: Informational                               K. Pithewan
Expires: January 8, 2017                                   Infinera Corp
                                                            July 7, 2016


                             FlexE Usecases
                 draft-hussain-ccamp-flexe-usecases-01

Abstract

   Traditionally, Ethernet MAC rates were constrained to match the rates
   of the Ethernet PHY(s).  OIF's implementation agreement [OIFMLG3] was
   the first step in allowing MAC rates to be different than the PHY
   rates.  OIF has recently approved another implementation agreement
   [OIFFLEXE1] which allows complete decoupling of the MAC data rates
   and the Ethernet PHY(s) that support them.  This includes support for
   (a) MAC rates which are greater than the rate of a single PHY
   (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are
   less than the rate of a PHY (sub-rate), (c) support of multiple FlexE
   client signals carried over a single PHY, or over a collection of
   bonded PHY(s).  This draft catalogs the usecases that are encountered
   when these Flexible rate Ethernet client signals are transported over
   OTN networks.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 8, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.




Hussain, et al.          Expires January 8, 2017                [Page 1]

Internet-Draft               FlexE Usecases                    July 2016


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  FlexE Transport Usecases  . . . . . . . . . . . . . . . . . .   4
     3.1.  FlexE unware transport  . . . . . . . . . . . . . . . . .   4
     3.2.  FlexE Aware . . . . . . . . . . . . . . . . . . . . . . .   6
       3.2.1.  FlexE Aware Case - No Resizing  . . . . . . . . . . .   6
     3.3.  FlexE Termination - Transport . . . . . . . . . . . . . .   9
       3.3.1.  FlexE Client at Both endpoints  . . . . . . . . . . .   9
       3.3.2.  Interworking of FlexE Client w/ Native Client at the
               other endpoint  . . . . . . . . . . . . . . . . . . .  10
       3.3.3.  Interworking of FlexE client w/ Client from OIF_MLG .  11
       3.3.4.  FlexE Client BW Resizing  . . . . . . . . . . . . . .  12
       3.3.5.  Back-to-Back FlexE  . . . . . . . . . . . . . . . . .  13
   4.  FlexE Transport over Wavelength(s) Usecases . . . . . . . . .  14
   5.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .  16
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  17
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  17
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  17
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  17
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  17
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  17
   Appendix A.  Additional Stuff . . . . . . . . . . . . . . . . . .  18
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  18

1.  Introduction

   Traditionally, Ethernet MAC rates were constrained to match the rates
   of the Ethernet PHY(s).  OIF's implementation agreement [OIFMLG3] was
   the first step in allowing MAC rates to be different than the PHY
   rates standardized by IEEE.  OIF has recently approved another
   implementation agreement [OIFFLEXE1] which allows complete decoupling
   of the MAC data rates and the Ethernet PHY(s) that support them.
   This includes support for (a) MAC rates which are greater than the
   rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b)
   MAC rates which are less than the rate of a PHY (sub-rate), (c)



Hussain, et al.          Expires January 8, 2017                [Page 2]

Internet-Draft               FlexE Usecases                    July 2016


   support of multiple FlexE client signals carried over a single PHY,
   or over a collection of bonded PHY(s).  The capabilities supported by
   the OIF FlexE implementation agreement version 1.0 are:

   a.  Support a large rate Ethernet MAC over bonded Ethernet PHYs, e.g.
       supporting a 200G MAC over 2 bonded 100GBASE-R PHY(s)

   b.  Support a sub-rate Ethernet MAC over a single Ethernet PHY, e.g.
       supportnig a 50G MAC over a 100GBASE-R PHY

   c.  Support a collection of flexible Ethernet clients over a single
       Ethernet PHY, e.g. supporting two MACs with the rates 25G, 50G
       over a single 100GBASE-R PHY

   d.  Support a sub-rate Ethernet MAC over bonded PHYs, e.g. supporting
       a 150G Ethernet client over 2 bonded 100GBASE-R PHY(s)

   e.  Support a collection of Ethernet MAC clients over bonded Ethernet
       PHYs, e.g. supporting a 50G, and 150G MAC over 2 bonded Ethernet
       PHY(s)

   Optical Transport Networks (defined by [G709] and [G798]) have, until
   recently, only dealt with bit (or codeword) transparent transport of
   Ethernet client signals.  The introduction of the FlexE capabilities
   at the OTN client interfaces requires the OTNs to examine the various
   usecases.  This Internet-Draft examines the various usecases that
   arise when transporting the Flexible Rate Ethernet signals in Optical
   transport networks.  This list of usecases will help identify the
   Control Plane (i.e.  Routing and Signaling) extensions that may be
   required).

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  Terminology

   a.  Ethernet PHY: an entity representing 100G-R Physical Coding
       Sublayer (PCS), Physical Media Attachment (PMA), and Physical
       Media Dependent (PMD) layers.

   b.  FlexE Group: a group of from 1 to 254 bonded Ethernet PHYs.

   c.  FlexE Client: an Ethernet flow based on a MAC data rate that may
       or may not correspond to any Ethernet PHY rate (e.g., 10, 40, m x
       25 Gb/s).



Hussain, et al.          Expires January 8, 2017                [Page 3]

Internet-Draft               FlexE Usecases                    July 2016


   d.  FlexE Shim: the layer that maps or demaps the FlexE clients
       carried over a FlexE group.

   e.  FlexE Calendar: Representation of a FlexE group of n PHYs as a
       calendar of 20n slots logical length with 20 slots per PHY for
       scheduling of slots (i.e., a PHY bandwidth) among the FlexE
       clients.

3.  FlexE Transport Usecases

3.1.  FlexE unware transport

   The FlexE shim layer in a router maps the FlexE client(s) over the
   FlexE group.  The transport network is unware of the FlexE.  Each of
   the FlexE group PHY is carried independently across the transport
   network over the same fiber route.  The FlexE shim in the router
   tolerates end-to-end skew across the network.  This usecase allows to
   utilize Network Processor Unit (NPU) and router port rate full
   capacities with legacy transport equipment that provides PCS-codeword
   transparent transport of 100GbE.  It allows striping of PHYs in the
   FlexE group over multiple transport line cards.






























Hussain, et al.          Expires January 8, 2017                [Page 4]

Internet-Draft               FlexE Usecases                    July 2016


    ==================================================================

       +                            FlexE Ethernet Client(s)       +
       +-----------------------------------------------------------+
       +                                                           +
                        + FlexE skew tolerance
                        +----------------------------------------+
                        +  for end-to-distance                   +

   +-----------+ 2x100GE +---------+   +----------+     +------------+
   |           |         |         |   |          |     |            |
   | Router1   |         |         |   |          |     |            |
   |FlexE Shim +---------+ A-end   |   |  Z-end   +-----+Router 2    |
   |           |         | (FlexE  |   |  (FlexE  |     |(FlexE Shim)|
   |           +---^-----+ unaware)|   |  unaware)+-----+            |
   |           |   |     |         |   |          |     |            |
   |           |   |     |         |   |          |     |            |
   +-----------+   +     +---------+   +----------+     +------------+
                    FlexE Group

                        \----------Transport----------/
                                   network
   +--------------+                                  +----------------+
   | FlexE Clients|                                  | FlexE Client(s)|
   +--------------+                                  +----------------+
   | FlexE Shim   |                                  |  FlexE Shim    |
   +----+----+----+                                  +----+------+----+
   |PHY |  |  PHY |                                  |  PHY |   | PHY |
   +---+---+--+---+                                  +---+--+   +--+--+
       |      |          +-----+           +-----+       |         |
       |      +----------+ PHY |           | PHY |-------+         |
       |                 +-----+           +-----+                 |
       |                 | ODU4+-----------+ ODU4|                 |
       |                 +-----+           +-----+                 |
       |                                                           |
       |                 +-----+           +-----+                 |
       +-----------------+ PHY |           | PHY +-----------------+
                         +-----+           +-----+
                         | ODU4+-----------+ ODU4|
                         +-----+           +-----+


    ==================================================================

                     Figure 1: FlexE unaware transport






Hussain, et al.          Expires January 8, 2017                [Page 5]

Internet-Draft               FlexE Usecases                    July 2016


3.2.  FlexE Aware

3.2.1.  FlexE Aware Case - No Resizing

   This scenario represents an optimization of the FlexE unaware
   transport presented in Section 3.1, and illustrated in Figure 1.  In
   this application (see Figure 2), the devices at the edge of the
   transport network do not terminate the FlexE shim layer, but are
   aware of the format of the FlexE overhead.  They "snoop" the FlexE
   overhead to determine the subset of the set of all calendar slots
   that are available for use (i.e. these calendar slots may be used, or
   unused).  The transport network edge removes the unavailable calendar
   slots at the ingress to the network, and adds the same unavailable
   calendar slots back when exiting the network.  The result is that the
   FlexE Shim layers at both routers see exactly the same input that
   they saw in the FlexE unware scenario -- with the added benefit that
   the line (or DWDM) side bandwidth has been optimized to be sufficient
   to carry only the available calendar slots in all of the Ethernet
   PHY(s) in the FlexE group.  This mode may be used in cases where the
   bandwidth of the Ethernet PHY is greater than the bit rate supported
   by a wavelength (and it is known that that all calendar slots in the
   PHY are not "available").

   The transport network edge device could learn of the set of
   unavailable calendar slots in a variety of ways; a few examples are
   listed below:

   a.  The set of unavailable calendar slots could be configured against
       each Ethernet PHY in the FlexE group.  The FlexE demux function
       in the transport network edge device (A) compares the information
       about calendar slots which are expected to be unavailable (as per
       user supplied configuration), with the corresponding information
       encoded by the customer edge device in the FlexE overhead (as
       specified in [OIFFLEXE1]).  If there is a mismatch between the
       unavailable calendar slots in any of the PHYs within a FlexE
       group, the transport edge node software could raise an alarm to
       report the inconsistency between the provisioning information at
       the transport network edge, and the customer edge device.

   b.  The Transport network edge could be configured to act in a
       "slave" mode.  In this mode, the FlexE demux function at the
       Transport network edge (A) receives the information about the
       available/unavailable calendar slots by observing the FlexE
       overhead (as specified in [OIFFLEXE1]) and uses this information
       to select (a) the set of wavelengths (with appropriate
       capacities) or (b) the bandwidth of the ODUflex (or fixed rate
       ODUs) that could carry the FlexE PCS end-to-end.




Hussain, et al.          Expires January 8, 2017                [Page 6]

Internet-Draft               FlexE Usecases                    July 2016


   In the basic FlexE aware mode, the transport network edge does not
   expect the number of unavailable calendar slots to change
   dynamically.

   Note that the process of removing unavailable calendar slots from a
   FlexE PHY is called "crunching" (see [OIFFLEXE1]).  The following
   additional notes apply to Figure 2:

   a.  The crunched FlexE PHYs are independently transported through the
       transport network.  The number of used (and unused) calendar
       slots can be different across the FlexE group.  In particular, if
       all the calendar slots in a FlexE PHY are in use, the crunching
       operation leaves the original signal intact.

   b.  In this illustration, the different FlexE PHY(s) are transported
       using ODUflex containers in the transport network.  These ODUflex
       connections can be of different rates.

   c.  When the crunched FlexE PHY(s) have a rate that is identical to
       that of a standard Ethernet PHY, it is possible that the
       transport network may utilize standard ODU containers such as
       ODU2e, ODU4 etc.





























Hussain, et al.          Expires January 8, 2017                [Page 7]

Internet-Draft               FlexE Usecases                    July 2016


   ================================================================

                                FlexE Ethernet Client(s)
              +-----------------------------------------------------+
                          FlexE skew tolerance
                  +---------------------------------------------+
                           for end+to+distance

          +--------+ 2 x 100GE +---------+      +---------+    +------+
          |  R1    |           |         |      |         +----+  R2  |
          |  (FlexE+-----------+  NE A   |      |  NE Z   |    |(FlexE|
          |  Shim) |           | (FlexE  |      | (FlexE  +----+ Shim |
          |        +-----^-----+ aware)  |      | aware)  |    |      |
          |        |     |     |         |      |         |    |      |
          +--------+     +     +---------+      +---------+    +------+
                    FlexE Group
                               \+--------+Transport+--------+/
                                          network
         +-------------+                                +-------------+
         |FlexE clients|                                |FlexE clients|
         +-------------+                                +-------------+
         | FlexE Shim  |                                | FlexE Shim  |
         +------+------+                                +------+------+
         |  PHY |  PHY |                                |  PHY |  PHY |
         +---+--+--+---+                                +---+--+---+--+
             |     |                                        |      |
             |     |       +--------+            +--------+ |      |
             |     +-------+PHY-c   |            |PHY-c   +-+      |
             |             +--------+            +--------+        |
             |             |ODUflex +------------+ODUflex |        |
             |             +--------+            +--------+        |
             |                                                     |
             |             +--------+            +--------+        |
             +-------------+PHY-c   |            |PHY-c   +--------+
                           +--------+            +--------+
                           |ODUflex +------------+ODUflex |
                           +--------+            +--------+

                 | Legend:
                 | R1, R2 - Routers (supporting the FlexE clients)
                 | NE A, Z  - Transport Network Edge nodes
                 | PHY-c  -   Crunched FlexE PHY(s)


   ===============================================================

                      Figure 2: FlexE Aware Transport




Hussain, et al.          Expires January 8, 2017                [Page 8]

Internet-Draft               FlexE Usecases                    July 2016


3.3.  FlexE Termination - Transport

   These usecases build upon the basic router-transport equipment
   connectivity illustrated in Figure 1.  The FlexE shim layer at the
   router maps to the set of FlexE clients over the FlexE group, as
   usual.  This section considers various usecases in which the
   equipment located at the edge of the transport network is fully aware
   of the FlexE OH, and FlexE Shim layers on the transport network edge,
   and the customer edge are peers.  In the router to network direction,
   the transport edge node terminates the FlexE shim layer, and extracts
   one or more FlexE client signals, and transports them through the
   network.  That is, these usecases are distinguished from the FlexE
   unaware cases in that the FlexE group, and the FlexE shim layer end
   at the transport network edge, and only the extracted FlexE client
   signals transit the optical network.  In the network to router
   direction, the transport edge node maps a set of FlexE clients to the
   FlexE group (i.e.  performing the same functions as the router which
   connects to the transport network).The various usecases differ in the
   combination of service endpoints in the transport network.  In the
   FlexE termination scenarios, the distance between the FlexE Shims is
   limited the normal Ethernet link distance.  The FlexE shims in the
   router, and the equipment need to support a small amount skew.

3.3.1.  FlexE Client at Both endpoints

   In this scenario, service consists of transporting a FlexE client
   through the transport network, and possibly combining this FlexE
   client with other FlexE clients into a FlexE group at the endpoints.
   The FlexE client signal can be transported in two manners within the
   OTN: (i) directly over one or more wavelengths (ii) mapped into an
   ODUflex (of the appropriate rate) and then switched across the OTN.
   Figure 3 illustrates the scenario involving the mapping of a FlexE
   client to an ODUflex envelope; this figure only shows the signal
   "stack" at the service endpoints, and doesn't illustrate the
   switching of the ODUflex entity through the OTN.  The ODUflex mapping
   will be beneficial in scenarios where the rate of the FlexE client is
   less than the capacity of a single wavelength deployed on the DWDM
   side of the OTN network, and allows the network operators to packet
   multiple FlexE client signals into the same wavelength -- thereby
   improving the network efficiency.  Although Figure 3 illustrates the
   scenario in which one FlexE client is transported within the OTN, the
   following points should be noted:

   a.  When the FlexE Shim termination function recovers multiple FlexE
       client signals (at node A), the FlexE signals can be transported
       independently.  In other words, it is not a requirement that all
       the FlexE client signals be co-routed.




Hussain, et al.          Expires January 8, 2017                [Page 9]

Internet-Draft               FlexE Usecases                    July 2016


   b.  Conversely, at the egress node, FlexE clients from different
       endpoints can be combined via the FlexE shim, eventually exiting
       the transport edge node over an Ethernet group.

    ==================================================================

    +--------+ 2 x 100GE +---------+       +----------+      +--------+
    |        |           |         |       |          |      |        |
    | Router1|           |         |       |          |      |        |
    | FlexE  +-----------+ A-end   |       |  Z-end   +------+Router2 |
    | Shim   |           | (FlexE  |       |  (FlexE  |      |FlexE   |
    |        +-----^-----+  term)          |  term)   +------+ Shim   |
    |        |     |     |         |       |          |      |        |
    |        |     |     |         |       |          |      |        |
    +--------+     +     +---------+       +----------+      +--------+
              FlexE Group
                        \+--------+Transport+--------+/
                                    network

    +-----------+   +--------------+    +-------------+   +-----------+
    | Client(s) |   | Client       |    | Client      |   | Client(s) |
    +-----------+   +--------+-----+    +------+------+   +-----------+
    | FlexE Shim|   | Shim   |     |    |      | Shim |   | FlexE Shim|
    +-----------+   +--------+ ODU |    | ODU  +------+   +-----------+
    | PHY(s)    |   | PHY(s) | flex|    | flex |PHY(s)|   | PHY(s)    |
    +---+-------+   +---+----+--+--+    +---+--+---+--+   +---+-------+
    |               |           |           |      |          |
    +---------------+           +-----------+------+----------+


     =================================================================

       Figure 3: FlexE termination: FlexE clients at both endpoints

3.3.2.  Interworking of FlexE Client w/ Native Client at the other
        endpoint

   The OIF implementation agreement [OIFMLG3] currently supports FlexE
   client signals carried over one or more 100GBASE-R PHY(s).  There is
   a calendar of 5G timeslots associated with each PHY, and each FlexE
   client can make use of a number of timeslots (possibly distributed
   across the members of the FlexE group.  This implies that the FlexE
   client rates are multiples of 5Gbps.  When the rates of the FlexE
   client signals matches the MAC rates corresponding to existing
   Ethernet PHYs, i.e. 10GBASE-R/40GBASE-R/100GBASE-R, there is a need
   for the FlexE client signal to interwork with the native Ethernet
   client received from a single (non-FlexE capable) Ethernet PHY.  This
   capability is expected to be extended to any future Ethernet PHY



Hussain, et al.          Expires January 8, 2017               [Page 10]

Internet-Draft               FlexE Usecases                    July 2016


   rates that the IEEE may define in future (e.g. 25G, 50G, 200G etc.).
   In these cases, although the bit rate of the FlexE client matches the
   MAC rate of other endpoint, the 64B66B PCS codewords for the FlexE
   client need to be transformed (via ordered set translation) to match
   the specification for the specific Ethernt PHY.  These details are
   described in Section 7.2.2 of [OIFMLG3] and are not eloborated any
   further in this document.

   Figure 4 illustrates a scenario involving the interworking of a 10G
   FlexE client with a 10GBASE-R native Ethernet signal.  In this
   example, the network wrapper is ODU2e.

    ==================================================================

    +--------+ 2 x 100GE +-------+           +-------+      +--------+
    |        |           |       |           |       |      |        |
    | Router1|           |       |           |       |      |        |
    |(FlexE  +-----------+ A-end |           | Z-end | 10GE |Router 2|
    | Shim)  |           |(FlexE |           |       +------+        |
    |        +-----^-----+ term) |           |       |      |        |
    |        |     |     |       |           |       |      |        |
    |        |     |     |       |           |       |      |        |
    +--------+     +     +-------+           +-------+      +--------+
             FlexE Group
                        \+---------Transport---------+/
                                    network

    +-----------+   +---------------+
    | Client(s) |   | Client        |     +------------+    +---------+
    +-----------+   +-------+-------+     |   10GE PCS |    | 10GE PCS|
    | FlexE Shim|   | Shim  |       |     +-------+----+    +---------+
    +-----------+   +-------+  ODU  |     | ODU2e | PHY|    | PHY     |
    | PHY(s)    |   | PHY(s)|  2e   |     +---+---+--+-+    +-----+---+
    +---+-------+   +---+-------+---+         |      |            |
        |               |       |             |      |            |
        |               |       |             |      |            |
        +---------------+       +-------------+      +------------+


     =================================================================

        Figure 4: FlexE client interop with Native Ethernet Client

3.3.3.  Interworking of FlexE client w/ Client from OIF_MLG

   As explained in the Introduction section (Section 1 OIFMLG3 [OIFMLG3]
   introduced support for carrying 10GE and 40GE client signals over a
   group of 100GBASE-R Ethernet PHY(s).  While the most recent



Hussain, et al.          Expires January 8, 2017               [Page 11]

Internet-Draft               FlexE Usecases                    July 2016


   implementation agreement doesn't call it out explicitly, it is
   expected that the FlexE clients (as defined in [OIFFLEXE1]), and
   10GBASE-R/40GBASE-R clients supported by OIFMLG3 [OIFMLG3]) will
   interoperate.

   Figure 5 illustrates a scenario involving the interworking of a 10G
   FlexE client with a 10GBASE-R client supported by an OIFMLG3
   interface.  In this example, the network wrapper is ODU2e.

    ==================================================================

    +--------+ 2 x 100GE +---------+       +---------+      +---------+
    |        |           |         |       |         |      |         |
    | Router1|           |         |       |         |      |         |
    | FlexE  +-----------+ A-end   |       |  Z-end  +------+Router 2 |
    | Shim   |           | (FlexE  |       |         |      |(MLG-3.0)|
    |        +-----^-----+ term)   |       |         +------+         |
    |        |     |     |         |       |         |      |         |
    |        |     |     |         |       |         |      |         |
    +--------+     +     +---------+       +---------+      +---------+
              FlexE Group

                         \+--------+Transport+--------+/
                                    network

   +-----------+   +-------------+      +--------------+   +----------+
   | Client(s) |   | Client      |      | 10GE PCS     |   | 10GE Cl. |
   +-----------+   +--------+----+      +------+-------+   +----------+
   | FlexE Shim|   | Shim   |    |      |      | MLG3  |   | MLG3     |
   +-----------+   +--------+ ODU|      | ODU  +-------+   +----------+
   | PHY(s)    |   | PHY(s) | 2e |      | 2e   | PHY(s)|   | PHY(s)   |
   +---+-------+   +---+----+--+-+      +---+--+---+---+   +---+------+
       |               |       |            |      |            |
       +---------------+       +------------+      +------------+


     =================================================================

   Figure 5: FlexE client interop with Ethernet Client supported by MLG3

3.3.4.  FlexE Client BW Resizing

   This section covers an extension of the scenario presented in
   Section 3.3.1.  Each FlexE client signal defined in [OIFFLEXE1] has a
   rate which is a multiple of 5G, and occupies the required number of
   calendar slots (5G granularity) in the FlexE group (possibly
   distributed among the PHY(s) which have been bonded.  The OIF
   implementation agreement defines two calendars, one currently active,



Hussain, et al.          Expires January 8, 2017               [Page 12]

Internet-Draft               FlexE Usecases                    July 2016


   and the future calendar to which the sender wants to transition to.
   This capability can be used to coordinate a synchronized switchover
   of calendars between the two FlexE Shim functions -- one located in
   the customer eddge device (typically a router), and the transport
   network edge.  In this scenario, there are three independent resizing
   domains which must be coordinated (see Figure 3).

   a.  Between the router 1 and Transport edge node A-end

   b.  Between the transport edge nodes A, Z

   c.  Between the transport edge node Z, and router 2

   It is possible to coordinate the resize operations in these domains
   in such a manner that the FlexE clients get the benefit of an end-to-
   end bandwidth change (increase/decrease), without involving any
   additional provisioning steps in the provider network.  Note for the
   FlexE unaware use case (Section 3.1), the client BW can be resized by
   FlexE shim coordination between router 1 and router 2.

3.3.5.  Back-to-Back FlexE

   This section covers a degenerate FlexE aware scenario where router1,
   router2, and router3 are interconnected through back-to-back FlexE
   groups without an intermediate transport network (see Figure 6).

    ==================================================================

           +--------+ 2 x 100GE +---------+ 3 x 100GE +---------+
           |        |           |         |           |         |
           | Router1|           |         |           |         |
           | FlexE  +-----------+ Router2 +-----------+ Router3 |
           | Shim   |           | FlexE   +-----------+ FlexE   |
           |        +-----^-----+ Shim    +-----^-----+ Shim    |
           |        |     |     |         |     |     |         |
           |        |     |     |         |     |     |         |
           +--------+     +     +---------+     +     +---------+
                     FlexE Group           FlexE Group


     =================================================================

                       Figure 6: Back-to-Back FlexE








Hussain, et al.          Expires January 8, 2017               [Page 13]

Internet-Draft               FlexE Usecases                    July 2016


4.  FlexE Transport over Wavelength(s) Usecases

   The list of aforementioned FlexE usecases can also be supported by
   mapping FlexE directly over one or more wavelengths.  An example for
   the FlexE unaware transport over wavelength is depicted in Figure 7.
   Equivalent network diagrams for the other usecases can be obtained by
   replacing an OTN container with an Optical Channel (OCh).












































Hussain, et al.          Expires January 8, 2017               [Page 14]

Internet-Draft               FlexE Usecases                    July 2016


    ==================================================================

       +                            FlexE Ethernet Client(s)       +
       +-----------------------------------------------------------+
       +                                                           +
                        + FlexE skew tolerance
                        +----------------------------------------+
                        +  for end-to-distance                   +

   +-----------+ 2x100GE +---------+   +----------+     +------------+
   |           |         |         |   |          |     |            |
   | Router1   |         |         |   |          |     |            |
   |FlexE Shim +---------+ A-end   |   |  Z-end   +-----+Router 2    |
   |           |         | (FlexE  |   |  (FlexE  |     |(FlexE Shim)|
   |           +---^-----+ unaware)|   |  unaware)+-----+            |
   |           |   |     |         |   |          |     |            |
   |           |   |     |         |   |          |     |            |
   +-----------+   +     +---------+   +----------+     +------------+
                    FlexE Group

                        \----------Transport----------/
                                   network
   +--------------+                                  +----------------+
   | FlexE Clients|                                  | FlexE Client(s)|
   +--------------+                                  +----------------+
   | FlexE Shim   |                                  |  FlexE Shim    |
   +----+----+----+                                  +----+------+----+
   |PHY |  |  PHY |                                  |  PHY |   | PHY |
   +---+---+--+---+                                  +---+--+   +--+--+
       |      |          +-----+           +-----+       |         |
       |      +----------+ PHY |           | PHY |-------+         |
       |                 +-----+           +-----+                 |
       |                 | OCh +-----------+ OCh |                 |
       |                 +-----+           +-----+                 |
       |                                                           |
       |                 +-----+           +-----+                 |
       +-----------------+ PHY |           | PHY +-----------------+
                         +-----+           +-----+
                         | OCh +-----------+ OCh |
                         +-----+           +-----+


    ==================================================================

             Figure 7: FlexE unaware transport over wavelength






Hussain, et al.          Expires January 8, 2017               [Page 15]

Internet-Draft               FlexE Usecases                    July 2016


5.  Requirements

   This section summarizes solution requirements for the usecases
   described in this document to help identify the Control Plane (i.e.
   Routing and Signaling) extensions that may be required.

   a.  The solution SHALL support a FlexE group to address
       abovementioned usecases including FlexE unaware (where FlexE mux
       and demux can be separated by longer distances), FlexE aware
       (where FlexE mux and demux can be separated by shorter
       distances), and FlexE partially aware.

   b.  The solution SHALL support a flexible mechanism for configuring a
       FlexE group -- such as a signaling protocol or a SDN controller/
       management system with network access to the FlexE mux/demux at
       each end of the FlexE group.

   c.  The solution SHALL support the ability to add/remove Ethernet
       PHYs to/from a FlexE group.

   d.  The solution SHOULD allow decoupling of FlexE group's initial
       configuration and bring up operation from an addition (or
       removal) of FlexE clients to the FlexE group.  For instance, it
       SHOULD be possible to configure and bring up a FlexE group
       without any FlexE client (e.g., with all calendar slots set to
       unused or unavailable).

   e.  The solution SHALL allow adding or removing a FlexE client to a
       FlexE group without affecting traffic on other clients.

   f.  The solution SHALL allow resizing of FlexE client BW through
       coordination of calendar updates within a single FlexE group.
       There SHOULD be no expectation that FlexE client BW resizing be
       hitless in all network scenarios.

   g.  For the FlexE unaware case, each of the 100GBASE-R PHYs in the
       FlexE group SHALL be carried independently across transport
       network using a PCS codeword transparent mapping.  All PHYs of
       the FlexE group SHALL be interconnected between the same two
       FlexE shims.  The Ethernet PHYs SHOULD be carried over the same
       fiber route across the transport network (i.e., co-routed)

   h.  For the FlexE partially aware case, each of the 100GBASE-R PHYs
       in the FlexE group SHALL be carried independently across
       transport network.  All PHYs of the FlexE group SHALL be
       interconnected between the same two FlexE shims.  The Ethernet
       PHYs SHOULD be carried over the same fiber route across the
       transport network.  In the transport network, in mux direction,



Hussain, et al.          Expires January 8, 2017               [Page 16]

Internet-Draft               FlexE Usecases                    July 2016


       the OTN mapper SHALL be able to discard unavailable slots (e.g.,
       this can be based on static configuration as the rate of a
       wavelength is not expected to change in-service).  In the
       transport network, in the demux direction, the OTN mapper SHALL
       be able to restore unavailable slots to match the original PHY
       rate.

   i.  For the FlexE aware case, the FlexE group SHALL be terminated at
       the transport network edge.  It SHOULD be possible to carry
       (switch) each FlexE client extracted from the FlexE group
       independently across transport network using OTN mapping (e.g.,
       ODUflex).

6.  Acknowledgements

7.  IANA Considerations

   This memo includes no request to IANA.

8.  Security Considerations

   None.

9.  References

9.1.  Normative References

   [G709]     ITU, "Optical Transport Network Interfaces", February
              2016.

   [G798]     ITU, "Characteristics of optical transport network
              hierarchy equipment functional blocks", February 2012.

   [OIFFLEXE1]
              OIF, "FLex Ethernet Implementation Agreement Version 1.0
              (OIF-FLEXE-01.0)", March 2016.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

9.2.  Informative References

   [OIFMLG3]  OIF, "Multi-Lane Gearbox Implementation Agreement Version
              3.0 (OIF-MLG-3.0)", April 2016.





Hussain, et al.          Expires January 8, 2017               [Page 17]

Internet-Draft               FlexE Usecases                    July 2016


Appendix A.  Additional Stuff

   This becomes an Appendix.

Authors' Addresses

   Iftekhar Hussain
   Infinera Corp
   169 Java Drive
   Sunnyvale, CA  94089
   USA

   Phone: +1-408-572-5200
   Email: IHussain@infinera.com


   Radha Valiveti
   Infinera Corp
   169 Java Drive
   Sunnyvale, CA  94089
   USA

   Phone: +1-408-572-5200
   Email: rvaliveti@infinera.com


   Khuzema Pithewan
   Infinera Corp
   169 Java Drive
   Sunnyvale, CA  94089
   USA

   Phone: +1-408-572-5200
   Email: kpithewan@infinera.com

















Hussain, et al.          Expires January 8, 2017               [Page 18]