Back in 1999 I was working in South America. Most service providers in the region were operating TDM and FR networks and the hot technology they were evolving to was ATM. It promised many good things, including a true converged network capable or transporting data, voice and video. Of course many at the time were not thinking of packet data (nor packet video), but actual PCM voice channels over CES.
Why am I bringing this back today? Well, because back in those days I remember having so many conversations with SPs about delivering virtual private networks using IP technology, or in fact, a nascent technology that leveraged an IP control plane: tag switching, which evolved into standard MPLS. For Cisco, this was a key differentiation in the Stratacom product line of ATM switches.
But most people in those carriers were very circuit orientated. The easiest way to setup a private network for them was to mimic the TDM world of circuits and channels ... with a PVC - or a mesh of them.
It is funny to see that, in essence, today's latest and greatest solutions for implementing virtual private networks (network virtualization) still relies on circuits in some sort of way ... IP overlays now.
The kings of the ATM SP market at the time in the region were Nortel and NewBridge. Cisco had important franchises as well. SPs were looking for ways to scale better the circuit based ATM backbones to deliver services to end users (some were thinking about going into households even, which eventually happen through the earlier phases of ADSL deployments in early 2000s).
How to build and manage all those thousands of PVCs? ATM proposed a neat way ... SVCs! VCs which were created by a software layer and at the request of the application! ... Good things were about to happen. And then it came the fierce competition ...
Nortel and many others were pushing for PNNI, a routing protocol plane, network based, which would run distributed in all ATM switches in a hierarchical topology. PNNI was essentially a routing protocol for establishing ATM VCs. Good about it? It was standard ... promised vendor interoperability and in fact there were bake-offs to prove it and so on ... Cisco was betting on PNNI and actually had a pretty robust stack implemented on its LS1010s.
NewBridge on the other hand had a radically different approach. They had a very strong management solution (the 46020) coming along from the TDM and FR ages. Essentially for them, a software running on a server would control the setup and management of PVCs and SVCs alike! ... better, it could interface through APIs with an application layer for, say, deliver IP and VoD services to ADSL consumers for instance ...
The "SDN" approach to managing ATM never worked, it failed to scale and proved to be a lock-in for customers who could never think to imagine running an ATM switch that was not NewBridge.
Of course, the end of the story I just related is well know ... ATM and PNNI slowly died, being overruled by IP and MPLS. NewBridge was acquired by Acaltel (which actually reused its 46020 for managing the 7050/7040s boxes and did well with it), and Nortel ... well ... sorry for them.
I can't help to think of this and see some analogies with recent trends and announcements in the networking industry. By no means I pretend the end could be similar though ... much have changed in the compute industry in particular to help scale a controller managing overlay networks. But it is fun to think of the irony of things, and how circuits come back to haunt us IP heads ...
Showing posts with label Networking. Show all posts
Showing posts with label Networking. Show all posts
Friday, February 10, 2012
Saturday, October 15, 2011
On SDN, and OpenFlow - Are we solving any real problem?
I must admit that I started looking at SDN and OpenFlow with a lot of skepticism. It is not the first time I was facing a networking technology which proposed a centralized intelligence to setup the network paths for traffic to go through. Such thinking always reminds me of old circuit-based networks and NewBridge 46020, which proposed TDM path setup to FR and ATM, and when making in-roads into ADSL, offered it as the solution to all evils. ATM LANE was, in a different way, another "similar" attempt.
Being a long term CCIE makes me an IP-head, no doubt about it, and a controller-based network is something that I really need to open my mind to, in order to even consider it. I am trying though ...
The way I look at it, the question about SDN and OpenFlow alike is: what problem are we solving?
So far, I see three potential problems we are trying to solve:
- scale: help building more scalable networks spending less money
- complexity: simplify running a complex network
- agility: simplify or fasten implementing network services & network virtualization
In this blog post I try to look at SDN/OF from those three potential problems. I make little to no distinction between SDN and OF and that is wrong because they aren't quite the same thing. Things like 802.1x, or even LISP to some extent, could be somehow considered Software Defined Networking, in that the forwarding and/or the policy is defined in a "central" software engine or database lookup.
But for the purpose of this post, I really look at OF as THE way for implementing SDNs. But before I begin ...
A Common Comparison I consider flawed ...
Many people put existing Wireless LANs as a way to say the controller based approach is proven. Sure, look at wireless networks today, almost all are using a controller based approach ... well ... yes, but no. The biggest difference with OF from a networking perspective is that the controller FORWARDs all the traffic, which is tunneled in an overlay from each of the access points. So the controller IS part of the datapath, and an intrinsic part of it. In fact, can be the bottleneck.
Moreover, the capillarity of a WLAN network is orders of magnitude lower than that of a datacenter fabric, so any analogy is, IMHO, flawed.
Scalability
From a scalability perspective, my first take at SDN and OpenFlow was focused on two points which I looked at as big limitations:
1. a totally decoupled control plane (perhaps centralized - albeit distributed in a cluster) requires an out of band management network which could limit scale and reliability (plus add to the cost)
2. programming "flows" on device TCAMs using OF will not scale or at least will not provide any savings in CAPEX lead by networking hardware itself
I see point one above less as a limitation now, so long as we really succeed at achieving a large simplification of the network in all other areas (beyond management), thanks to the SDN approach.
It isn't unusual to have an OOB management network in tier-1 infrastructures anyways. In the SDN/OF approach however, the OOB network is really a critical asset (even more than critical ...). It must have redundancy with fast convergence built into it and be built to scale as well. We also need to factor the cost of running and operating this network too. Also, as the policy and/or number of flows becomes higher, the cost of the cluster itself may be non-negligible.
Point two from above is still one where I need to better understand how things will be done. A a first glance, I thought this was going to be a big limitation because I thought each network forwarding element would be managed like a remote linecard, programmed perhaps using OpenFlow. In such case, the hardware table sizes of each network element would limit the entire infrastructure because for L3 forwarding you want to keep consistent FIBs and for L2 forwarding even more to minimize flooding. Hence, if your forwarding elements are limited to, say, 16K IPv4 routes, that's the size of your OpenFlow network ... there are ways to optimize that by programing only part of the FIB, which is possible as the controller knows the topology. But then if there are flow exceptions ...
But then of course, things change if you consider that ... why would you need to do "normal" L2 or L3 lookups for switching packets? You can forget about L2 and L3 altogether (potentially). And then, I assume the "controller" could keep track of the capabilities of each node including table sizes, and program state only as needed and where needed. This adds complexity to the controller, but should help scaling.
But can this scale if hardware is programmed per flow?
I still don't see this happening really. Two issues with this: scale the flow setup (a software process), and scale the hardware flow tables. I understand the flow setup is not necessarily THE problem, but still, let's review it. Let's say the first packet is sent to the controller for the flow to be setup, all of this over the OOB network. This will add delay to initial communication and put load on the controller, but no reason why this can't scale out with multiple controller servers in a cluster or splitting the forwarding elements between different instances of the controller. All of this adds to the cost of the solution though (and add management complexity too).
But what I don't see is this scaling at the forwarding chip level. I wonder how to program hardware with the SDN approach. The OpenFlow way seems to be to do it on a per flow basis, leveraging table pipelining.
Of course it all depends on what do we call a flow. If we take source/destination IP addresses plus tcp/ip port, any aggregation switch will easily see hundreds of thousands of flows at any given time. Even at the ToR level the number of flows will ramp up rapidly. This will kill the best silicon available from vendors such as Broadcom, Fulcrum or Marvel so easily. We can indeed limit to source-destination mac addresses, or IP for that matter, but then that limits a lot what you CAN do with the packet flows. So if host A wants to communicate with host B that is two flows a->b and b->a. That is if you define a flow by source/destination mac address. In this case, let's assume you have 48 servers connected to a ToR switch. Let's say there are 10 VMs per server (40 is very common in today's Enterprises by the way). Let's say each VM needs to talk to two load balancers and/or default gateway plus to 10 other VMs. This means each VM would generate 12-14 flows. So this means each ToR switch would see 48 x 10 x 12 = 5,760 (times two, because they have to be bi-directional). Now that isn't too much, chips like Trident can fit 128K L2 entries, which in this case would mean flows if we define them as per mac-address. But think of the aggregation point which has 100+ ToR switches connected. Now those switches need to handle 576,000 flows (times two). Way more if you assume more than one vNic per VM.
At any rate, if you want to handle overlapping addresses, you also need to add some other field to the flow mask ... So I still don't see how this can scale at all, certainly not using "cheap" hardware.
In the end, if you want to run fast, you'll need to pay for a Ferrari, whether you drive it yourself or have a machine do it for you.
But I see a benefit if we can run the network forwarding elements in a totally different way than what we do today. The options for virtualizing the network can be much richer, this is true, but would come at a cost (discussed below). And also, I see a point to scale the network beyond what current network protocols allow which can be interesting. I certainly understand the interest from companies running very large datacenters, which tend to have very standardized network topologies which can benefit a lot from the SDN approach. But at the same time, I do not see why standard routing protocols can't be made to scale to larger networks too ...
I doubt OpenFlow will be the right approach in the enterprise for quite a while, because in that world, you rarely build from scratch 100%. There is always a need to accomodate for legacy and this will mandate for "traditional" networking for quite a while, no doubt (if SDN ever really takes up, that is). Sure, I know companies like Big Switch are looking at ways to use OF as an overlay into existing networks. We will have to wait and see how this works ...
Simplify Running a Complex Network
This point is a tough one. What is simple to some is complex to others. Someone with networking background will not consider that running an ISIS network which implements PIM-SM is difficult, while someone with software development background will see it very complicated. Likewise, the same software developer may think that running a cluster of servers which control other "servers" (that each do packet forwarding) is very simple.
SDN looks, on powerpoint, very promising for network management simplification. But when you begin to dig in for details, on how to do troubleshooting, how to look deep into the hardware forwarding tables, how to ensure path continuity or simply test it etc, you begin to see that what was simple in concept, becomes more complex.
Simplify Network Services
A lot of the writing I have seen around this focuses on the idea that once we have a "standard" way of programming the forwarding hardware (OpenFlow, that is), then all forwarding actions become sort of like instruction sets on a general purpose CPU. Hence, all network problems become solvable by writing programs that operate on the network using such instructions.
I have seen typically two examples given for quick-wins of this approach: load balancing and network virtualization. Both hot topics on any data center. Others point to fancier ones, like unequal load balancing, load based routing, or even shutting down unused nodes. The latter speaks for itself and is foolish thinking ...
All others CAN be done with traditional networking technologies, and if they are not implemented is typically for very good reasons.
On the point of load balancing and network virtualization, what I have seen so far are discussions at very high level, which show how this can indeed be done. OpenFlow-heads praise how this is going to be not only simple, but even free! ...
Implement a load balancer, nothing simpler and cheaper. The fantastic OF Controller will simply load balance flows depending on the src address for instance. Done. Zero dollars to it. Of course, it obviates the point of hardware (and software) flow table scalability - already mentioned above. Of course it obviates the fact that a load balancer does A LOT MORE than push packets out of various ports depending on source address ... it keeps track of real ip addresses, polls the application to measure load, off loads server networking tasks, etc. There's a reason why people invest in appliances from Cisco or F5 to do load balancing. Switches (from multiple vendors) have been able to do server load balancing for a long time, but what you can do there isn't just enough for most applications. OF changes nothing there.
Network Virtualization is another one where the complex becomes so simple thanks to SDN and OF. I admit to write here from ignorance of the actual work of companies like Big Switch or Nicira, of course. But most of what I read resolves into implementing an overlay network with a control plane running on software. Nothing different to many other approaches today and what would be done using VXLAN or NVGRE. At any rate, I would argue LISP is a better choice, but alas, it does not solve L2 adjacencies which are required for clustering and other reasons (which OF doesn't solve either).
I have seen many others proposing that thanks to OF, one can easily program the controller to use fields like MPLS tags, or VLAN tags, etc. to do segmentation ala carte. This is true, and fine. But I wonder, how is this good?!
And how is it different from doing traditional networking? Sure, Cisco, Brocade, Juniper, F10 and others could have decided to change the semantics of existing network fields to implement segmentation and many other features. And sometimes we have seen this done. But in so doing, they become proprietary. They don't interoperate.
IF a controller software vendor X provides a virtualization solution that works that way (by redefining the semantics of existing fields), it offers a solution that locks-in the customer with that software vendor. A solution for which a tailor-made gateway will be required to connect with a standards based network, or to a network from any other company.
Imagine company Z who runs a DC with a controller from software vendor X. Imagine company Z merges with company Y, who runs with a controller from a different vendor, ... imagine the trouble. Today, company Z running Juniper merges with company Y running Cisco and they connect their networks using OSPF or BGP, or just plain STP if they are not very skilled ...
I am sure I am missing the obvious when so many bright people praise OpenFlow. I just can't see how it solves problems that we can't solve today, or how it does it better. Or how it really will make it for a better industry. Many would like to see a world where networking wont be dominated by two vendors. OF, at best, could change who those two vendors are, nothing more.
Conclusion ... (for now)
I think OpenFlow is a very interesting technology, and the SDN paradigm one that can contribute to many good things in the industry. But up until today, almost everything that I read about both topics is of very high level, and idealistic to the point of being naive. In my opinion, a lot of the assumptions made on the problems that OpenFlow will solve are made without knowledge of other existing solutions. I am yet to see an expert in IP and MPLS praise OpenFlow for solving problems that we couldn't solve with those two, or solve them in a clearly advantageous way.
So far to me, from my ignorance of things I admit, OpenFlow looks at just a different way to do the same. And for that ... what for?
Being a long term CCIE makes me an IP-head, no doubt about it, and a controller-based network is something that I really need to open my mind to, in order to even consider it. I am trying though ...
The way I look at it, the question about SDN and OpenFlow alike is: what problem are we solving?
So far, I see three potential problems we are trying to solve:
- scale: help building more scalable networks spending less money
- complexity: simplify running a complex network
- agility: simplify or fasten implementing network services & network virtualization
In this blog post I try to look at SDN/OF from those three potential problems. I make little to no distinction between SDN and OF and that is wrong because they aren't quite the same thing. Things like 802.1x, or even LISP to some extent, could be somehow considered Software Defined Networking, in that the forwarding and/or the policy is defined in a "central" software engine or database lookup.
But for the purpose of this post, I really look at OF as THE way for implementing SDNs. But before I begin ...
A Common Comparison I consider flawed ...
Many people put existing Wireless LANs as a way to say the controller based approach is proven. Sure, look at wireless networks today, almost all are using a controller based approach ... well ... yes, but no. The biggest difference with OF from a networking perspective is that the controller FORWARDs all the traffic, which is tunneled in an overlay from each of the access points. So the controller IS part of the datapath, and an intrinsic part of it. In fact, can be the bottleneck.
Moreover, the capillarity of a WLAN network is orders of magnitude lower than that of a datacenter fabric, so any analogy is, IMHO, flawed.
Scalability
From a scalability perspective, my first take at SDN and OpenFlow was focused on two points which I looked at as big limitations:
1. a totally decoupled control plane (perhaps centralized - albeit distributed in a cluster) requires an out of band management network which could limit scale and reliability (plus add to the cost)
2. programming "flows" on device TCAMs using OF will not scale or at least will not provide any savings in CAPEX lead by networking hardware itself
I see point one above less as a limitation now, so long as we really succeed at achieving a large simplification of the network in all other areas (beyond management), thanks to the SDN approach.
It isn't unusual to have an OOB management network in tier-1 infrastructures anyways. In the SDN/OF approach however, the OOB network is really a critical asset (even more than critical ...). It must have redundancy with fast convergence built into it and be built to scale as well. We also need to factor the cost of running and operating this network too. Also, as the policy and/or number of flows becomes higher, the cost of the cluster itself may be non-negligible.
Point two from above is still one where I need to better understand how things will be done. A a first glance, I thought this was going to be a big limitation because I thought each network forwarding element would be managed like a remote linecard, programmed perhaps using OpenFlow. In such case, the hardware table sizes of each network element would limit the entire infrastructure because for L3 forwarding you want to keep consistent FIBs and for L2 forwarding even more to minimize flooding. Hence, if your forwarding elements are limited to, say, 16K IPv4 routes, that's the size of your OpenFlow network ... there are ways to optimize that by programing only part of the FIB, which is possible as the controller knows the topology. But then if there are flow exceptions ...
But then of course, things change if you consider that ... why would you need to do "normal" L2 or L3 lookups for switching packets? You can forget about L2 and L3 altogether (potentially). And then, I assume the "controller" could keep track of the capabilities of each node including table sizes, and program state only as needed and where needed. This adds complexity to the controller, but should help scaling.
But can this scale if hardware is programmed per flow?
I still don't see this happening really. Two issues with this: scale the flow setup (a software process), and scale the hardware flow tables. I understand the flow setup is not necessarily THE problem, but still, let's review it. Let's say the first packet is sent to the controller for the flow to be setup, all of this over the OOB network. This will add delay to initial communication and put load on the controller, but no reason why this can't scale out with multiple controller servers in a cluster or splitting the forwarding elements between different instances of the controller. All of this adds to the cost of the solution though (and add management complexity too).
But what I don't see is this scaling at the forwarding chip level. I wonder how to program hardware with the SDN approach. The OpenFlow way seems to be to do it on a per flow basis, leveraging table pipelining.
Of course it all depends on what do we call a flow. If we take source/destination IP addresses plus tcp/ip port, any aggregation switch will easily see hundreds of thousands of flows at any given time. Even at the ToR level the number of flows will ramp up rapidly. This will kill the best silicon available from vendors such as Broadcom, Fulcrum or Marvel so easily. We can indeed limit to source-destination mac addresses, or IP for that matter, but then that limits a lot what you CAN do with the packet flows. So if host A wants to communicate with host B that is two flows a->b and b->a. That is if you define a flow by source/destination mac address. In this case, let's assume you have 48 servers connected to a ToR switch. Let's say there are 10 VMs per server (40 is very common in today's Enterprises by the way). Let's say each VM needs to talk to two load balancers and/or default gateway plus to 10 other VMs. This means each VM would generate 12-14 flows. So this means each ToR switch would see 48 x 10 x 12 = 5,760 (times two, because they have to be bi-directional). Now that isn't too much, chips like Trident can fit 128K L2 entries, which in this case would mean flows if we define them as per mac-address. But think of the aggregation point which has 100+ ToR switches connected. Now those switches need to handle 576,000 flows (times two). Way more if you assume more than one vNic per VM.
At any rate, if you want to handle overlapping addresses, you also need to add some other field to the flow mask ... So I still don't see how this can scale at all, certainly not using "cheap" hardware.
In the end, if you want to run fast, you'll need to pay for a Ferrari, whether you drive it yourself or have a machine do it for you.
But I see a benefit if we can run the network forwarding elements in a totally different way than what we do today. The options for virtualizing the network can be much richer, this is true, but would come at a cost (discussed below). And also, I see a point to scale the network beyond what current network protocols allow which can be interesting. I certainly understand the interest from companies running very large datacenters, which tend to have very standardized network topologies which can benefit a lot from the SDN approach. But at the same time, I do not see why standard routing protocols can't be made to scale to larger networks too ...
I doubt OpenFlow will be the right approach in the enterprise for quite a while, because in that world, you rarely build from scratch 100%. There is always a need to accomodate for legacy and this will mandate for "traditional" networking for quite a while, no doubt (if SDN ever really takes up, that is). Sure, I know companies like Big Switch are looking at ways to use OF as an overlay into existing networks. We will have to wait and see how this works ...
Simplify Running a Complex Network
This point is a tough one. What is simple to some is complex to others. Someone with networking background will not consider that running an ISIS network which implements PIM-SM is difficult, while someone with software development background will see it very complicated. Likewise, the same software developer may think that running a cluster of servers which control other "servers" (that each do packet forwarding) is very simple.
SDN looks, on powerpoint, very promising for network management simplification. But when you begin to dig in for details, on how to do troubleshooting, how to look deep into the hardware forwarding tables, how to ensure path continuity or simply test it etc, you begin to see that what was simple in concept, becomes more complex.
Simplify Network Services
A lot of the writing I have seen around this focuses on the idea that once we have a "standard" way of programming the forwarding hardware (OpenFlow, that is), then all forwarding actions become sort of like instruction sets on a general purpose CPU. Hence, all network problems become solvable by writing programs that operate on the network using such instructions.
I have seen typically two examples given for quick-wins of this approach: load balancing and network virtualization. Both hot topics on any data center. Others point to fancier ones, like unequal load balancing, load based routing, or even shutting down unused nodes. The latter speaks for itself and is foolish thinking ...
All others CAN be done with traditional networking technologies, and if they are not implemented is typically for very good reasons.
On the point of load balancing and network virtualization, what I have seen so far are discussions at very high level, which show how this can indeed be done. OpenFlow-heads praise how this is going to be not only simple, but even free! ...
Implement a load balancer, nothing simpler and cheaper. The fantastic OF Controller will simply load balance flows depending on the src address for instance. Done. Zero dollars to it. Of course, it obviates the point of hardware (and software) flow table scalability - already mentioned above. Of course it obviates the fact that a load balancer does A LOT MORE than push packets out of various ports depending on source address ... it keeps track of real ip addresses, polls the application to measure load, off loads server networking tasks, etc. There's a reason why people invest in appliances from Cisco or F5 to do load balancing. Switches (from multiple vendors) have been able to do server load balancing for a long time, but what you can do there isn't just enough for most applications. OF changes nothing there.
Network Virtualization is another one where the complex becomes so simple thanks to SDN and OF. I admit to write here from ignorance of the actual work of companies like Big Switch or Nicira, of course. But most of what I read resolves into implementing an overlay network with a control plane running on software. Nothing different to many other approaches today and what would be done using VXLAN or NVGRE. At any rate, I would argue LISP is a better choice, but alas, it does not solve L2 adjacencies which are required for clustering and other reasons (which OF doesn't solve either).
I have seen many others proposing that thanks to OF, one can easily program the controller to use fields like MPLS tags, or VLAN tags, etc. to do segmentation ala carte. This is true, and fine. But I wonder, how is this good?!
And how is it different from doing traditional networking? Sure, Cisco, Brocade, Juniper, F10 and others could have decided to change the semantics of existing network fields to implement segmentation and many other features. And sometimes we have seen this done. But in so doing, they become proprietary. They don't interoperate.
IF a controller software vendor X provides a virtualization solution that works that way (by redefining the semantics of existing fields), it offers a solution that locks-in the customer with that software vendor. A solution for which a tailor-made gateway will be required to connect with a standards based network, or to a network from any other company.
Imagine company Z who runs a DC with a controller from software vendor X. Imagine company Z merges with company Y, who runs with a controller from a different vendor, ... imagine the trouble. Today, company Z running Juniper merges with company Y running Cisco and they connect their networks using OSPF or BGP, or just plain STP if they are not very skilled ...
I am sure I am missing the obvious when so many bright people praise OpenFlow. I just can't see how it solves problems that we can't solve today, or how it does it better. Or how it really will make it for a better industry. Many would like to see a world where networking wont be dominated by two vendors. OF, at best, could change who those two vendors are, nothing more.
Conclusion ... (for now)
I think OpenFlow is a very interesting technology, and the SDN paradigm one that can contribute to many good things in the industry. But up until today, almost everything that I read about both topics is of very high level, and idealistic to the point of being naive. In my opinion, a lot of the assumptions made on the problems that OpenFlow will solve are made without knowledge of other existing solutions. I am yet to see an expert in IP and MPLS praise OpenFlow for solving problems that we couldn't solve with those two, or solve them in a clearly advantageous way.
So far to me, from my ignorance of things I admit, OpenFlow looks at just a different way to do the same. And for that ... what for?
Sunday, July 24, 2011
Yes, the math is misleading ...
I have had more people reading this that I anticipated, and a couple of good comments which I want to reply to in a proper way, so I decided to make it another post ...
Mark, thank you for your comments. They really add to my post as they provide more insight and context, which was lacking on the interview: the hypothetical 3MW DC is for a cloud provider built on L3 from the access.
And in that sense, I stand by my comment: the math presented at that interview is indeed misleading. I will explain below why I say this ... But first to your comments :-)
1. When I mentioned "claim to support 384 10GE ports" I do not imply that I doubt it to be the reality. I am writing here based on my knowledge and experience only :-) I take your word fot it of course.
2. Good catch on the Arista 7050-64 power consumption. Believe me it was not intentional. I have corrected this :-)
3. Agreed that cloud providers like standardization and could prefer to operationalize cabling with a pair of ToRs per rack. Again, at the interview it wasn't mentioned which kind of customer we are talking about ... Enterprise customers could think differently and would very much welcome racking ToRs every other rack because it simplifies the network operation by quite a bit (manage 84 devices vs 250).
4. There was no "Cisco defense" because I do not take this as any attack :-) ... and I write here for the fun of it. I simply stated the fact that you picked on platforms which don't allow a fair comparison. You say that Cisco's best practice design is with 5548 and FCoE ... Where did you read a paper from Cisco that recommends such a thing for an MSDP? ... However an Enterprise datacenter with a high degree of virtualization will in many cases require to support storage targets on FC, and FCoE is great solution to optimize the cost of deploying such infrastructure. L2 across the racks is also a common requirement in this case, not just for vMotion, but for supporting many cluster applications. L3 access in most Enterprise DC simply (and sadly) does not apply ...
As I said, your comment brings insight because in the interview there is no comment of what was the hypothetical datacenter for, or what kind of logical design you would use. No context at all. You say it is to be built using L3 at the access layer. I concur with you that the limitations I mentioned do not necessarily apply, for the access switches wont need to learn that many ARP entries, and the distribution switches can work with route summarization from the access, so smaller tables could do the job.
I am well aware that most MSDPs use L3 from the ToR layer. I am sure that you are also well aware that most Enterprise virtualized and non-virtualized datacenters, do not use L3 from the ToR. A 5,000 server DC could be found in either market space. I am sure you are also well aware that Cisco's recommended design with Nexus 5500 and Nexus 7000 (also leveraging FCoE) is thought for Enterprise datacenters primarily where it provides a lot of flexibility.
I still can't see how the topology looks like in your design anyways, with 12 spine switches. I just cannot see how to add up 16 uplinks per ToR/leaf to 12 spine switches and keep ECMPs ... I guess that the design is made up of pairs, so six pairs of 7500s, each then aggregating about 40 ToRs, with 8 links from ToR to each of the 7500s. But if this is the case, each of the 7500s pair has no connection to the other pairs unless a core layer is deployed, which perhaps is assumed in the example (but not factored in for power calculations??).
So again, I think your comment confirms my point, while also acknowledge that for non-L3 designs, the limitations that I mentioned do apply.
What do I mean that you confirm my point?: The math in the interview is presented without context, so it creates the perception that the comparison applies to any and every environment. In fact, it only applies in a specific environment (L3 design), and completely reverses in others (L2 desing ... probably a more common one btw). So you pick an area to your advantage, exaggerated it (by picking Cisco's most sub-optimal platform for such design), and presented it as general. That is misleading.
To answer to Doug ... Let me first say that I respect you a lot, and I have very very much respect for Arista. It is a company with a lot of talent. But talent lives in many places, including Cisco too. My blog post was to state that the math is misleading. Nothing more. I still believe it is, as stated above. You say "Nillo had to significantly modify the design, away from the specification" ... what? :-) ... what specification? None was provided during the interview ... how could I change it?!
To your challenge, two things to say:
1. Can YOU spec a 3MW datacenter where you can do life vMotion between any two racks and allow any server to mount any storage target from any point in the network? :-D
2. I write here for the fun of it and on my free time ... oh ... and I choose what I write about ;-) (not really the "when", because my free time isn't always mine ;-) ).
... I thought nobody would care a bit about what I write ... wow ...
Mark, thank you for your comments. They really add to my post as they provide more insight and context, which was lacking on the interview: the hypothetical 3MW DC is for a cloud provider built on L3 from the access.
And in that sense, I stand by my comment: the math presented at that interview is indeed misleading. I will explain below why I say this ... But first to your comments :-)
1. When I mentioned "claim to support 384 10GE ports" I do not imply that I doubt it to be the reality. I am writing here based on my knowledge and experience only :-) I take your word fot it of course.
2. Good catch on the Arista 7050-64 power consumption. Believe me it was not intentional. I have corrected this :-)
3. Agreed that cloud providers like standardization and could prefer to operationalize cabling with a pair of ToRs per rack. Again, at the interview it wasn't mentioned which kind of customer we are talking about ... Enterprise customers could think differently and would very much welcome racking ToRs every other rack because it simplifies the network operation by quite a bit (manage 84 devices vs 250).
4. There was no "Cisco defense" because I do not take this as any attack :-) ... and I write here for the fun of it. I simply stated the fact that you picked on platforms which don't allow a fair comparison. You say that Cisco's best practice design is with 5548 and FCoE ... Where did you read a paper from Cisco that recommends such a thing for an MSDP? ... However an Enterprise datacenter with a high degree of virtualization will in many cases require to support storage targets on FC, and FCoE is great solution to optimize the cost of deploying such infrastructure. L2 across the racks is also a common requirement in this case, not just for vMotion, but for supporting many cluster applications. L3 access in most Enterprise DC simply (and sadly) does not apply ...
As I said, your comment brings insight because in the interview there is no comment of what was the hypothetical datacenter for, or what kind of logical design you would use. No context at all. You say it is to be built using L3 at the access layer. I concur with you that the limitations I mentioned do not necessarily apply, for the access switches wont need to learn that many ARP entries, and the distribution switches can work with route summarization from the access, so smaller tables could do the job.
I am well aware that most MSDPs use L3 from the ToR layer. I am sure that you are also well aware that most Enterprise virtualized and non-virtualized datacenters, do not use L3 from the ToR. A 5,000 server DC could be found in either market space. I am sure you are also well aware that Cisco's recommended design with Nexus 5500 and Nexus 7000 (also leveraging FCoE) is thought for Enterprise datacenters primarily where it provides a lot of flexibility.
I still can't see how the topology looks like in your design anyways, with 12 spine switches. I just cannot see how to add up 16 uplinks per ToR/leaf to 12 spine switches and keep ECMPs ... I guess that the design is made up of pairs, so six pairs of 7500s, each then aggregating about 40 ToRs, with 8 links from ToR to each of the 7500s. But if this is the case, each of the 7500s pair has no connection to the other pairs unless a core layer is deployed, which perhaps is assumed in the example (but not factored in for power calculations??).
So again, I think your comment confirms my point, while also acknowledge that for non-L3 designs, the limitations that I mentioned do apply.
What do I mean that you confirm my point?: The math in the interview is presented without context, so it creates the perception that the comparison applies to any and every environment. In fact, it only applies in a specific environment (L3 design), and completely reverses in others (L2 desing ... probably a more common one btw). So you pick an area to your advantage, exaggerated it (by picking Cisco's most sub-optimal platform for such design), and presented it as general. That is misleading.
To answer to Doug ... Let me first say that I respect you a lot, and I have very very much respect for Arista. It is a company with a lot of talent. But talent lives in many places, including Cisco too. My blog post was to state that the math is misleading. Nothing more. I still believe it is, as stated above. You say "Nillo had to significantly modify the design, away from the specification" ... what? :-) ... what specification? None was provided during the interview ... how could I change it?!
To your challenge, two things to say:
1. Can YOU spec a 3MW datacenter where you can do life vMotion between any two racks and allow any server to mount any storage target from any point in the network? :-D
2. I write here for the fun of it and on my free time ... oh ... and I choose what I write about ;-) (not really the "when", because my free time isn't always mine ;-) ).
... I thought nobody would care a bit about what I write ... wow ...
Monday, July 11, 2011
One layer, two layers … how many do your really need?
I do not know how many times over the last few months I have had to deal with the "we want a network with less layers, one layer is the ideal" comment. In fact it is a recurring question. The number of customers that over the past 10 years has asked me why do we need a core, or whether they can collapse aggregation and access is significant.
In the end, it comes down to the perception that by collapsing you may save money, on smaller deployments, and to simplify the network, on larger ones. I think that this perception is often wrong, and the truth is these days several networking vendors are trying to exploit it in a misleading way.
Why and when do you need more than one layer?
I will look at this from a data center perspective, so I will consider that end-points connecting to the network are primarily servers. But the analysis is similar for campus, with a different set of endpoints (desktops, laptops, wireless access points, printers, badge readers, etc.) which have very different bandwidth & security requirements.
If the number of servers you have fits in a single switch, then you can do with a one layer approach [I know, some people may read this and be thinking "some solutions out there convert the network in a single switch" … I'll get to those in other blog posts too].
So from a physical point of view this is obvious: if you can connect all your servers to one single switch, you need one layer. Now you want to have redundancy, so you will want the servers to connect each to at least two switches. So you have two switches which then connect to the rest of your network.
If you have 40 servers with just two NIC per server, you can probably do with a pair of 48-port switches. What if you have 400 instead of 40? Then you use a switch with +400-ports and you are done. But well, then you realize that you have to put that big switch somehow far from your servers and you spend lots of money on fiber cables and also complicate operations.
What if the switch needs to be replaced/upgraded? You need to impact ALL your servers ... And also, what if I need 4,000 servers now instead of 400?
So you say it is best to use smaller switches and place them closer to your servers. Top of Rack is where most people put them. Why is this good?
- can use less fiber between racks
- can use copper within the rack (cheaper)
- simplifies (physical) operations [i.e. when you need to upgrade/replace the switch you impact less devices]
So most people would connect servers to a pair of smaller switches placed at the Top of the Rack (ToR). Now you have to interconnect those ToR switches together. So we need a 2-layer network. The switch that we use to interconnect the ToR switches would be called a "distribution" or "aggregation" layer switch. Because you want redundancy, you would put two of those.
Pretty basic so far. In practical terms, there is almost no way to get away of a 2-layer network.
Ok, but more than 2 layers is really an overkill! This is just vendors trying to rip money off of people!
Same principle applies to begin with. A pair of distribution switches would have a limited number of ports. Let's say for simplification that you use two uplinks per ToR switch, one to each distribution switch. If you have 40 ToR switches you MAY do with say a 48-port switch for distribution [notice the MAY in uppercase, for there are considerations other than the number of switchports here].
Say you have 400 ToR devices now ... you need a pair of +400-port switches in distribution layer. Say you have 4,000 ToR devices ... Not so easy to think of a single device with 4,000 ports. But the key point here now is say you have 40,000, or more ... How do you make sure that whatever the number, you know what to do and can grow the network.
This is where a third layer and a hierarchical modular design MAY come in place [notice the upper case on MAY, for there are other options here, discussed below]. By adding another layer, you multiply the scalability. Now you can design PODs of distribution pairs with a known capacity (based on the number of ports and other parameters like size of forwarding tables, scalability of control plane, etc) which you can interconnect with one another through a core layer.
So, do you need a core layer? Depends on the size of your network. But the point is that regardless of what any vendor may claim today, as the network grows above the physical limits of the devices you use to aggregate access switches, you need to add another layer to make the network grow in a scalable way.
OK, but I can also grow with a 2-layer design ...
Yes, and this is becoming more and more popular. In such architecture the ToR switch is typically defined as a 'leaf' switch, which connects to a 'spine' switch (spine would be similar to the distribution switch). You can have multiple 'spine' switches, so you can scale the network to a higher bandwidth for each ToR. If a ToR requires more than two uplinks, you can add more spine switches to provide more bandwidth.
But the size of the network measured by the number of leaf/ToR switches is still limited by the maximum port density of the spine/distribution switch. When you exceed that number you are bound to use another layer.
More reasons to consider multiple layers ...
So far we have seen the obvious physical reasons calling for 2 or 3 layer design. Needless to say that as platforms become denser in number of supported ports with each product generation, what before required 3 layers may be later accomplish in 2. But at scale, you always need to fall into multiple physical layers.
But there are other reasons why you want to use multiple layers, and those are operational more than anything. If you have all your servers connected to a pair of switches, or for that matter all your ToRs connected to a pair of switches which ALSO connect to the rest of the network, maintenance, replacements, and upgrades on those switches may have a big impact on the overall datacenter.
So in practical terms, it may be interesting to reduce the size of a distribution pair even if that requires adding a third layer, so that you reduce the failure domain. What I mean here is that let's say you have 4,000 servers and you can hook them all off of a single pair of distribution switches. It may be interesting to think of doing two PODs of 2,000 each even if that requires using double the number of distribution switches. Why? A serious problem in your pair of switches will not bring down all your computing capacity.
Of course people would argue "hey, I already have TWO distribution switches for THAT reason". Fair enough. But if you goof an update or you are impacted by a software bug that is data-plane related (i.e. related to traffic through the switches) there is a chance that you will affect both. This is rare, but happens.
The 2-layer approach already shields you in that level for when you need to upgrade your access switches, say to deploy a new feature or correct a software bug, you can test on a sandbox switch first, and then deploy the upgrade serially so you make sure there is no impact to the network.
There are other reasons to have a hierarchical network (with 2 or 3 layers depending on size), such as network protocol scaling by allowing aggregation (again, comes to minimize size of failure domain, this time on L3 control plane), applying policy in hierarchical way (allows scaling hardware reasources) and others, but I am not touching on any of these on purpose. The reason is that there are some that now would dispute that if you do NOT rely on network control protocols you get away with their limitations. This is certainly one of the ideas behind most people working on SDN and OpenFlow, as well as on some vendor proprietary implementations.
What I wanted to do with this post is prove that regardless of the network control plane approach (standards based with distributed control - OSPF, ISIS, TRILL -, standards based with centralized control - i.e. OpenFlow -, or proprietary implementations - i.e. BrocadeOne or JNPR's QFabric), at scale you ALWAYS need more than one physical network layer, and in large scale datacenters it is almost impossible to move away from three.
So from a physical point of view this is obvious: if you can connect all your servers to one single switch, you need one layer. Now you want to have redundancy, so you will want the servers to connect each to at least two switches. So you have two switches which then connect to the rest of your network.
If you have 40 servers with just two NIC per server, you can probably do with a pair of 48-port switches. What if you have 400 instead of 40? Then you use a switch with +400-ports and you are done. But well, then you realize that you have to put that big switch somehow far from your servers and you spend lots of money on fiber cables and also complicate operations.
What if the switch needs to be replaced/upgraded? You need to impact ALL your servers ... And also, what if I need 4,000 servers now instead of 400?
So you say it is best to use smaller switches and place them closer to your servers. Top of Rack is where most people put them. Why is this good?
- can use less fiber between racks
- can use copper within the rack (cheaper)
- simplifies (physical) operations [i.e. when you need to upgrade/replace the switch you impact less devices]
So most people would connect servers to a pair of smaller switches placed at the Top of the Rack (ToR). Now you have to interconnect those ToR switches together. So we need a 2-layer network. The switch that we use to interconnect the ToR switches would be called a "distribution" or "aggregation" layer switch. Because you want redundancy, you would put two of those.
Pretty basic so far. In practical terms, there is almost no way to get away of a 2-layer network.
Ok, but more than 2 layers is really an overkill! This is just vendors trying to rip money off of people!
Same principle applies to begin with. A pair of distribution switches would have a limited number of ports. Let's say for simplification that you use two uplinks per ToR switch, one to each distribution switch. If you have 40 ToR switches you MAY do with say a 48-port switch for distribution [notice the MAY in uppercase, for there are considerations other than the number of switchports here].
Say you have 400 ToR devices now ... you need a pair of +400-port switches in distribution layer. Say you have 4,000 ToR devices ... Not so easy to think of a single device with 4,000 ports. But the key point here now is say you have 40,000, or more ... How do you make sure that whatever the number, you know what to do and can grow the network.
This is where a third layer and a hierarchical modular design MAY come in place [notice the upper case on MAY, for there are other options here, discussed below]. By adding another layer, you multiply the scalability. Now you can design PODs of distribution pairs with a known capacity (based on the number of ports and other parameters like size of forwarding tables, scalability of control plane, etc) which you can interconnect with one another through a core layer.
So, do you need a core layer? Depends on the size of your network. But the point is that regardless of what any vendor may claim today, as the network grows above the physical limits of the devices you use to aggregate access switches, you need to add another layer to make the network grow in a scalable way.
OK, but I can also grow with a 2-layer design ...
Yes, and this is becoming more and more popular. In such architecture the ToR switch is typically defined as a 'leaf' switch, which connects to a 'spine' switch (spine would be similar to the distribution switch). You can have multiple 'spine' switches, so you can scale the network to a higher bandwidth for each ToR. If a ToR requires more than two uplinks, you can add more spine switches to provide more bandwidth.
But the size of the network measured by the number of leaf/ToR switches is still limited by the maximum port density of the spine/distribution switch. When you exceed that number you are bound to use another layer.
More reasons to consider multiple layers ...
So far we have seen the obvious physical reasons calling for 2 or 3 layer design. Needless to say that as platforms become denser in number of supported ports with each product generation, what before required 3 layers may be later accomplish in 2. But at scale, you always need to fall into multiple physical layers.
But there are other reasons why you want to use multiple layers, and those are operational more than anything. If you have all your servers connected to a pair of switches, or for that matter all your ToRs connected to a pair of switches which ALSO connect to the rest of the network, maintenance, replacements, and upgrades on those switches may have a big impact on the overall datacenter.
So in practical terms, it may be interesting to reduce the size of a distribution pair even if that requires adding a third layer, so that you reduce the failure domain. What I mean here is that let's say you have 4,000 servers and you can hook them all off of a single pair of distribution switches. It may be interesting to think of doing two PODs of 2,000 each even if that requires using double the number of distribution switches. Why? A serious problem in your pair of switches will not bring down all your computing capacity.
Of course people would argue "hey, I already have TWO distribution switches for THAT reason". Fair enough. But if you goof an update or you are impacted by a software bug that is data-plane related (i.e. related to traffic through the switches) there is a chance that you will affect both. This is rare, but happens.
The 2-layer approach already shields you in that level for when you need to upgrade your access switches, say to deploy a new feature or correct a software bug, you can test on a sandbox switch first, and then deploy the upgrade serially so you make sure there is no impact to the network.
There are other reasons to have a hierarchical network (with 2 or 3 layers depending on size), such as network protocol scaling by allowing aggregation (again, comes to minimize size of failure domain, this time on L3 control plane), applying policy in hierarchical way (allows scaling hardware reasources) and others, but I am not touching on any of these on purpose. The reason is that there are some that now would dispute that if you do NOT rely on network control protocols you get away with their limitations. This is certainly one of the ideas behind most people working on SDN and OpenFlow, as well as on some vendor proprietary implementations.
What I wanted to do with this post is prove that regardless of the network control plane approach (standards based with distributed control - OSPF, ISIS, TRILL -, standards based with centralized control - i.e. OpenFlow -, or proprietary implementations - i.e. BrocadeOne or JNPR's QFabric), at scale you ALWAYS need more than one physical network layer, and in large scale datacenters it is almost impossible to move away from three.
Friday, June 17, 2011
Why FC has a brighter future ahead thanks to FCoE
I must admit that I have been an sceptic about FCoE since Cisco introduced it in 2008 up until recent times. With limited storage knowledge and background, I though that going forward NAS would prevail for most environments, and iSCSI would be the winning option for block based storage needs. Now I think there's going to be space for all of these, but FC has a better chance when dealing with block storage.
Why am I changing my mind? Well, a bit more knowledge, but also recognizing the facts. Overall FC market is still growing. Some analysts estimate 9-10% Y/Y growth at the end of 2010 for total FC market in vendor reported revenue. And it is remarkable that FCoE is becoming a larger part of that market. Depending on reports, it would now be up to 10% of the total FC market (considering adapters, switches and controller ports). And the really important thing is that this is more than doubling its contribution to the total market vs one year ago.
Those are some facts. About knowledge, I know recognize that FC networks bring not only stability and performance required for Tier 1 applications and mission critical environments, but also offer management over the SAN that is unmatched by other SAN protocols. The problem for many customers was that deploying FC was prohibitively expensive: the need to deploy a completely separate network, (which usually had to be design-stamped by the storage vendor) and the lack of FC interfaces on low end (and even mid-range) arrays made it impossible for small and mid-sized organizations to even consider it.
This is where I see a new angle now. FCoE is going to make FC as ubiquitous as Ethernet, and almost as affordable.
Customers jumping on iSCSI would normally pick on using the same switches for LAN and iSCSI (perhaps dedicating ports for the latter, and certainly should be dedicating separate vlans, subnets, etc.). Server side? Just another pair of GE ports. Usually not adding much to the cost if you were considering quad NIC cards on top of whatever comes as LOM.
FC would have added a lot. A pair of 4Gbps ports would represent an HBA or about $2K, and then you had to add separate dedicated Fabric switches at the access layer.
Well with FCoE, you put FC at the same level as deploying iSCSI for many people. First of all, FCoE now comes standard in NICs (CNAs) from many vendors, including Emulex, Qlogic, Intel, Cisco or Broadcom. Intel's approach of OpenFCoE adds no cost to deploying just 10GE, and renders the price per port to about $400 range. Such CNAs are already certified by most relevant storage vendors including EMC and NetApp. Vendors which, btw, are also adding FCoE support to their mid-range and high end arrays.
Now the network piece. 10GE ToR ports are about $500-600 a port. Cisco solutions there, thanks to FEX approach, allow to deploy a very cost effective 10GE access network, where all ports can support FCoE. So, in essence, you are at the same position as with iSCSI now: can use affordable ports at the server, have support for most common OS, can use the same network infrastructure, and can pick from multiple storage vendors. Nice thing on top? You can get management tools like CIsco's DCNM to give you visibility of LAN and SAN traffic (which you do not get with iSCSI).
Bottom line, I think FCoE is making FC easier and cheaper to deploy, so I expect more and more customers to consider using this technology, on top of those already using FC to adopt FCoE as well (using less power, lower capes, simplified operations, etc.).
Of course there is space for NAS deployments as well, and iSCSI will continue to grow, but I think FCoE makes FC much more competitive as a technology than it ever was for deploying a SAN.
Why am I changing my mind? Well, a bit more knowledge, but also recognizing the facts. Overall FC market is still growing. Some analysts estimate 9-10% Y/Y growth at the end of 2010 for total FC market in vendor reported revenue. And it is remarkable that FCoE is becoming a larger part of that market. Depending on reports, it would now be up to 10% of the total FC market (considering adapters, switches and controller ports). And the really important thing is that this is more than doubling its contribution to the total market vs one year ago.
Those are some facts. About knowledge, I know recognize that FC networks bring not only stability and performance required for Tier 1 applications and mission critical environments, but also offer management over the SAN that is unmatched by other SAN protocols. The problem for many customers was that deploying FC was prohibitively expensive: the need to deploy a completely separate network, (which usually had to be design-stamped by the storage vendor) and the lack of FC interfaces on low end (and even mid-range) arrays made it impossible for small and mid-sized organizations to even consider it.
This is where I see a new angle now. FCoE is going to make FC as ubiquitous as Ethernet, and almost as affordable.
Customers jumping on iSCSI would normally pick on using the same switches for LAN and iSCSI (perhaps dedicating ports for the latter, and certainly should be dedicating separate vlans, subnets, etc.). Server side? Just another pair of GE ports. Usually not adding much to the cost if you were considering quad NIC cards on top of whatever comes as LOM.
FC would have added a lot. A pair of 4Gbps ports would represent an HBA or about $2K, and then you had to add separate dedicated Fabric switches at the access layer.
Well with FCoE, you put FC at the same level as deploying iSCSI for many people. First of all, FCoE now comes standard in NICs (CNAs) from many vendors, including Emulex, Qlogic, Intel, Cisco or Broadcom. Intel's approach of OpenFCoE adds no cost to deploying just 10GE, and renders the price per port to about $400 range. Such CNAs are already certified by most relevant storage vendors including EMC and NetApp. Vendors which, btw, are also adding FCoE support to their mid-range and high end arrays.
Now the network piece. 10GE ToR ports are about $500-600 a port. Cisco solutions there, thanks to FEX approach, allow to deploy a very cost effective 10GE access network, where all ports can support FCoE. So, in essence, you are at the same position as with iSCSI now: can use affordable ports at the server, have support for most common OS, can use the same network infrastructure, and can pick from multiple storage vendors. Nice thing on top? You can get management tools like CIsco's DCNM to give you visibility of LAN and SAN traffic (which you do not get with iSCSI).
Bottom line, I think FCoE is making FC easier and cheaper to deploy, so I expect more and more customers to consider using this technology, on top of those already using FC to adopt FCoE as well (using less power, lower capes, simplified operations, etc.).
Of course there is space for NAS deployments as well, and iSCSI will continue to grow, but I think FCoE makes FC much more competitive as a technology than it ever was for deploying a SAN.
Subscribe to:
Posts (Atom)