ACI Example - Fast deployment of infrastructure

Many people focus on ACI from the point of view of network virtualization. ACI indeed delivers a powerful network virtualization solution, through an integrated VXLAN overlay which can be use in a programmatic way. These virtualization capabilities are built with a policy model concept that links well with application definition. This is what most people stay with.

But the ACI policy model extends beyond using it for providing application connectivity. The APIC also provides many functions that are useful for a network/fabric administrator, in terms of topology management, switch on boarding, policy-based configurations and so on.

This blog is a simple example to illustrate what it takes to bring up a new rack for instance. In this example, I will add a new ToR to an existing fabric and will leverage the APIC policy model to provide connectivity for an ESXi host.

We start with a working fabric of only one leaf, as below:

Now I want to add another physical switch to that fabric. We can imagine that I have just racked a new ToR (leaf), and plugged the uplink ports eth1/49 and eth1/50 using 40GE to the spines. I can already see APIC has seen the switch:

The switch still has no management address, because we have not registered it to the fabric. But the network admin does not need to console to the switch, think of its management address, or use any configuration management tools to provision it. All we have to do now is register the switch:











We give it an ID, a name, and that is it, the switch is now added to the Pod1 we are working with. All that is necessary for it to work with the fabric is taken care off. At the switch console we see the name has been changed already as well:

Because we already had a leaf working with connected servers, we have previously created a switch profile for it, with associated interface selector profiles, all we need to do is add the new switch to the switch selector, for the right interface configuration to become available. In our case, this is setting a number of ports to GE, with CDP enable, etc. and other ports to 10GE, CDP and LLDP, etc.

That is it. In this lab I have created two interface policies, with single links, one for GE connected ESXi hosts and another for 10GE connected ESXi hosts (CDP, MTU and other settings are part of the profile). The same model can be applied if using vPCs of course. The right ports have already been configured for GE and with proper VLANs from a pre-defined pool. As soon as we plug the ESXi hosts and apply their configuration they already show the leaf on CDP:

The ESXi hosts were already configured as well, as they were working before connected to a Nexus 9K standalone lab. ESXi infrastructure traffic is mapped to application profiles, and traffic or a particular kind to its own EPG (vMotion, VSAN, iSCSI, NFS ...). As soon as we add the EPG bindings, we see for instance all iSCSI hosts (herewith statically mapped):

And we can then also benefit from immediate visibility into each of the vSphere traffic types, without adding any other tools (which can of course also be used!):

This is but a really basic example of how a fully programmable fabric is useful, beyond providing network virtualisation …

Thanks to my good friend @alonso_Inigo for helping me ramp up on so many things! :)

Let's talk about a 64 Tbps Firewall

Much has been written about SDN, ACI and NSX. The debates are heated, often times perceived as if enemies are fighting one another. In the end, we are just writing about technology, and we should all remember that. Of course we all defend what we think is best, and of course we all try to bring forward considerations that are better suited to whatever position we are defending. This is human. 

I certainly do not want to contribute to the perception that it's all about bashing one solution or another. Certainly, I do not want to bash any solution, and NSX in particular. But I believe we all need to contribute somehow to have a more moderate debate, and I just dislike exaggeration. With all my respect for @bradhedlund, tweets like the one below fall into exaggeration, in my humble opinion:

@cobedien @dkalintsev @thomas0002 @jonisick Go for it. All east west traffic secured by a firewall for 96K VMs. Or is that asking too much?
— Brad Hedlund (@bradhedlund) December 24, 2014

@cobedien @dkalintsev @thomas0002 @jonisick 96K vm at 30:1 is 3200 hosts w/ 20G. You add 64TB east west firewall. I'll add the controllers
— Brad Hedlund (@bradhedlund) December 24, 2014

To put some context, this tweet comes out of a discussion over a blog post by @cobendien and @chadh0517 [which you can read here]. The post is about how open is Cisco ACI and what is the Total Cost of Ownership (TCO). A comparison is provided in terms of TCO, presenting a Cisco ACI design and another one built with NSX and Arista commodity switches. Brad's argument is that the TCO analysis is flawed, because NSX provides E/W firewall which should be added to the ACI design. Brad is talking about using NSX DFW to provide for an environment of up to 96,000 virtual machines (which is the size of the design that is presented in the original post). 

The logic here goes that to compare ACI and NSX costs, you need to add an enormous amount of hardware to provide for firewall and load balancing capabilities that NSX provides. NSX Load Balancing however appears to be nothing more than HA Proxy with a front end for provisioning. HA Proxy is, as we all know, free. A device package for HA Proxy is easy to do in order to integrate it with ACI. Similarly, NSX Edge firewall capabilities are very similar to what you can get on a Linux VM using IP Tables. The key argument therefore lies on the Distributed Firewall feature, which provides for East-West stateful packet filtering on a per vNIC level.

I am certain that many customers see value in NSX, and specifically on the Distributed Firewall feature. However, based on my understanding of NSX, it is unfair (and not true) to pretend that 3,200 hosts running NSX are equal to a 64 TB firewall. This isn't true in essence, nor in math. In my opinion, that is an exaggeration and I believe it should be avoided, and I have seen similar ones many times.

I will try to provide details about why I think this way, based of course on my humble knowledge of NSX as it is today. I do hope I'll be corrected if and where I am wrong. Comments are opened, and welcomed!

The need for East/West Filtering

I do not contest that some customers have a clear interest, and in some cases a real need, to have per VM East-West traffic filtering. I do contest that per VM East-West traffic filtering is a requirement on every case, and/or that the best implementation in order to accomplish that is by running it at the hypervisor kernel level. 

In first place, it is fairly obvious that not everything runs in a VM. Many (many) apps still run in bare metal, therefore a security solution should consider East-West filtering to be comprehensive for both environments. Second, implementing advanced traffic filtering at the hypervisor level complicates having a seamless solution for policy across multiple hypervisors, because the data plane would need to be replicated on every kernel, something which isn't evident to accomplish (even if most vendors today have an open API for accessing the kernel for network functions). Third, the "cost" of filtering at every virtual ingress point may be higher than it appears, and it isn't for free in any case. Fourth, and a direct consequence of the previous point, implementing advanced filtering isn't available today at very high speed on x86 architectures. 

This leads me to the tweet line below, part of the same tweet exchange from above:

@savgoust @cobedien @dkalintsev @thomas0002 @jonisick that's your opinion. Point is, NSX provides a stateful firewall for E/W, ACI doesn't
— Brad Hedlund (@bradhedlund) December 26, 2014

What a Firewall is, and what a Firewall isn't 

I am not a security expert, but I believe that the definition of a Firewall isn't a matter of "opinion". It is a matter of the functionality provided. That said, not every place in the network needs advanced packet analysis and filtering, so the functionality required from a firewall isn't always the same, and that's the real story. 

One of the fundamental arguments for proposing a DFW like that provided by NSX (i.e. a stateful packet filtering engine at the vNIC level) is that it prevents threats from propagating East-West. Explained shortly: if you have a vulnerability exploited on a VM on a particular application tier, the exploited VM cannot be (easily) used to launch attacks against others in the same tier and/or other tiers of VMs.

But I think that this argument contains a bit of a marketing exaggeration. 

Imagine a typical three tier application. At the user facing side, the application tier presenting a web front end with a number of VMs running Apache, nginx, IIS … your pick. All those VMs listen on tcp/80, any of those will eventually have vulnerabilities. If one of those VMs is exploited through a vulnerability of the web server accessible on that port, the chances that ALL other VMs on the same tier are equally exploited are very high. This is the reason why your front end perimeter firewall should have capabilities to inspect HTTP traffic in order to protect the web server. For this you need a real firewall that can do HTTP inspection, HTML analysis, etc. The fact that once one Web VM exploited, your DFW impedes each Web VM from talking to other Web VM does very little in this sense, because the attack vector was on the legitimate port. Then, even if those exploited web VMs can only communicate on authorised protocol and ports with other application tiers (something which is also enforced by the ACI fabric since each tier will typically be on different EPG - or EPGs and can communicate only on a contract permitted basis). But we all know that again this protection is also not incredibly strong because, again, it is only filtering on protocols and ports. The real exploit may (will) happen through the legit port and protocol. This is the reason why modern firewalls (aka NGFW) these days inspect traffic without necessarily concerning themselves about port and protocol. To continue our example, SQL traffic may be allowed between the App and DB tiers, but you need the capacity of inspecting inside the SQL conversations to detect malicious attempts to exploit a bug on a particular SQL implementation … something which a NGFW does (but the ACI fabric filtering or NSX FW don't do). 

This does not mean that the ACI or NSX DFW filtering aren't important and/or useful. I am not writing this as a means to discredit the NSX DFW feature, which may be interesting in terms of compliance or for other reasons. Multi-tenancy may be a reason for using such filtering, other than security. ACI offers very similar functionality, being a natural stateless networking fabric it may appeal to security ops as well. 

This is truly not about bashing a technology, but I do want to put the reader into a mood where they think about these tools as parts of a solution: not as the holy grail (which they are not). I hope the reader will ask her/himself whether a feature is really valuable, or whether it simply is presented in such a way because it is … well … the only thing that a particular vendor has to offer. 

In reality, if security is really of top concern within the DC, you will likely still need a real firewall (from a real firewall vendor), and what matters in that case is how service redirection can happen in an effective and consistent manner (for physical, and virtual). 

3200 Hosts and a 64Tbps Firewall

There is a bigger point of exaggeration on the tweet line above. It's not only the hype about DFW capabilities and how much security can it really enforce. The other point that I think falls into both hype and exaggeration on the tweet exchange above is performance and scale. I'd like to spend some time on that to put things into perspective. 

I want to clarify that this isn't about "software can't scale, and hardware is better" or vice versa. It is about the fact that oversimplifying things isn't helpful for anybody, and that distributed performance is sometimes presented almost in mythical ways.

Let us analyse an environment with up to 96,000 VMs, on 3200 hosts each dual homed with 20Gbps. Such an infrastructure  cannot be considered in terms of a SINGLE FEATURE of the underlying software or hardware. There are many more dimensions to consider. And then also even if considering one single feature, the devil is in the details. 

Let us consider the following:

(1) Such an infrastructure needs not only E/W filtering, but N/S as well. Perimeter firewalling is what will first and foremost protect that front of end of yours. You will typically want to have a NGFW for this. Depending on the design, the NGFW resources may be virtualised and therefore be shared for both functions (perimeter and east/west). In the case of using NSX, the resources allocated for the DFW are only available for basic E/W filtering. No sharing in that case. This point must be considered also in the TCO.

(2) Following from above, the DFW feature isn't free to run. The tweet line above was presented as if NSX was adding that feature on top of a design at no additional cost. But the real cost of the DFW feature, or any other feature that runs on a hypervisor, cannot be evaluated only in terms of the software licensing, but also the general purpose cpu cores and memory that are consumed by this feature. Cores and memory that aren't available for other applications. Nothing is for free.

(3) To infer that because you have 3,200 hosts each with 20Gbps you have a 64 Tbps firewall is marketing math at its best. First, because to this date, we have not seen any independent validation (or vendor validation for that matter) of the real performance of the DFW (or any other NSX feature). This is a clear contrast with VMware vSphere features, or with VMware VSAN where there's a plethora of vendor and third party benchmark testing done publicly available. Second, because with current NSX scalability limits (as per data sheet), those 3,200 hosts cannot all be part of a single NSX domain, therefore need to be split into smaller Distributed Firewalls with no policy synchronisation. More details on this below.

(4) Understanding that real security (beyond protocol and port filtering) will require a real NGFW filtering at least between critical Apps and/or App tiers, the NSX solution must also be complemented with an offering from a company like Palo Alto (and to this date, only that company as far as I know). This has an impact in performance, in cost, and in overall resources (because each PA VM-Series also requires vCPUs and vRAM which are incremental to those consumed by the DFW feature itself).

(5) Traffic must be routed in and out of that virtual environment. Those 96K VMs will have a need to communicate with users if nothing else, and with their storage in most cases too. In fact, if the server has two 10GE interfaces it is likely that a fair amount of that bandwidth will be dedicated to storage that won't be protected by the DFW feature in any case. This also means that you do not need 20 Gbps of firewall throughput per server. The amount of North South traffic will largely depend on what the infrastructure is being used for: what is running on those 96K VMs. Imagine that you need to onboard lots of data for a virtual Hadoop Cluster that you decide to run there, you may have significant peaks of traffic inbound. Assume North/South is 10% of total bandwidth, you then need in excess of 6Tbps of routed traffic in/out of the overlay … I will come back to this point later. 

To expand a bit on point (3) above, on the per-host performance, it is well known however that ESXi in general does not do 10GE at line rate for small packet sizes for instance (http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-VXLAN-Perf.pdf). This continues getting better and better of course, and I'd expect vSphere 6.0 will also improve in this area. The NSX performance figures that I have seen presented at VMworld SFO 2014 [Session NET1883] displayed results that for small packet sizes, DFW was maxing out at around 14Gbps on a test with 32 TCP flows on one direction, whilst for large packet sizes it could be up to 18Gbps per host. Since this test was done with unidirectional traffic only and with a (very) small number of flows (and not knowing how many cores where dedicated to achieving this performance because such information wasn't shared), we cannot infer what the real performance is for IMIX or EMIX, but it is clear that it will not be 20Gbps per server. 

The performance considerations of a distributed environment can be looked at in many ways. Distributed performance may scale better in certain cases, and worse in others. The performance of each point of distribution needs to be considered. In this case, there are per-host limitations which may or may not be acceptable. Again, leaving behind whether current NSX DFW performance is 14 or 18 Gbps per host or whatever (in any case, it will get better with time for sure), as you add NGFW capabilities today this is lowered by an order of magnitude (even if that will also get better over time). The PA VM-Series maxes at 1 Gbps per VM as per their data sheet. 

Oh, but you only send to the Palo Alto VM-Series the traffic that requires deep inspection … by filtering on port and protocol with the DFW? … This is a possibility, but then you are saying that you won't be sending to deep packet inspection such port and protocol because they are safe? … That argument doesn't hold. And particularly the Palo Alto folks know that, since they make a selling point around … well … being a firewall that cannot be fooled by port and protocol. Anyways, the level of security requirements will be customer dependent of course, not all environments are the same. 

Back to performance, arguably you can also say that any particular host won't be bursting at maximum performance all the time. So it does not matter if you can do only 1-2 Gbps, because maybe that is all that your application requires. But it is clear that when an application is busy on a particular host, likely traffic is high on that particular host as well, and you do need the firewall performance for that particular host at that very moment. 

In other words, a 3,200 host NSX environment isn't really a 64Tbps DFW at all. If anything, it'd be 10 or 12 smaller ones since you cannot have 96K VM under a single NSX Manager (see below where the 10-12 comes from). 

Of course, the same argument about statistical traffic requirements (i.e. all host won't require maximum performance at once) can be made for a semi-distributed NGFW design option. By semi-distributed I mean a scale out model built with appliances and dynamic service chaining. This isn't an all or nothing. You can distribute firewall functions to every host, but the alternative isn't to have a single large, monolithic firewall. You can have a scale-out cluster of dedicated hardware appliances too. 

For the sake of argument, consider using Palo Alto physical appliances on a scale out model, vs. per host Palo Alto VM-Series integrated with NSX. A vSphere cluster of 32 hosts could max out at 1 Gbps of firewall throughput per host (as per the Palo Alto VM-Series data sheet here). You could say that the vSphere NSX cluster is a 32 Gbps NGFW for E/W, assuming you use it to filter all traffic (which is probably not required in many environments). To accomplish this you need the VM-Series licenses, the NSX licenses, and dedicated physical hosts for running NSX controllers and manager. Also, you are dedicating a total of 256 cores on that vSphere cluster to Palo Alto alone (or the equivalent of 5-12 servers depending on CPU configuration on a dual socket server). This is not counting the cores required for DFW, which are harder to estimate. On such solution, no single VM on any particular host can exceed the maximum throughput of the VM-Series appliance (1 Gbps today). That is: traffic between any two VMs crossing the VM-Series can never reach 10Gbps. 

Consider now a design with Palo Alto physical appliances instead. If you would use two PA-5060s with service chaining, you have 40 Gbps if NGFW capacity for East/West. You also get better performance for any single VM to VM communication which can now fully use all host available bandwidth. More over, that NGFW serves not only ESXi hosts … but any other stuff you have connected to the fabric! … 

Now run the numbers for TCO of those two environments ... the numbers do not lie. I do not know the pricing of Palo Alto appliances or VM-Series, so I will let the reader contact their favourite reseller. Of course I can make the same case for ASA and compare (so I know the results :-) ) but I do not want this post to be a Cisco selling speech.

However by using NSX + Palo Alto, you get to use vCenter attributes to configure policy! and you can dynamically change the security settings for VMs as required. Well, actually, that would be possible without NSX as well from a policy point of view because nothing technically prevents Panorama from directly interfacing with vCenter and the latter from communicating the mapping of VM attributes with IP/Mac, etc. The solution has been productised in a way that Panorama must talk to NSX Manager, but I see no (technical) reason why it couldn't be done in other ways since it is vCenter who holds the key information.


Now let's come back to point (5) from above. To do that, we need to consider the design of the 96K VM infrastructure with greater detail. The design possibilities for such environment are so varied that I won't take a shot at doing this here. But I do want to point out some things for consideration, for instance, let's review a possible fabric design first, with a variation of what was proposed above:

- 3,200 hosts could fit in as little as 80 racks, using 1 RU dual socket servers (provided cooling can be done in the facilities)
- using redundant ToRs per rack, this translates into 160 ToRs. We would consider four spines.
- if using ACI, the fabric would have three APIC controllers in a cluster.

Now let's consider the virtual infrastructure:

(1) vCenter 5.5 maxes out 10,000 active VMs [http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf]. I do not know if customers would feel comfortable operating the environment at such high utilisation numbers, considering that vCenter is also a single point of failure. Also, if operating at maximum possible scale, the vCenter DB also must be considered carefully. For the sake of an exercise, let us assume you add 20% safety so you operate at 8,000 VMs per vCenter, you would need 12 vCenter. To make an easier math, let's assume 9,600 VM per vCenter and therefore you need  a total of 10 vCenter instances.

(2) The point above is important because NSX as of v6.1 has a mapping of 1:1 between the NSX Manager, controller cluster and vCenter. This means that you also need 10 different controller clusters and 10 different NSX Managers. Notice that there is no federation of controllers or managers today. This means that from a DFW standpoint (and from every other standpoint in fact), you have to replicate your policies 10 times (or more if you choose to use less VMs per vCenter). This is why I alluded above that if we are to believe Brad's marketing claim of 3,200 hosts being a 64 TB firewall, it should as a minimum be split into 10 chunks (and likely into more chunks in fact for practical purposes). 

(3) We are considering here that each controller cluster and NSX Manager can handle the load for up to 9,600 VMs over 320 physical hosts. We have not talked of how many Logical Switches, Distributed Logical Routers, etc. may be required on the environment, or the impact that such considerations may have on overall scalability of NSX. Again, an area where we all live in the grey mist of ignorance, since no public guidelines are available. But it is clear that if we choose 10 vCenters, the controller clusters will be operating near its maximum advertised capabilities (which is 10,000 VM) in any case. With this in mind if you design for maximum availability and minimal performance compromises, you need:

- 10 servers for vCenter
- 30 servers for NSX Controllers
- 10 servers for NSX Managers

The above must of course pay vSphere licenses as well, and would take up almost two racks worth of space, network connectivity and power. Arguably, you can say that you can run two or even three controllers per server, each belonging to a different cluster. This has an impact in the system availabily, because in such a case a single host failure (or maintenance) would impact two or three clusters, as opposed to only one (so you duplicate or triplicate - or more - the size of your failure domain). If you choose for up to three controllers per host, you'd use 10 servers for controllers, instead of 30, at the expense of a larger failure domain. In any case I doubt anybody would design to operate the environment at 96% of advertised capacity for vCenter nor NSX, so the actual figure will likely be larger. (although vSphere 6.0 is coming out soon and may raise some of these figures up).

It is worth remarking that, effectively, you have 10 isolated islands of connectivity on that infrastructure. How does a VM on one island talk to a VM on another island? … Through an NSX Gateway. So the East/West across islands requires Gateway function resources. Another point to consider in the 64 Tbps DFW marketecture which was ignored (and another cost).

Now comes the perimeter. The North South. Again, let's say you design for 10% of the deployed fabric capacity. No, let's say it's 5%. You want 3 Tbps of North/South capacity available. Let's forget the perimeter firewall because you could argue that whether you use ACI or NSX or both, you need it anyways and it could be made off of the same clusters of ASA or Palo's or whatever. But you have to route those subnets in which your 96K VMs live. For NSX, that means using NSX Edge, peering with the DLR. How many DLRs you need? Depends: on how many tenants you have, on how many subnets you need to isolate and route, etc. Let's forget those too. At VMworld session NET1883 they presented a test case where NSX Edge could route at about 7 Gbps. Again a test with minimal data shared: no RFC level testing, no IMIX consideration, no word on tolerated loss rates, or latency, … 

Let's say each NSX Edge routes 10 Gbps though, and you can load balance perfectly across several instances as required.  To route 3 Tbps in and out of that overlay you need 300 NSX Edge VMs. If they use any stateful services (i.e. NAT) and if you want redundancy? … then you need to add 300 more … in standby. 

Now, let's put two NSX Edge per physical server (each with a dedicated 10GE NIC), you are adding 300 servers to the mix. That is almost 8 racks worth of gear that pays full vSphere (and NSX) licenses that can do one thing alone: run NSX Edge. Let's forget the complexity of operating 300 mini-routers, each of them independent of one another. 

In the drawing above I show how NSX Edge is also required if East/West traffic flows between NSX Domains, but notice that I have not considered any requirement for E/W between the different NSX domains on the calculations above about how many NSX Edge VM you need. But that would add more need for NSX Edge VMs. We have also not considered the DLR VMs, that would also require dedicated servers.

Now imagine an upgrade! ...

And what about ACI?

With ACI the fabric can be shared with more than one vCenter. Also, gateway functions are not required to connect VMs with other stuff, whether it is users through the core or WAN, or bare metal applications. As for East/West, it really depends on every environment. Where the intention is to simply have filtering to segregate tenants or apps, it is possible that ACI stateless filtering model intrinsic to the Application Profile definition is sufficient. In other cases the filtering is required at the vSwitch level, and in that case the Application Virtual Switch (AVS) can be leveraged in vSphere environments. Or one could imagine that NSX is also leveraged in that sense, with ACI for service chaining. We have seen that using physical firewall appliances in a scale-out model with dynamic service chaining can be more cost effective and render better performance than the virtual model. The graph below illustrates a possible alternative, built using vCenter with ACI but without NSX.

Because you do not need the NSX Edge Gateway functions and you need less servers for management and control, you will see savings that can probably pay for the NGFW functions. This depends on each case and on the scale and requirements of course. But you can see that for this particular scenario, you probably require between 200-300 less servers, with accompanying licenses, space and power costs.

In the end, it's not all black and white

To wrap it up, this isn't about saying that NSX is a bad product. This isn't about saying that the NSX DFW feature is good or bad. It is about expanding the conversation and considerations beyond hype, and marketing. Fact is, many customers that are looking at both ACI or NSX have scalability requirements well below the design discussed here. Ultimately, customers may see value in both solutions as well, or in none of them! For instance using NSX DFW as a form of advanced PVLANs, but use ACI fabric service chaining for redirecting to NGFW running in scale out clusters. 

The important thing IMHO is that customers can make an educated decision. And while all vendors legitimately try to steer the conversations to their advantage, we should all try to avoid falling into blatant exaggerations. Particularly, when it touches security aspects. 

Paris OpenStack Summit Recap

While I have been involved with OpenStack in various ways for the last couple of years, the OpenStack Summit in Paris has been my first and it has been a great experience. Throughout my career I have attended many vendor conferences like Cisco Live, VMworld and others, industry events (CeBIT, Supercom, …) and I have also attended FOSDEM in the past. The OpenStack was a mix between an industry event, a vendor conference and FOSDEM.

The organisation was great, similar to what you get on large vendor events. The audience was certainly (a lot) more technical and with a lot more presence of developers than most vendor conferences (I have not attended developer conferences …), but clearly there's also a marketing spirit all around the Summit that is very different from the FOSDEM experience. But that said, the event shows the interest on open collaboration.

Considering this event is happening every six months and considering it is an open source focused conference, the volume and level of attendance is impressive. I believe there's been more than 4,600 attendees from 60 different countries.

The keynotes were interesting and there's been many good sessions. I believe most were recorded and the videos can be found on line here.

Learning Experience

There are a lot of sessions, really a lot. This means you have to make  hard choices in terms of which ones you attend, because no sessions are repeated and there are many overlapping topics. I believe the session abstracts could be a bit better some times as well ... As explained above, most were recorded so you can also check those you missed, but of course without the opportunity to ask questions, interact, etc.

So what about OpenStack? …  I have been a fan and an OpenStack "aficionado" for a while. Already a year ago it was clear that the maturity level was reaching a point that indicated that OpenStack becoming mainstream technology was not a dream.

At this Summit, we have seen about some large scale deployments by some (very) large Enterprises including BMW, BBVA, Tapjoy, CERN, SAP, Expedia and others. Impressive. We are talking of different use cases, different industries, different scale levels, but all successfully running OpenStack.

CERN is currently running 70,000 cores on OpenStack (I believe on RDO if I got it right), with plans to double that in the near future. The keynote presentation by Tim Bell was great. I strongly recommend watching it, not just from an OpenStack point of view but to learn more about CERN and what they do.

By all means, OpenStack is a real challenger to legacy virtualisation and cloud management proprietary vendor solutions. In fact, perhaps not a challenger, but many challengers … because there are various mature options for "consuming" OpenStack already. Whether it is as a product (Piston), a distro (RDO, HP ...), or even as a service (MetaCloud).

Challenges?

Sure. Many. This is one of the areas where the OpenStack Summit clearly differs from vendor events and trade shows. Folk here shares as much of the problems as they share their success.

I think it is fair to say that unless you know very well what you are doing and/or you have a small army of Openstack savvy engineers, you want to have a vendor backing you up (read list above). But, how is this different from implementing a cloud solution using proprietary software? ... It's not like you can download the vendor's code and start building your cloud right?

OpenStack challenges seem to be mostly coming from the very nature of the framework: a loose collection of components that work together but are built to be independent. So you need to pay attention to tuning things like your backend databases, message queue systems, understand how you provide HA for various components, etc.

Perhaps because of my background, or perhaps because of reality, but networking appears to be the most challenging part. It is the one area where the "native" OpenStack solutions are not really up to speed and you need to rely on some vendor option.

There's still a lot of discussion about whether overlays and/or full SDN solutions (where the physical fabric is orchestrated to provide virtualization services) is the best choice. There was a great panel about it.

One of the interesting points that came out of that panel is that L2 overlays and virtualization solutions are in fact looking to solve problems that come out of IPv4 limitations. However we should be thinking about IPv6 as the long term solution. I think this is definitely true and worth exploring. Much of what has been done with overlays is completely unnecessary if we consider IPv6.

Another thing comes clear is that much work is still required on OVS, both in terms of performance and stability.

Conclusions?

After interacting with multiple people, including developers, customers, vendors … there is a clear interest in moving away from expensive proprietary software stacks for implementing private and public clouds.

Many speakers (including some on several of the keynotes) talked about OpenStack providing a way for achieving freedom of choice, not just of hardware, but most importantly of software components as well. During the keynote on Monday, the Dr. Stefan Lenz from BMW in particular was very explicit to call out the issues of the past, when they were developing around a proprietary automation software - effectively locking themselves on to it - to later experience increasing licenses fees that you couldn't turn away from (because you were locked in).

Contrary to the mantra around … software lock-in is usually the most expensive one!

Final thought ...

The messaging on the keynotes was, understandably, highlighting the growing relevance of software in the economy and  in the world in general. Of course, the open source development model is presented in this context as the natural way of doing things in this new world. Pretty much, the message is that OpenSource will rule the world. A slide presented that in the near future, most solutions will likely use an 80/20 rule with 80% of code being open source.

So, considering that companies still need to create value, what is the way to differentiate on an open source dominated world? It's that 20% of value-added code enough? Either that, ... or the hardware where Open Source code runs adds value as well … (or both).

About a VMware OpenSack vs. RedHat OpenStack comparison

I watched one of the sessions from last VMworld about the VMware Integrated OpenStack solution. I found it very interesting, and there were a couple of slides that really caught my attention. The messaging goes that OpenStack sits atop of a compute, storage and network set of resources, but does not impose what those resources are made off. This is very true, and one of the nice things of OpenStack.

In the case of VMware, they presented how they can fill all of those resources on an OpenStack solution: compute with vSphere, storage with VSAN and network with NSX. Further to that they said vCAC and vCOPS are complementary and you could use them as well (making you wonder why on earth would you be running OpenStack if you have all of those products … but that is another story). Following that, there were a couple of slides presenting a comparison done between an OpenStack solution built by using VMware products, with another one built using Red Hat. The slide quoted a document by Principled Technologies as supporting material. That document can be found here:

http://blogs.vmware.com/virtualreality/2014/06/vmware-bests-red-hat-openstack-performance-cost-study-2.html
http://www.vmware.com/files/pdf/techpaper/OpenStack_VSAN_0614.pdf

I find very interesting that VMware singled out an OpenStack vendor and chose Red Hat. It is worth reading the document and reflecting about it.

In principle the goal of the testing was to compare VMware vs. RedHat for running an OpenStack cloud giving considerations to the hypervisor and storage parts of the solution (NSX and Neutron were left out of the consideration).

The title of the document is "Cost and Performance Comparison for OpenStack Compute and Storage Infrastructure". The testing is done by using common tools to measure storage performance and by running a Cassandra DB on VMs provisioned via OpenStack and measuring its performance as well using standard performance testing tools.

The conclusions are very neatly articulated in the document introduction and can be summarised as: VMware solution is more performing (159% more IOPS) and less expensive (26% lower cost over three years).

The first point isn't shocking (although I was surprised by the incredible performance advantage, given that I have seen studies showing KVM outperforming ESXi for other DB workloads). But the second point was certainly a surprise.

But as with all studies, what matters is how they reach to conclusions and what are the items that lead to the differences. Let's look at them.

The performance difference can be explained very easily by noticing a few things:

• the performance was measured only for the storage part of the solution. Not for memory-bound workloads, not for cpu-bound workloads, not for network I/O bound workloads.
• the tests ran to measure storage performance were biased towards read (70/30 read to write in all cases). This may be realistic or not, depending on workload, but probably reasonable. 
• the VMware solution (using VSAN) leverages SDD for caching, the RedHat solution (using RedHat Storage Server) does not.

There you go, the difference in performance is primarily justified because of the use of SSD for caching inside VSAN. If you were to use an SSD-based storage solution the performance difference would completely different, probably negligible and not necessarily to VMware's advantage.

In defence of those conducting the testing, RedHat currently does not offer a scale-out storage solution which can use SSD for caching only. You can use GlusterFS with SSDs, but it will be very expensive.

However if VSAN would be removed from the equation and both solutions compared using common storage from say NetApp (using SSD for caching) you probably get equivalent performance to the VSAN scenario. Arguably that would be a more open solution, because unlike VSAN, NetApp storage wouldn't be limited to working with vSphere only.

The price difference is coming from:

• using dedicated servers for running RedHat Enterprise Storage: effectively more than doubles the cost of hardware.
• the cost of the Red Hat Storage Server (I am unfamiliar with how this is licensed, so I can't comment and I take it as accurate of course).
• the cost of using a full blown RHEL for running KVM.

Before going forward, I would like to quote something from the test document, and refer the reader to consider the title of the test document itself:

"While Red Hat does provide Red Hat Enterprise Linux OpenStack Platform, we left OpenStack support out of these calculations as stated above because each OpenStack environment and support engagement is so variable"

This test was commissioned to compare OpenStack solutions, but OpenStack solution pricing was not considered … Confusing, isn't it?

In the test, they chose to use RHEL Server to run KVM. Given they are using RHEL strictly for running KVM, they should have chosen for RHEV instead, which is also supported on the Dell PowerEdge servers they had at hand. This matters because it is more lightweight, optimised for running KVM with greater VM density and … less expensive:

- Red Hat Enterprise Virtualization, Premium (24x7x365 support):
4 (socket pairs) x $1,499 = $5,996.
source: https://www.redhat.com/en/files/resources/en-rhev-vs-vmware-vsphere-competitive-pricing-11717847.pdf

This already reduces the cost difference a bit … but actually, to make this an apples to apples comparison in the context of running OpenStack, they could have included Ent+ for vSphere, because on RHEV you can actually use Neutron to implement distributed switches and/or to leverage plugins from SDN vendors, and you can't do that on vSphere ENT (which uses standard vSwitch only, not very cloud friendly I believe). Therefore, the pricing for VMware software should also increase in consequence.

Just adding those two licensing considerations, which lower the price of the RedHat solution and increases the VMware one by quite a bit, the price of the two solutions would be almost equivalent in practice. Then again, if you consider using a storage solution from a vendor like NetApp or EMC instead of the vendor's scale-out option, you can build a RedHat based solution with equivalent performance and lower cost.

It then becomes also an issue of considering a converged solution or a separate external shared storage. A VSAN approach would have a density advantage (uses less rack space), and perhaps would be easier to manage too. There also an important element to consider and is operational cost, retraining of staff, etc. An external storage based solution offers greater flexibility, because it can be shared for things other than vSphere. Also, given that storage needs grow faster than compute needs in most environments, an external storage may be cheaper to run in the long term, but this is dependent on each environment.

Net net, just like with all vendor-comissioned TCO studies, I recommend people to actually read the studies (as opposed to just retaining the conclusions) and reflect about them Then customise the study methodology for their own environments. Such studies are usually a source of valuable information and setting up a framework for a valid comparison, but you can never assume that they compare apples to apples.

Software Overlays: infinite scaling and pay as you grow cost model

Defendants of software overlays are often times upset when they are challenged on scalability compared to hardware based solutions. An example here: Scalability with NSX.

Honestly, I too think that the argument of software (running on x86) vs. ASICs isn't the most interesting one. However, I also think that to dismiss any argument by saying "you don't understand the architecture of software overlays" is a void reasoning. And yet it is what usually happens, and folk (like in the example above) resort to saying <my-overlay-of-preference>  is a distributed architecture and therefore, it scales.

The flaw in that reasoning is that you are considering ONE dimension of the problem: moving packets from A to B. But there are many other dimensions when it comes to any network system. There's building and maintaining network state, there's availability, there's managing latency and jitter, and many others.

I do not want to trash software overlay solutions that run on x86, by the way. I think the services they provide are better off delivered by the physical network which must exist anyway, but when it is not possible and you are in an environment with 100% virtualisation … software overlays are definitely to be considered. Even then, handling north-south traffic in and out of the overlay is a challenge that must not be overlooked. In the long run, an integrated solution will prevail in my opinion.

The key is "100%" virtualisation. Because when that is not possible, when there's going to be east-west traffic towards databases or other systems running bare metal (and many systems run and will run bare metal and/or not run on a hypervisor) overlays not only fall short, but also become increasingly expensive. Of course, when your business relies 80% on selling hypervisor licenses, your view of the world is somewhat different …

What software overlays don't really eliminate is upfront capital cost of building a network infrastructure. This is a fact.

They also do not fully provide a pay as you grow model. If you want to build an infrastructure with 100 physical hosts, you need at least 200 physical network ports (assuming redundant connections, not counting management, etc …). When you want to add another 100 physical hosts, you need another 200 physical network ports and to grow your network core/spine/whatever to accommodate for it. This is true whether you will run a software overlay using VXLAN or use plain VLANs or anything else (by the way, VLANs are still more than sufficient for many cases, and easily automated through any modern CMP, including OpenStack Neutron).

Adding or removing VMs to those 100 physical hosts is another story. If you choose to go for an overlay software model to provide connectivity on top of the physical network, and you choose to pay per VM instead of otherwise, … well that is a choice. Customers should do a TCO analysis and choose whatever they find most convenient, including support for multiple hypervisors, etc.

What you cannot do is to think that any vendor is providing you an infinite scale system (or near infinite scale system).

What you should not do either is to evaluate scalability across a single (simplified) dimension. No overlay system is fully distributed. Packet forwarding (a.k.a. data plane) may be distributed to the hypervisor, but control is centralised. Sure, vendors will say "control clusters are built on scale-out model" … but if that is the holy grail, ask yourself why you can't scale out as far as you want but instead you are limited to 5 servers in the cluster … maybe 7 … maybe … There must be some level of complexity when you can't just "throw out more servers and scale …".

Control and network state is more complex as you move up the stack. It is one thing for L2, another for L3, yet another when you add policy. There is no holy grail there … and if you believe you found it, you are wrong, you just haven't realised it yet.

There is only one real problem: scalability.

I urge any reader to think about that sentence, not just in the context of technology, but any other. Fighting world's poverty, for instance.

IP Networks & Operating Systems - Virtualization is a means to an end, not the goal

I have recently seen a slide where it was stated that decoupling networking hardware and software is a must for achieving network virtualisation. This would also enable hardware independence, and provide the missing tool for the holy grail: the software defined data center.

I suppose that the best thought leadership is all about making people think that the (potential) means to an end, are really the objective itself. For instance, when someone says "I need a coke", when in fact they are just thirsty, and  they may like a coke to alleviate their thirst. Similarly, while the goal is IT automation, a possible means to that end is a software defined data center, and implementing a software defined data center by using virtualization of everything-on-x86 is only an option for that in itself. In this sense, many are evangelising that you need virtualisation, and because you need virtualisation you need network virtualisation and therefore you need to be able to run virtual networks the way you run virtual servers. 

I don't argue against the benefits of server virtualisation, or the need for network virtualisation either for that matter, in certain environments. I just think it is interesting how many of the marketing messages are creating this perception that virtualisation is a goal in itself. Much in the same way that SDN messaging has been distorted in the last few years, where it no longer is about separating control and data plane and opening both of them, but rather about running both in software (even if tightly integrated) … But that is topic for another post.

Why Server Virtualization was a Necessity 

I believe server virtualisation solved a problem that had been created by poorly designed operating systems and applications, which could not fully leverage the compute capacity they had available. The x86 architecture was also not good for providing isolation to higher layers. In the end, you had a physical server running one OS with a App stack on top which was not capable of making use of the full compute capacity. Servers were under utilised for that reason.Therefore hypervisors solved a deficiency of operating systems and applications. Applications were, for the most, incapable of using multi-core capabilities and operating systems were unable to provide proper isolation between running applications. This is what the Hypervisor solved. And it was probably a good solution (maybe the best solution at the time), because clearly re-writing applications is much harder to do than instantiating many copies of the same app and load balancing them … However, had the OS provided the proper containment and isolation and the CPU provided performing support for that, a hypervisor would have been less required. Because in that case, even if you would not rewrite applications for better performance, you could still run multiple instances. In other words, if we would have had Zones on Linux 8 years ago, the IT world would have been perhaps somewhat different today. (although in fact, we had them … perhaps in the wrong hands though).
Anyways, it is clear that for instance today, running Apps on LXC is certainly more efficient than doing it in a hypervisor from a performance standpoint. It will be interesting to see how that evolves going forward. 

We may need network virtualisation, but we do not need a network hypervisor

Similarly, an IP Network does not natively accommodate for proper isolation and for multiple users with different connectivity and security requirements to share the network. IP networks are not natively multi-tenant, or having the ability to segregate traffic for various tenants or applications. They were conceived to be the opposite really.There are solutions like using MPLS VPN or plain VRFs: in a nutshell, you virtualise the network to provide such functions. You do that at the device level, and you can scale it at the network level (again, MPLS VPN being an example of that, although it only uses IP as control plane, it uses MPLS at the data plane). VPLS is another example, albeit for delivering ethernet-like services.

Arguably, MPLS VPNs and/or VPLS are not the right solution for providing network isolation and multi-tenancy in a high density data center environment. So there are alternatives to achieving this using various overlay technologies. Some are looking to do this with a so-called network hypervisor, essentially running every network function on x86 as an overlay.For those supporting this approach, anything that is "hardware" bound is wrong. Some people would say that VPLS, MPLS VPN, VRF, etc. are hardware solutions and what we need are software solutions. 

I believe this is not true.  A VRF on a network router or switch involves software, which will program the underneath hardware to implement different forwarding tables for a particular routing domain and set of interfaces. A virtual router running as a VM and connecting logical switches is pretty much the same thing, except that its forwarding table is going to be implemented by an x86 processor.I do not like this partial and simplistic vision of hardware vs. software solutions. There are only hardware+software solutions. The difference is whether you use hardware specialised for networking or hardware for general computing. The first is of course significantly more performing (by orders of magnitude), whilst the second provides greater flexibility. The other aspect is provisioning and configuration. Some would argue that if you run network virtualisation in software (again, meaning on x86 on top of a hypervisor) it is easier to configure/provision. But this is a matter of implementation only. 

Conceptually, there is no reason why provisioning network virtualisation on specialised hardware would be any harder than doing it on general compute hardware.

You will always need a physical network … make the best out of it 

Because you always need to have a physical network in a data center, it is also evident that if the network infrastructure provides the right isolation and multi-tenancy with a simplified provisioning and operation, it represents a more efficient way of achieving the goal of automating IT than duplicating an overlay on top of a physical infrastructure (much like LXC are more efficient than a hypervisor). This leads to the title of the post. 

The goal is not to do virtualisation. Virtualisation is not a goal. The goal is not to do things in software vs. hardware either. 


The goal is enable dynamic connectivity & policy for applications that run the business supported by an IT organisation. And to do so fast, and in an automated way, in order to reduce risk of human errors. Whether you do it on specialised sophisticated hardware, or on general compute x86 processors is a matter of implementation, with merits and de-merits on both approaches. Efficiency is usually achieved when software sits as close to specialised hardware as possible.

On networking and political systems

I don't really know why I decided to write tonight. There is plenty of better stuff to do than this. Maybe it is because I am in a bad mood because I got my bicycle stolen today (and it's the second in 12 months …). Or maybe because I haven't posted anything in a long time. Whatever the reason, and for whoever cares to spend time reading, here it goes …

A couple of weeks ago I read the blog post on "Democratising Capacity (or how to interpret Cisco math)", by JR Rivers CEO of Cumulus Networks (and also an ex-Cisco employee). I recommend reading it. 


It provides a critic on a claim that Cisco made about being capable to provide more affordable network solutions than bare-metal offerings (here). His analysis concludes that this isn't true, and criticises the lack of transparency on pricing of Cisco, and the closeness of the system (since you have to buy the optics and even the cables from Cisco - something which isn't accurate, by the way). 

I know that the current buzz in the industry is all about "open" and "dis-aggregated", and is all cool and well. And that all that is evil in networking industry is to blame on Cisco (as if they had been the only vendor in the industry for the last 20 years or more).  I also know that white-box switches with Linux-based operating systems have been selling for years before (without being cool, and without being very successful until now). The code quality and feature set of those white-boxes switches isn't particularly great. 


Cumulus Linux adds a change to that. I think ONIE is interesting, and bare-metal switches supporting it, together with Cumulus Linux will one day make a more competitive offering than white-box switches have been in the past, if Cumulus Networks proves to be capable of delivering a good and rich operating system and quality support. 

And this is a good thing. Competition is good. Period.

But there are a few things on that post that I think are questionable, and I will humbly question them.

Let's begin:  "The attributes of transparency, choice, and degrees of freedom, not price, are driving all of the mega-scale customers to bare-metal networking solutions". 

Of course I can't speak for what the mega-scale DC operators are doing and why, but what I have been reading on this topic and from the coverage of the OCP Summit this week, everything points to price as the main motivator for all of those that make the move (example on Facebook just this week). 

I also don't understand the critic on lack of transparency and choice. Certainly on freedom of choice, IT organisations can choose from many hardware vendors. Now, and last year and for the last 20 years (and more).  Granted, they can't decide to source the hardware from HP and run NX-OS on it, or to buy from Cisco and run JunOS, or whatever. Nor I think many customers would want that either. In any case, choices, … there's always been plenty. 

On transparency ... I think here the critic is mainly about pricing transparency. The Cisco list price is clearly available (and JR Rivers has access to it, as he quotes it). Cisco standard discounts and partner programs are available as well. Are there special discounts for large volume deals? Sure. Like in every industry. Every vendor does it. 

In fact, Cumulus Networks does it too. JR is guilty of what he criticises (if I got the critic right): the list price for a certain Cumulus Linux SKU is $699/switch/year, but thanks to an annual cap on volume can be as low as $150/switch/year. Is this valid for every customer? what is the level of volume required to get to this figure? Is it 10,000 licenses per year? what is the number I pay if I say 7,000 licenses per year?… It depends, I am sure. It is a negotiation.
How is this different from Cisco, or any other vendor, giving an special discount to a customer on a volume order? 

Further, the  list price he quotes for the AS5610-52X from Edge-core Networks is actually lower than the retail price you can find on line.  The actual model on Edge-core Website is the AS5600-52X (which is offered as a white-box or in bare-metal option supporting cumulus). I googled "AS5600-52X price" and I got here:

• $5,999 (http://unixsurplus.com/product/accton-data-center-switch-as5600-52x-48-port-x10ge-4x-40ge-tor-spine-switch-l3 )

• $5,999 (http://www.ebay.com/itm/Accton-Data-Center-Switch-AS5600-52X-48-port-x10GE-4x-40GE-TOR-Spine-Switch-L3-/141140747532)

There are more links but they either show higher pricing (with slightly different SKUs) or show no pricing.  I did not do an exhaustive search. Of course, JR cannot answer for the price transparency of another vendor (and he does say he encourages them to be transparent).

Besides, this is anecdotic. I am sure that JR is well aware of what is the actual list price, and I am sure he is bloody right. What I simply try to illustrate is that pricing transparency … is missing on the bare-metal model as of yet, but not so on established vendors. 


Freedom of choice. Well. Freedom to choose … Cumulus Linux. Because at the moment, if you are a customer that wants to buy the AS5610-52X from Edge-core in bare metal option, you can only run Cumulus Linux on it. Again, maybe in the future you will also be able to run another OS. Not today.

There is also another angle. And this is the most important one. We are not comparing apples to apples. And this is key. I could mention silly minor details like the fact that the Nexus 3000 from Cisco has double the DRAM and a more powerful CPU than the AS5600-52X. Or that it is also more energy efficient (even if it has double the DRAM capacity and more powerful CPU, consumes less power). But it is at the OS level that there are more differences. 

JR says a "normal" customer needs to buy the Enterprise LAN software license (N3K-LAN1K9). I don't know what "normal" means in this context. Most customers use this product for L2 server access, therefore not requiring that license. Some customers implement L3 networks end to end, again not requiring that license since on the base license OSPFv2 is included (for up to 256 routes). You need the license if you use a larger OSPF routing table (hardly required in an access situation), or if you need BGP or VRF.

However there are many things that you get on the Cisco base license that you do NOT get yet on Cumulus Linux. To begin with, vPC, to enable servers to use LACP channels to redundant switches for instance. Or L3 multicast routing (on the Cumulus Linux data sheet there is no mention of PIM support, since it is not part of Quagga I guess it is just not there at all). I could make this list very long, but suffice with that. JR comparison should start removing those $8,000 from the Cisco count ...

It is also not apples to apples to compare the support from Cumulus Networks and the support from Cisco. I think this is so obvious today that requires no further explanation. I can tell Nike that I will do publicity for way less than Ussain Bolt does, but I don't believe they will think that I can run nearly as fast as he does.

And … there is one final detail that is not clear to me yet. JR mentions the  AS5610-52X List is $4,200. The Cumulus Linux License & Support is $150 (yearly).  But … what about the HARDWARE support. I mean hardware replacement. The Cisco Smartnet also includes advanced hardware replacement, with Next Business Day Delivery (which Cisco honours on a global basis). 

Does this mean that the $150/switch/year on that calculation include advanced hardware replacement for RMAing a defective unit of the Edge-core switch on a global basis? … Important detail.

Finally ... price isn't really the point that matters most, for most people. It is about value. In my experience, (most) IT organisations care that their network works well. Therefore, what they need is objective information and facts to make an educated decision about what is best for them in this sense. 

Religion and political ideals about democratising I-don't-know-what belong somewhere else, not in technical-decision making.