On networking and political systems

I don't really know why I decided to write tonight. There is plenty of better stuff to do than this. Maybe it is because I am in a bad mood because I got my bicycle stolen today (and it's the second in 12 months …). Or maybe because I haven't posted anything in a long time. Whatever the reason, and for whoever cares to spend time reading, here it goes …

A couple of weeks ago I read the blog post on "Democratising Capacity (or how to interpret Cisco math)", by JR Rivers CEO of Cumulus Networks (and also an ex-Cisco employee). I recommend reading it. 


It provides a critic on a claim that Cisco made about being capable to provide more affordable network solutions than bare-metal offerings (here). His analysis concludes that this isn't true, and criticises the lack of transparency on pricing of Cisco, and the closeness of the system (since you have to buy the optics and even the cables from Cisco - something which isn't accurate, by the way). 

I know that the current buzz in the industry is all about "open" and "dis-aggregated", and is all cool and well. And that all that is evil in networking industry is to blame on Cisco (as if they had been the only vendor in the industry for the last 20 years or more).  I also know that white-box switches with Linux-based operating systems have been selling for years before (without being cool, and without being very successful until now). The code quality and feature set of those white-boxes switches isn't particularly great. 


Cumulus Linux adds a change to that. I think ONIE is interesting, and bare-metal switches supporting it, together with Cumulus Linux will one day make a more competitive offering than white-box switches have been in the past, if Cumulus Networks proves to be capable of delivering a good and rich operating system and quality support. 

And this is a good thing. Competition is good. Period.

But there are a few things on that post that I think are questionable, and I will humbly question them.

Let's begin:  "The attributes of transparency, choice, and degrees of freedom, not price, are driving all of the mega-scale customers to bare-metal networking solutions". 

Of course I can't speak for what the mega-scale DC operators are doing and why, but what I have been reading on this topic and from the coverage of the OCP Summit this week, everything points to price as the main motivator for all of those that make the move (example on Facebook just this week). 

I also don't understand the critic on lack of transparency and choice. Certainly on freedom of choice, IT organisations can choose from many hardware vendors. Now, and last year and for the last 20 years (and more).  Granted, they can't decide to source the hardware from HP and run NX-OS on it, or to buy from Cisco and run JunOS, or whatever. Nor I think many customers would want that either. In any case, choices, … there's always been plenty. 

On transparency ... I think here the critic is mainly about pricing transparency. The Cisco list price is clearly available (and JR Rivers has access to it, as he quotes it). Cisco standard discounts and partner programs are available as well. Are there special discounts for large volume deals? Sure. Like in every industry. Every vendor does it. 

In fact, Cumulus Networks does it too. JR is guilty of what he criticises (if I got the critic right): the list price for a certain Cumulus Linux SKU is $699/switch/year, but thanks to an annual cap on volume can be as low as $150/switch/year. Is this valid for every customer? what is the level of volume required to get to this figure? Is it 10,000 licenses per year? what is the number I pay if I say 7,000 licenses per year?… It depends, I am sure. It is a negotiation.
How is this different from Cisco, or any other vendor, giving an special discount to a customer on a volume order? 

Further, the  list price he quotes for the AS5610-52X from Edge-core Networks is actually lower than the retail price you can find on line.  The actual model on Edge-core Website is the AS5600-52X (which is offered as a white-box or in bare-metal option supporting cumulus). I googled "AS5600-52X price" and I got here:

• $5,999 (http://unixsurplus.com/product/accton-data-center-switch-as5600-52x-48-port-x10ge-4x-40ge-tor-spine-switch-l3 )

• $5,999 (http://www.ebay.com/itm/Accton-Data-Center-Switch-AS5600-52X-48-port-x10GE-4x-40GE-TOR-Spine-Switch-L3-/141140747532)

There are more links but they either show higher pricing (with slightly different SKUs) or show no pricing.  I did not do an exhaustive search. Of course, JR cannot answer for the price transparency of another vendor (and he does say he encourages them to be transparent).

Besides, this is anecdotic. I am sure that JR is well aware of what is the actual list price, and I am sure he is bloody right. What I simply try to illustrate is that pricing transparency … is missing on the bare-metal model as of yet, but not so on established vendors. 


Freedom of choice. Well. Freedom to choose … Cumulus Linux. Because at the moment, if you are a customer that wants to buy the AS5610-52X from Edge-core in bare metal option, you can only run Cumulus Linux on it. Again, maybe in the future you will also be able to run another OS. Not today.

There is also another angle. And this is the most important one. We are not comparing apples to apples. And this is key. I could mention silly minor details like the fact that the Nexus 3000 from Cisco has double the DRAM and a more powerful CPU than the AS5600-52X. Or that it is also more energy efficient (even if it has double the DRAM capacity and more powerful CPU, consumes less power). But it is at the OS level that there are more differences. 

JR says a "normal" customer needs to buy the Enterprise LAN software license (N3K-LAN1K9). I don't know what "normal" means in this context. Most customers use this product for L2 server access, therefore not requiring that license. Some customers implement L3 networks end to end, again not requiring that license since on the base license OSPFv2 is included (for up to 256 routes). You need the license if you use a larger OSPF routing table (hardly required in an access situation), or if you need BGP or VRF.

However there are many things that you get on the Cisco base license that you do NOT get yet on Cumulus Linux. To begin with, vPC, to enable servers to use LACP channels to redundant switches for instance. Or L3 multicast routing (on the Cumulus Linux data sheet there is no mention of PIM support, since it is not part of Quagga I guess it is just not there at all). I could make this list very long, but suffice with that. JR comparison should start removing those $8,000 from the Cisco count ...

It is also not apples to apples to compare the support from Cumulus Networks and the support from Cisco. I think this is so obvious today that requires no further explanation. I can tell Nike that I will do publicity for way less than Ussain Bolt does, but I don't believe they will think that I can run nearly as fast as he does.

And … there is one final detail that is not clear to me yet. JR mentions the  AS5610-52X List is $4,200. The Cumulus Linux License & Support is $150 (yearly).  But … what about the HARDWARE support. I mean hardware replacement. The Cisco Smartnet also includes advanced hardware replacement, with Next Business Day Delivery (which Cisco honours on a global basis). 

Does this mean that the $150/switch/year on that calculation include advanced hardware replacement for RMAing a defective unit of the Edge-core switch on a global basis? … Important detail.

Finally ... price isn't really the point that matters most, for most people. It is about value. In my experience, (most) IT organisations care that their network works well. Therefore, what they need is objective information and facts to make an educated decision about what is best for them in this sense. 

Religion and political ideals about democratising I-don't-know-what belong somewhere else, not in technical-decision making. 

Musings on the difference between server and network virtualization

There are many differences of course. For instance, volume is one, as it is the size of the overall server vs. networking market.

However, thinking about it, a key difference lies in what is needed vs. what is possible, and whether they match. Confusing wording, isn't it? ... 

With server virtualisation, the possible was joined with the needed. It was possible to run more workloads on existing hardware, meaning many servers were severely under-utilised, yet you still had to buy those servers. This was essentially (still is) a problem of the application and OS development really, for which a Hypervisor came to be a temporal solution. 

So it was possible to run more workloads on existing CPU/memory footprint (meaning the horse power was there already) and it was required. Why was it required? To spend less money in server investments (and in network along the way too, for you need less ports to connect less servers, obviously), of course. 

There was the big hit. An IT organisation could afford paying for a (somewhat) expensive virtualisation software because it would save you from buying a lot of hardware. Sure, on top of the virtualisation solution a lot more value was built soon afterwards, notably in HA and DR areas. But they payout was there from the need to use orders of magnitude less hardware. Oh, and you did not even had to change your preferred server vendor.

The same example is true for NIC virtualisation for instance. This is why technologies like HP VC with its virtual NICs or Cisco VIC have known success: you need to pay for less adapters. Same is true for FCoE to some extent. Essentially, you had idle capacity, and by applying a new technology you could make use of it, therefore investing less in hardware (be it servers, NICs, or whatever …). 

Every customer, large and small, could see this clear economic benefit.

THAT is a key difference with network virtualisation. You do not have idle network capacity that you need to use to buy less network kit. You can not afford to buy less network capacity by simply adding network virtualisation (certainly, not with an overlay solution). You have to buy the same number of ports to connect your servers. Your traffic volume won't be less either, therefore you cannot afford to buy less powerful aggregation or core boxes to interconnect your access switches. 

This is not to imply that network virtualisation does not bring any value. Some will be quick to point out agility, others scalability. Both arguments subject to debate, but in any case, none of them (a) applicable to every customer, and (b) having a distinct economic impact in every environment. 

Why Canonical (and Microsoft) may have it right ...

It's been a while since the last post ... and I am shifting gears quite a bit away from networking too. But I was very interested by the announcement of Ubuntu Phone.

I have been an amateur user of Ubuntu Desktop for quite a while and I have always thought it was an amazing operating system. When I saw the activity on twitter I went to check on their website and I found something very sleek, neat and attractive: much like the desktop of course. There are a few "demos" available on YouTube and they display a very modern, elegant and more important, different user interface. An example is here:

I think it may have a niche competing with Android on low end budget SmartPhones and in any SmartPhone that is sold in budget sensitive markets. Manufacturers will look at alternatives to Android for multiple reasons, and one of them will be differentiation. Ubuntu Phone will provide that. It's not the only one offer of course, but it's a very nice one for sure.

There's been some critics about the fact that it is late to market (very possibly), since Android and iOS are dominant, and the battle for the third place is ripe for the taking by Microsoft. More interesting are the critics that say Ubuntu has got it wrong in that they are offering a mobile OS which is also a Desktop OS. The critic goes that Desktop is dead, and we should not try to keep hanging on to the old desktop habits and so on.

The timing is indeed an issue. Looks like the first devices officially available with Ubuntu would not show up before end of 2013 or early 2014 and Canonical has had a poor track when it comes to mobile OS in the past. On the other hand, the battle for the third place is so fragmented that they may still have a chance and quite frankly, it does not look as of today that Windows Phone is making serious inroads.

A Desktop? A Phone? or both ... bad idea?

Ubuntu Phone claims that on high end devices you will be able to use it as a desktop. So the concept is that you can run both Phone Apps and Desktop or Web Apps on it, and you can also dock it and connect it to a screen, keyboard and mouse and use it as a real desktop, with a real (good) desktop interface. So Ubuntu Phone is also Ubuntu Desktop. Microsoft is proposing a bit of the same thing with  Windows 8, although so far they have been pushing the concept into the tablet (Surface), not really into the SmartPhones.

Is this a bad idea? Is this hanging off to a forgotten past? ... I think not. I believe the Desktop isn't dead. I have an iPad, and an iPhone, and I haven't stopped using my MacBook. Real work gets done on a desktop. When you need to spend time building or working on complex spreadsheets, or writing a content rich document, or a presentation, or programming ... you want a Desktop. ... or ... do you? Well. I think you need a Desktop and by this I mean a comfortable place, a nice (big) screen, mouse and/or trackpad and a proper (comfortable) keyboard. Where the applications run on is less important ...

And this is where I think that Ubuntu Phone got it right. There are two other niche markets where Ubuntu may grow. One is the geek power users who already use Ubuntu Desktop. The other is, potentially, anybody who sees the benefit of having their Desktop with them all the time. I do not see why not. I'd love to have my MacBook inside my iPhone, so that when I am in a place to work, I can dock it and work on it ... Sure, today's high end Smart Phones aren't as powerful as a desktop/laptop computer, but think in two, three years they may be powerful enough. 

In fact, today they may be already ... for you can have a thin client on them and then off load the heavy work to a virtualized OS which runs where there's power for it. For enterprises, this is good. 

Time will tell ... but for now, I like what I see. I can't wait to get my hands on it even if it's running on a VM only. Probably the main reason why I like is because it is different. Unlike Google's Android, they did not just went on to copy Apple's user interface as a means to penetrate the market and then slowly tweak it to claim it's different and show up some "innovation". No. They created something new, something different. Well done.

A sad failure?

Very recently I have read an article on Network World about some investment firm warning on issues with Juniper's problems and/or delays on shipping certain recent products and technologies. Amongst them QFabric was cited.

It is all "alleged" information, as much as I know, but it comes to no surprise. I don't have a close knowledge of the service provider core routing market to speak about the T4000 or the PTX system, but I have followed very very closely QFabric since it was announced (and before it was announced as well).

It is quite shocking that one year, yes, one full year, after a big announcement like that one, from a company like Juniper so quick to come up with customers endorsing them, no single customer reference is out there. Of course this may change any day now, possibly, but it will not hide the failure already. It clearly means when it was announced it was not ready (it shipped six months after), and it seems to indicate that when it shipped it had a number of issues ...

Despite all the mind share captured, all the good press, all the bloggers fascinated with Juniper's QFabric marketing claims (naively  I must add, in most cases), the fact is that JNPR's switching market share remains marginal. Marginal, after being in the marketplace for over 4 years now considering the EX product line, which still is the only one selling in some volume afaik. Contrast this with Cisco's UCS for instance, and the impact it has had in little more than 2 years as a comparison.

I may be wrong, and I really hope I am wrong, but I think JNPR will need to recognize the failure and change the strategy. To me, based on what we know now, it is a failure, and a sad one. Sad, because I expected better from Juniper, and sad because healthy competition is great for every company in the market, and most important, for customers.

And I think it is a failure for various reasons:

1. Architecturally. It is very, very complicated to scale a distributed ethernet switch to the levels that Qfabric is intended to do.  And if it can be done (and having experience with distributed ethernet switches to very low level I have serious doubts about it)  it is very difficult that such thing is done while maintaining the system economic and simple to operate. It already isn't either of those two.

2. QFabric missed the industry trends. There are two key trends:

- Large L2/L3 fabrics (or simply put, networks :-) ), with  some form of edge-based overlays to deliver network virtualization. While QFabric could provide the basis for this, it fails at scale. It is too big for smaller deployments, too small for large ones. But more importantly, it has no value proposition. If I want to build a L3 fabric to then run a Distributed Edge Overlay (DEO) I can use standard OSPF or ISIS engineered networks which will support a much larger number of switches in the fabric than QFabric allows and can be potentially built using a number of vendors, and even combination of vendor equipment etc. Why would you do it with Qfabric?!

- Software Defined Networking: you can jump on the band wagon by adding marketing, claiming open APIs and whatever. Organizations looking at this have anyone one of two angles: integration into cloud stacks (i.e. OpenStack, where JNPR's Qfabric role is marginal/none), or open capabilities to manage forwarding elements from a controller (i.e. ala BigSwitch) using Open interfaces like OpenFlow. JNPR's approach today fits none. QFabric is proprietary in its architecture and does not lend itself well to allowing companies build tradicional fabrics not migrate to OF controllers as it is today.

In the end ... for those who are happy with building networks using tradicional ethernet technologies and evolving with those, QFabric does not offer any compelling value proposition. And for those facing real problems with today's state of ethernet technologies ... well ... the solution cannot be to simply build a larger ethernet switch.

But that said, I know there's great talent working on QFabric, and I really hope they will get it right, as right as it can be.

As always, opinions posted here are my own ...

Some musings for old networkers ...

Back in 1999 I was working in South America. Most service providers in the region were operating TDM and FR networks and the hot technology they were evolving to was ATM. It promised many good things, including a true converged network capable or transporting data, voice and video. Of course many at the time were not thinking of packet data (nor packet video), but actual PCM voice channels over CES.

Why am I bringing this back today? Well, because back in those days I remember having so many conversations with SPs about delivering virtual private networks using IP technology, or in fact, a nascent technology that leveraged an IP control plane: tag switching, which evolved into standard MPLS. For Cisco, this was a key differentiation in the Stratacom product line of ATM switches.

But most people in those carriers were very circuit orientated. The easiest way to setup a private network for them was to mimic the TDM world of circuits and channels ... with a PVC - or a mesh of them.

It is funny to see that, in essence, today's latest and greatest solutions for implementing virtual private networks (network virtualization) still relies on circuits in some sort of way ... IP overlays now.

The kings of the ATM SP market at the time in the region were Nortel and NewBridge. Cisco had important franchises as well. SPs were looking for ways to scale better the circuit based ATM backbones to deliver services to end users (some were thinking about going into households even, which eventually happen through the earlier phases of ADSL deployments in early 2000s).

How to build and manage all those thousands of PVCs? ATM proposed a neat way ... SVCs! VCs which were created by a software layer and at the request of the application! ... Good things were about to happen. And then it came the fierce competition ...

Nortel and many others were pushing for PNNI, a routing protocol plane, network based, which would run distributed in all ATM switches in a hierarchical topology. PNNI was essentially a routing protocol for establishing ATM VCs. Good about it? It was standard ... promised vendor interoperability and in fact there were bake-offs to prove it and so on ... Cisco was betting on PNNI and actually had a pretty robust stack implemented on its LS1010s.

NewBridge on the other hand had a radically different approach. They had a very strong management solution (the 46020) coming along from the TDM and FR ages. Essentially for them, a software running on a server would control the setup and management of PVCs and SVCs alike! ... better, it could interface through APIs with an application layer for, say, deliver IP and VoD services to ADSL consumers for instance ...

The "SDN" approach to managing ATM never worked, it failed to scale and proved to be a lock-in for customers who could never think to imagine running an ATM switch that was not NewBridge.

Of course, the end of the story I just related is well know ... ATM and PNNI slowly died, being overruled by IP and MPLS. NewBridge was acquired by Acaltel (which actually reused its 46020 for managing the 7050/7040s boxes and did well with it), and Nortel ... well ... sorry for them.

I can't help to think of this and see some analogies with recent trends and announcements in the networking industry. By no means I pretend the end could be similar though ... much have changed in the compute industry in particular to help scale a controller managing overlay networks. But it is fun to think of the irony of things, and how circuits come back to haunt us IP heads ...

Nicira: fear them not

So Nicira is finally out of stealth mode. This is good news. Much of what we have seen from their website now is confirming rumors and expectations. In the press however, there continues to be a bit too much hype in my opinion. Talks about the next VMWare are a bit out of plance I think, if for no other reason, because the server and networking industries are very very different (in dollar value to begin with ...).

The general assumption is that networks are very static and difficult to manage and adapt to business needs. Michael Bushong from Juniper writes that they  "are far too big and complicated to run by hand and are therefore operated by a maze of management, provisioning and OSS/BSS systems".

I guess when you are coming from a small installed base as a networking vendor in this space you want to exaggerate the issues faced by DC and Enterprise networks today. It is true that networks are not managed by hand, and are managed by OSS/BSS systems, but then isn't this a good thing anyways? And more importantly, isn't this true for server, server images, and storage as well? I wouldn't say managing hundreds or thousands of VMs with different images, patching levels, etc is a simple task that anybody want to run by hand.

But it is true that networks are static and that automating network configuration isn't an easy task. Adding ports to VLANs can be automated in a somewhat easy way. But things like stretching VLANs, or moving entire subnets around are a more difficult task. Now, a network engineer would claim that the problem isn't the network itself, but the way applications are built.

After all, if you build a network based on L3 with proper subnet planning, you will never have an issue allocating network resources for any VM you provision on the network, and all a VM needs for communication is an IP address. But the issue is that applications aren't built to run in "just any subnet", for once, they need to communicate within the subnet for many tasks with other components of the application. And then there's policy and security which if tied to the IP Address, becomes a nightmare to manage and enforce. And decoupling the policy and security rules from the IP address isn't easy to do today either.

There are many things I agree with Mr. Bushong on though, and one is that "programmability is about adding value to the network control, rather than a threat of commoditization".

Several years ago I read a paper on Microsoft's VL2 proposal. It is very similar to what Nicira is doing in concept: at the server network stack you build a tunneling mechanism that facilitates endpoint communications. At that moment I thought such approach wouldn't be feasible for it demands to change the server TCP/IP stack, a daunting task. But virtualization has changed that, because now we CAN change the stack at the vSwitch level, while the server OS, close to the application, remains unchanged. Nicira has also added one more thing: a northbound API to provisioning systems that can harmonize the network connectivity for endpoints with other resources (server, storage, etc.).

In itself, Nicira's solution isn't providing anything new: builds overlays to facilitate endpoint communication. This can be done with VXLAN  as well, what Nicira is providing is, supposedly, a control plane capable of creating and managing those overlays in an automated and scalable way. The latter point is to be confirmed of course.

Many seem to think that this will be the end of networking as we see it, and that the physical network becomes a commodity. I think this isn't true. First because building a large, scalable, fast and performing L3 network ins't rocket science but it isn't something many have succeeded at. There is a reason why Internet operators rely on just two companies for that: Cisco and Juniper.

Second because as you want to improve efficiency of utilization of your physical topology, and provide differentiation to applications that require it, your PHYSICAL network must have a way to view and interact with your overlays.

And there is more. Once you have built an architecture that enables you to create such overlays to allow endpoint connectivity, what happens when connectivity needs to be done with elements outside of your overlay? You need a gateway out, and a gateway in. I can see ways in which you leveraging OF and a controller you can scale the gateway out from the vSwitch itself, but scaling the gateway in is more difficult and chances are it will be done via appliances of some sort, which then need to be redundant etc.

So I think the more Nicira's we have the better. The more development we see that facilitates moving towards cloud architectures, the more demand for performing and intelligent networks. I do not see commoditization happening in the network for the same reason hypervisors haven't commoditize the CPUs. Intel and AMD are now building CPUs that offer more services to the hypervisor layer, in the same way, we will see networks which offer better services to the overlay networks.

Net net, I am one who thinks that much of the complexity in networking isn't created because of wrongdoings in the past, or just legacy technologies. It is because dealing with a network isn't like dealing with endpoints. It is a complex and evolving challenge in itself.

Bright times ahead for the industry …

On Soft Switching and Virtualized Networking in General (Part I)

Wow, it's been almost three months since I last wrote here. Too bad. I actually have at least four posts on the works, but I never seem to have time to finish any of them. Apart from lack of time, sometimes I am too ambitious and perhaps each post could be divided in several smaller posts.

That's what I decided to do with one of the subjects already and this will be the part I. I want to write about the challenge of networking virtual machines, and go through the various options considering pros and cons and where different solutions fit.

As always, the writing represents my opinions only, and are based on my limited knowledge of the subject. On this first part I review the challenges of networking for virtual machines and talk about what others write about it. Then on part II I will talk more about Soft Switching economics and where I think it fits and on part III I will do a similar thing about the hardware approach (VM-FEX). Finally, I am planning a part IV with details of current VM-FEX implementation on Cisco Nexus switches.

The Challenge of Networking in Virtualized Environments

Much has been written about this topic and as a networking-head fully involved with virtualization I find it so interesting that I spend quite some time reading and thinking about it. Finding the free time to write about it is another story.

There is no doubt that it is a challenge. Networking a large number of servers isn't that easy a task already, but with Virtualization you have to add the challenge of a much denser number of endpoints (the VMs) on the network, and also the mobility aspect of virtual machines to complicate matters. But scale and mobility are just two of the dimensions to this problem, with security, performance and manageability probably being the other top ones.

In general, I believe there are two approaches to solving the challenge of networking in virtualized environments: software based solutions, integrated into the hypervisor, and hardware based solutions seeking to off-load switching from the hypervisor. I think both have a place.

Martin Casado, the network heretic, has written much about why he thinks soft-switching "kicks mucho ass" and will be the winning solution. He dislikes solutions which leverage NIC virtualization (SR-IOV, 802.1BR) or those which force traffic to be pushed off of the server to have it hair-pinned if needed (802.1BR, 802.1bg). Many other bloggers have bashed tagging based solutions (i.e. 802.1BR) saying they aren't needed and they are just another attempt from hardware vendors to sell more and new hardware. It is undeniable that vendors develop technology hoping to get a profit from it, and it's quite a legitimate thing to do as well in my opinion, but I am dead sure that no vendor develops a technology to force a customer upgrade. Plain simple: if the technology isn't solving a problem, it wont sell. People isn't stupid. 

The problem we are solving (networking in highly virtualized environments) accepts multiple solutions, and it is quite different for different people. I think we can distinct two big type of organizations: IaaS service providers and Virtualized deployments in Enterprise space (i.e. Private Clouds). In the first type of customers, soft-switching may have an edge, while on the second hardware switching can be more interesting.

Casado dislikes hardware switching and I think has a good case on this for cloud service providers as I just mentioned. The best way to understand why he dislikes those hardware based options and thinks soft-switching will prevail are well explained in his blogs (which I recommend reading entirely, there's four parts), but in a nutshell he believes that software will always be feature-richer and with the performance and low price per core today the economics of soft-switching are ideal.

I disagree with him to some extent. Certainly to the extent of scoping the analysis to the entire industry. I think his analysis considers only a part of the industry, and that both options (soft and hard switching) will have its place and space to bring solutions to virtualized server networking. Let's see if I can explain why I think this way in the subsequent posts. Then I will explain the benefit of hardware based solutions and how they work. 

I must stress again that my writing on this blog is done in my free time and reflects only my own personal interests and opinions.