IP (Internet Protocol)  

Also see below: IP 101

November 6, 2000
by Peter Morrissey

In many ways, the task of a router is very simple. It must look at the destination address of an IP packet and then forward the packet to another router or to a directly connected host. However, if you've ever attempted to get two devices to communicate on an IP network, you know that it doesn't always work.

In this Network Design Manual article we'll show you the steps involved in getting a packet from Point A to Point B on an IP network. We'll talk about the settings that are necessary on a host to communicate on a routed network and what they really mean. Once you understand this, you'll be able to ask the right questions when things don't go as expected and even do a little bit of your own troubleshooting.

Anatomy of an IP Address

When one host has data to send to another host, it sends it to its local IP process, which builds an IP packet for transmission on the network. The packet has a header, which includes the destination IP address as well as the source address of the local host. The destination and source IP addresses stay with the data until it arrives at the host so that all the routers it traverses know in what direction to send it. The source address provides the destination host with an address to which it can send its response.

To understand how this happens requires a deeper knowledge of the IP address. It's obvious that an IP address consists of four separate numbers, separated by dots. Each number represents one byte worth of data, limiting it to a range of 256 possible values, commonly represented as decimal values from 0 to 255 or binary values from 00000000-11111111.

One thing that is not so obvious is that each IP address is always divided into two sections. The first section always delineates the network address, or subnet. This is the part routers are primarily concerned with. The second section, combined with the first section forms a unique node address. The subnet mask is configured to tell the host or client which part is the network address and which part is the node address.

Subnet Masks

The subnet mask is a required setting on any host that communicates on an IP network. It is commonly represented in the same format as an IP address, except that the values usually are either 255 or 0. (Exceptions to this are becoming more common--we'll talk about those later). If you've ever seen a subnet mask, you will notice that a series of one or more 255 values are always designated in succession, followed by a series of 0 values. If you line a subnet mask up over an IP address by the dots, each number below a 255 will be part of the network address. The remaining numbers under the 0s will be the node address.

With this type of subnet mask, the boundary between the network and node address is always on a dot between the last 255 and the first 0, sometimes referred to as the byte boundary.

What really matters to a computer of course is the binary representation. The above mask and IP address are represented in the following manner as binary:

With the binary representation, an uninterrupted sequence of 1s in the mask, line up exactly over the subnet portion of the address. With this type of mask, seeing the binary version doesn't add any value, but later it will be necessary to see the binary data to fully understand masks that don't end neatly on one-byte boundaries.

Part of the decision as to how to set up the subnet mask is left to the network designer, but the subnet mask must be configured consistently throughout the network. Another factor that limits the definition of a subnet mask is its class. For example, with a "Class B" address, the first two bytes always have to be included in the network portion of the address, and thus the subnet mask will always start out with 255.255 or 11111111.11111111. With a "Class C" address, the first three bytes have to always be included in the network portion of the address. In this case the subnet mask would have to start with 255.255.255.

In each instance, there is a lot of flexibility in the way that the rest of the address is masked. The example above is a Class B address. The network address is 128.230.100. The node address is 4. The example above used a subnet mask of 255.255.255.0. This makes it possible to have 254 unique, routed networks or subnets because the mask can include the third byte in the network address. (You can't use 255 because that's reserved for broadcast address, and 0s, used in an older style of broadcast address, can't be used either; 0s at the end of an address are also used to symbolize network addresses.) Even though the first two bytes cannot be changed, the third byte can have any value from 1 to 254. This means that with a three-byte or 24-bit subnet mask, the following distinct networks or subnets are possible with the address:

There is one byte left over for node addresses, so each network or subnet will have 254 node addresses. If the minimum subnet mask of 255.255.0.0 were used, there could only be one routed network. But two bytes would be left over for node addresses. So the range of possible node addresses would be 128.230.1.1 to128.230.254.254 or approximately 65,000 nodes. The disadvantage of this is that you cannot divide your network into routed subnets.

With this particular address, if you divide only the subnet mask on the byte boundaries, you have 254 subnets limited to 254 nodes each, or you have one big network with 65,000 nodes. These two extremes have some obvious trade-offs. There may be cases where you would want more than 254 nodes on a routed subnet, but the other extreme flattens the network completely, giving you no ability to divide your network into routed subnets.

Fortunately, the subnet mask doesn't have to divide the network address and the node address on a dot boundary. In some cases it can be divided somewhere in between. When the network address and node address are not divided on a dot boundary, the subnet mask is called a variable-length subnet mask. A variable-length subnet mask is commonly represented in decimal form, such as 255.255.254.0. But its binary representation has to be considered to understand where the network address ends and the node addresses can begin. The following example uses a subnet mask of 255.255.254.0:

To understand this, we'll have to look at it in binary form. Here the mask does not line up on the byte boundaries:

One thing you should notice is that there are contiguous 1s from left to right in the subnet mask. This is one of the rules of a subnet mask: All the binary 1s have to be contiguous, and all the following 0s overlaying the node address have to be contiguous. A mask that doesn't follow this pattern is not valid. In any case, the valid node addresses are all possible numbers that can be represented under the 0s, so the lowest number is going to be: 128.230.4.1 or 10000000.11100110.00000100.00000001. And the highest number is going to be 128.230.5.254 or 10000000.11100110.00000101.11111110. You will also notice that this mask has one less one bit than the previous mask. This is sometimes referred to as a 23-bit mask.

Because subnet masks have to be contiguous 1s followed by contiguous 0s, the last byte of a subnet mask can only have the following binary or decimal values:

Broadcast Addresses

There are times when a device has to send an IP packet to every node on a network. In such an instance, it will use the broadcast address of that network. The subnet mask is also used to derive the broadcast address. This is done by replacing all of the binary digits in the node address, as governed by the trailing 0s in the mask, to values of binary ones. This is combined with the network address, which results in the broadcast address. Following is the broadcast address that would go with our above example.

Broadcast addresses also are easier to figure out when the masks end on the one-byte boundaries. When this is the case, it can easily be represented in decimal by substituting decimal 255s in place of the bytes where the mask indicates the node address belongs. Here's an example:

To Route or Not to Route

The rules of IP networking dictate that any time a host has an IP packet to transmit, it must first determine whether the destination IP address is on its own network or another network. If the address is not on its own network, the host must send that packet to a router. The host determines this by comparing the network portion of its own address with the network portion of the address with which it wants to communicate, or the destination IP address.

At this point in the process, the subnet mask comes into play. The subnet mask determines how much of the address is the network address. By doing so, it helps ascertain the comparable portion of the host's and the destination' s IP address. If the network portion of the addresses is different, the packet is sent to the router; if it is the same, a router is not needed. The packet can be sent directly on the local network.

In the following 23-bit mask example the destination address is on the same network as the source IP address. The portion of the destination address directly beneath the contiguous binary 'ones' of the mask address is the same as that of the source address:

If the mask is changed to a 24-bit address, the result will be different. Here, those sections of the source and destination addresses overlaid by the binary 'ones' of the mask are different, thus proving the addresses are on different networks:

In this example, the packet must be sent to the router. It's because of such differences that every host has to have a router or gateway address configured. If a host can communicate with other hosts on its own subnet but not with those not on its subnet, then either the mask or the router address is incorrectly configured on the host.

IP Over Ethernet

It's important to remember that IP operates on layer 3 of the OSI model. But for IP's bits to be physically transmitted on a network, it must rely on the protocols in layers one and two of the OSI model. Enter Ethernet.

At this point it is important to remember that data on an Ethernet network cannot move from point A to point B without knowing the Ethernet address of point B. Also, an Ethernet NIC (Network Interface Card) will ignore all Ethernet packets without an Ethernet address as the destination address. The only exception is the Ethernet broadcast address, which turns the destination Ethernet address to all binary 'ones.'

Continuing with the example above, once packet transmission via the local network vs. the router is determined, another step is necessary. This step involves the Address Resolution Protocol (ARP). ARP sends a broadcast out on the network using the IP broadcast address discussed earlier. This broadcast, at which all network devices will look, includes a query for the Ethernet address of the device with the destination IP address.

Every device on the local network-note, the router will not propagate the broadcast to other networks--will then look at the Ethernet packet, and the device with the destination IP address in question will respond to the Ethernet address query. Once this address is received, the transmitting host can place the IP packet inside an Ethernet frame and send it to the destination host. When that host receives the packet it throws away the Ethernet frame, leaving the IP packet.

Packet transmission via a router is no different. If the destination IP address is on another network and has to be sent through a router, ARP must still determine the Ethernet address of that router. And like the host, when the router receives the packet it strips off the Ethernet address. Remember, routers are only interested in the destination IP address.

Routing

When a router receives a packet, it is really only concerned with the packet's destination network address. This is the only information the router needs from the packet to do its job.

Like hosts, routers use masks to determine the extent of the network address within the destination IP address. Every router keeps a table listing all the networks it knows, along with the interface it needs to transmit the packet in the right direction. This is called a routing table. Also like a host, a router uses masks to determine whether it is directly connected to a network, automatically adding a corresponding entry to its table. It is possible to manually enter routes into a routing table. These are usually called static routes.

If there is only one router in the network, then it isn't necessary to worry about anything other than directly connected networks. If there are a few networks that are not directly connected, then it will be necessary to tell the router the direction in which to send packets destined for the network. This can be done by manually configuring the routes. If there are numerous routes, then it will be necessary to rely on routing protocols to do this automatically.

Routing protocols give routers the ability to automatically inform each other of any directly connected routes. Not all routing protocols are created equal, though. For example, with some routing protocols, such as RIP version 1 and IGRP, only one subnet mask can be used on the whole network. With other protocols, it is possible to have different subnet masks on different parts of the network. The next section, "Choosing an Interior Gateway Routing Protocol," will talk more about routing protocols.

Troubleshooting Tips

A handy tool for troubleshooting routed networks is the "traceroute" utility. By entering the traceroute command followed by the destination IP address that you are trying to reach from a UNIX workstation, you will be given the IP address of every router that is reachable along the path between the two machines. The command looks like this:

Similarly, you can run Microsoft's version of this command (tracert) from the DOS command line on an NT or Win9x machine.

The "ping" command is also an indispensable tool for troubleshooting basic communications problems. It is available on Microsoft, UNIX and Novell hosts. Ping followed by a destination IP address will attempt to send a packet to the destination host. The host will then echo a reply. On a Unix machine, this command looks like this:

If your machine is able to communicate with local hosts but not with hosts on the other side of a router, the most common cause is either a misconfigured router address or subnet mask. If you run into this problem, be sure double check your router and subnet mask settings before you start worrying about the network. The easiest way to check the current configuration on a Windows 9x host is to run the "winipcfg" command. On an NT host, it's slightly different, "ipconfig." You can get this information from a UNIX host by running the following command.

Once you've verified that these settings are correct, you can ping the router's address to verify that it is reachable. If you still cannot reach the destination host, it's certainly possible that there is a network problem between the host and the router.

Choosing an Interior Gateway Protocol

When you enable routing protocols and their corresponding processes, you unleash a powerful, dynamic force that automates the population of routing tables on your network. But using any major IGP (Interior Gateway Protocol), such as RIP/RIP2, OSPF or IGRP/ EIGRP has trade-offs.

The first rule of thumb is to implement IGPs only where a single administrator has responsibility for the operation and performance of the network. Otherwise, you can expect serious problems such as configuration errors, which may bring down the network or cause stability problems. In addition, when problems occur, you'll want to be able to fix them quickly without waste time arguing about which administrator caused them. For networks where many administrators share responsibility, such as the Internet, consider an EGP (Exterior Gateway Protocol), like BGP4. We'll explore BGP3 in the next section.

If you have only one router, you may not need a routing protocol, which is necessary only where you have multiple routers that need to share information. Even then, if you have only a handful of networks, you can update the tables manually via static routes, though they don't scale very well. That's where routing protocols come into play.

Routing Information Protocol

The Routing Information Protocol (RIP) is based on a program called "routed" that was developed to run on the BSDI version of Unix. It was standardized in RFC 1058 in 1988. Version 2, specified in RFC 1388, added support for VLSMs (Variable Length Subnet Masks), but it did not address the protocol's major weaknesses: delay. In networks with multiple paths to the same destination, for instance, it takes some time before RIP will move to use an alternative path.

In spite of the popular phrase, "RIP: May it RIP," RIP is well tested, widely implemented and ideally suited for networks without redundant routes. RIP v.1 is still used to support most of Syracuse University's 10,000-node network. It does the job, and the university has not suffered a serious network outage in over two years.

RIP is classified as a distance vector protocol, which means it uses distance, as measured in routing hops, to determine a packet's optimal path. Routers send out advertisements to one another every 30 seconds. Each router that receives a given advertisement increases the hop count by one. If advertisements are received from multiple routers, the path to the router with the lowest hop count is the path chosen. Should the preferred route be unavailable, the route with the higher hop count is used as a backup.

With RIP as well as other routing protocols, the routers on a network must go through a process to determine alternate paths when one path becomes unavailable. This process is called convergence. As stated above, the fact that RIP takes a long time to converge is a major problem. RIP was designed to wait until it has missed six updates, totaling 180 seconds, before it will consider a route unreachable. It then waits for the next advertisement of another available route before it updates the routing table with the new route. This means at least three minutes will pass before a backup route can be used, which is certainly long enough for most users to notice a lag and for most applications to time out. Of course this latency won't pose a problem, if you have only one route to any destination anyway.

The other fundamental problem with RIP is that it ignores the speed of links involved when choosing a path. For example, if one path consisting entirely of Fast Ethernet links is one hop further away than a path that includes a 10-Mbps Ethernet link, the path with the slower 10-Mbps Ethernet link will be incorrectly selected as optimal.

The original version of RIP was unable to use VLSMs, which kept administrators from slicing and dicing address space to make the most efficient use of limited IP addresses. RIP 2 fixes this by advertising the subnet mask in use with every routing advertisement. Because the earlier version did not advertise a subnet mask, it had no way to convey the details of different-length masks for different networks or subnets.

Routing protocols should also prevent your packets from going in circles, or falling into routing loops, a problem that affects networks with redundant links. RIP assumes that if there are more than 15 routing hops from one end of a network to another, loops must be involved. Therefore when a route reaches 16 hops it considers it to be unavailable. Obviously, this limits RIP to networks in which no more than 15 routers must be traversed.

Obviously, RIP's biggest problems involve larger networks with redundant paths. If your network does not have redundant paths, RIP should work just fine. It is an Internet standard implemented by just about every vendor that supports routing. RIP is implemented in most server operating systems and is very easy to configure and troubleshoot. If you have a larger network, or one with redundant paths, however, you should seriously consider one of the other routing protocol choices.

Editor's Note: Tune in next week, as we tackle alternatives to RIP such as the venerable OSPF Version 2.

OSPF Version 2

OSPF (open shortest path first), an Internet standard like RIP, was designed to aggressively address RIP's shortcomings. It was first standardized in RFC 1247 in 1991. The latest version, OSPF v.2, can be found in RFC 2328 (version 1 was never implemented). Unlike RIP, OSPF is a link-state routing protocol, which means that routing changes are based on the status and speeds of the physical links in the networked routers. Moreover, these changes are immediately propagated to every router on the network.

When an OSPF router is first activated, it uses OSPF's "hello protocol" to discover any neighbors to which it is connected. It then exchanges link-state information with these routers in the form of LSAs (link-state advertisements). Using this information, each router creates a database that consists of every interface, its corresponding neighbor and a metric representing the speed of that interface. Each router then uses LSAs to pass this information along to all neighboring routers. Every LSA that a router receives from a neighbor is passed along to its other neighbors in turn until every router receives the LSAs of every other router in the network.

It's important to note that a link-state database is distinct from a routing table. From LSA information, each router calculates a path to every destination on the network, building a tree with itself at the root. This comprises its SPF (shortest path first) tree, which forms the basis of the routing table. LSAs are exchanged every 30 minutes, unless there is a change in network topology. If an interface goes down, for example, the information is propagated across the network at once.

If there is a redundant path, the convergence will last as long as it takes to recalculate the SPF tree and update the routing tables for the affected network. This can happen in a few seconds or less, depending on the size of the network. Because of these calculations, routers running OSPF require more CPU resources. This becomes even more critical on an unstable network where interfaces are up and down frequently and a lot of CPU resources are required on the router. The flooding of LSAs may also cause problems on WAN links if they happen too frequently.

One way that OSPF compensates for increased CPU and memory demands is by dividing the network into separate, hierarchical domains, called areas. Routers exchange LSAs only with other routers in their own areas. There is also a backbone area known as Area 0. All areas must be adjacent to Area 0. A border between two areas is defined on an ABR (area border router). ABRs have at least one interface in Area 0 and one interface in another non-backbone area.

The best designed OSPF networks contain contiguous networks to each area, which can be summarized on the backbone through VLSMs (variable-length subnet masks). This makes it possible to describe multiple networks in one routing table entry. The rule of thumb is that there should be about 50 routers per area, but this can vary based on the number of interfaces per router and their stability.

Where RIP is unable to consider the speed of interfaces in determining the best path through the network, OSPF is able to consider a "cost" that is derived by the speed of each interface. However, the formula for determining the cost is not standardized so the default settings may vary.

In all, OSPF is a powerful alternative to RIP, but it makes many demands on router resources and requires more planning. If you're running RIP and it isn't causing you any problems, it may pay to stick with it. But if you want to take advantage of redundant links on your network with a standards-based protocol, OSPF is the way to go.

Enhanced Interior Gateway Routing Protocol

In Cisco shops, there are some advantages to implementing Cisco's EIGRP protocol (enhanced interior gateway routing protocol), the successor to IGRP. For one, it quickly propagates changes in link status, much like OSPF but with less overhead. The biggest disadvantage of EIGRP, however, is that it's not a standard, so it locks you into Cisco products.

Like OSPF, EIGRP routers discover their neighbors and exchange hello packets. EIGRP sends hello packets every five seconds. If three are missed, the neighboring router is considered dead and alternative routes are used. EIGRP also sends incremental updates regarding topology changes on an as-needed basis. And unlike RIP, it doesn't use bandwidth to advertise regular updates.

EIGRP calls the next router in the path toward a network destination a "successor." It also keeps track of the next-hop routers that can provide loop-free backup routes, which it calls "feasible successors." This information is tracked in a topology table. If a route becomes unavailable, the topology table can be quickly consulted for feasible successors. If it finds one, convergence is instantaneous. If none are found, the router will start querying local neighbors for another route, and update its topology table and router table accordingly.

For instance, when a link state changes on a local router, it will recompute its topology table based on the new information. Where OSPF would immediately flood the change in link state to every router on the network, EIGRP will only involve routers that are directly affected by the change. This uses bandwidth as well as router CPU resources more efficiently. Also, EIGRP won't use more than 50% of the available bandwidth, resulting in big gains on low-bandwidth WAN links. Another advantage of EIGRP is that it supports Novell/IPX and AppleTalk environments. This could mean less training for workers in multiprotocol environments.

Multihoming With BGP

If you work for an ISP, you probably have an appreciation of the BGP (Border Gateway Protocol)'s built-in flexibility and protection. BGP was designed for multiple communicating networks with no single administrative entity, so it's tailor-made for the Internet. On the other hand, if you work at an organization with an Internet connection, you may not even have to bother learning how to spell BGP, much less how it works. You can get along by using a default route that points all external traffic to your ISP and let its staff work out the details. However, if you're thinking of adding another Internet connection to a second ISP for backup purposes or load-balancing, you'd better familiarize yourself with BGP. Note: We highly recommend that you work closely with your ISP before experimenting with features that propagate outside your network.

What Is BGP?

BGP is the only widely implemented EGP (Exterior Gateway Protocol) and the only routing protocol used to link networks to one another on the Internet. BGP was first specified in 1989 in RFC 1105. Version 4 was specified in 1994 in RFC 1654 and updated in RFC 1771. There have also been a number of documented extensions. Version 4's most significant contribution is its ability to aggregate advertisements from multiple contiguous routes in one routing-table entry. This is known as CIDR (Classless Interdomain Routing). BGP4 was implemented when big routing tables began overwhelming routers. What CIDR does is protect you from many potential outages and instability on the Internet, and provide great relief for address depletion by more efficiently dividing addresses.

When enabled, BGP4 establishes relationships with adjacent routers, referred to as neighbors. Unlike OSPF (Open Shortest Path First) and EIGRP (Enhanced Interior Gateway Routing Protocol), which will automatically discover routing neighbors, BGP won't exchange routing-table information until both routers have configured one another's IP addresses and ASNs (Autonomous System Numbers) on their corresponding interfaces. Once this is completed, the neighboring routers are considered peers.

Neighboring routers send small "keep-alive" messages to one another. If a neighbor stops receiving keep-alive messages for a predefined "hold time," it will update its routing table to reflect the loss in available routes. BGP also sends incremental updates when routes become unavailable. Otherwise, the full routing tables are exchanged only when two routers first establish or re-establish a peering relationship as described above.

BGP is a Path Vector Protocol, which is similar to a Distance Vector Protocol, but with a key difference. A Distance Vector Protocol chooses routes based on the hop count (or routers traversed) and link speeds. BGP, in contrast, chooses a route that traverses the least number of ASes (Autonomous Systems). As a routing advertisement passes through an AS, it prepends (adjusts the path length advertised) the ASN of the origin AS to the path of other ASes it has traversed. By default, the path with the fewest ASNs is stored in the routing table as the optimal path to a destination network. One AS can contain multiple routers, so it's possible for the actual hop count to be higher than the AS path indicates.

BGP Optimization

With BGP's built-in flexibility, however, you can enhance this default behavior. For instance, you may want to control the path traffic takes as it leaves your network. When peering with multiple neighbors in an external AS, or in different external ASes, there will be multiple paths to the same destination network. By default, BGP determines the optimal path by picking the route that traverses the fewest number of ASes. However, BGP does not take link speed or network load into consideration when computing paths, so the shortest path may not be the optimal one.

You can get around this by using BGP's Local-Pref attribute, which forces BGP to take a particular next-hop route in a scenario with multiple choices. You simply tell the router that all, or even some, of the routes advertised to one of your router interfaces should receive a higher Local-Pref weight than the same routes advertised to another interface. Because Local-Pref is always considered before the computed path distance, the interface you designate with the highest Local-Pref will be chosen as the best route.

Controlling traffic as it comes back into your network is more difficult. With geographically diverse networks, where one ISP connection is a lot closer to one part of the network than to another, you may want to use the MED (multiexit discriminator) attribute, which specifies the path external traffic should use when destined for one of your internal networks. Although the MED attribute is a simple way to control incoming traffic, it will work only if both Internet connections go to the same ISP, since it won't be propagated outside that ISP's AS. Of course, prepending is another way to control incoming traffic.

BGP routing can also be controlled through the community attribute that puts a predefined code on a group or community of routes so the receiving router takes a predefined action based on the value of that code. This code can be user-defined, but the most common is a reserved or well-known community, called No-Export. When a BGP router sees a route come in with the No-Export community, it will not advertise the route outside its own AS. This can be handy for balancing incoming traffic.

The ISP Balancing Act

You'll most likely use BGP if you have multiple connections to the Internet. But if you're interested only in load-balancing, you'll want to stick with one ISP. For redundant ISPs, you'll need two ISPs, and balancing the load will be more difficult.

When you have one ISP and one router at your end, balancing outgoing traffic can be simple, and you have the most control over the paths your packets traverse. In fact, if both interfaces are on the same router at your end, you can even avoid BGP. For example, Cisco Systems gives you the ability to load-balance between two static routes. It's possible that your ISP could do the same for the traffic entering your network.

As we discussed above, you can use the MED attribute to control how traffic enters your network, as long as both connections go to the same AS. If both connections go to the same ISP, there's a very good chance both connections will go to the same AS. For traffic leaving your network, you can also use the Local-Pref attribute to control how traffic leaves your network.

BGP should automatically take care of failing over from one ISP to another and thus provide redundancy with its default settings. However, load-balancing with multiple ISPs can be very challenging. The best approach is to start with the default BGP settings and monitor the transmitted and received traffic. Once you get a baseline, use the Local-Pref attribute to fine-tune traffic leaving your network based on the destination external network or on the AS. There are more than 60,000 networks advertised on the Internet, so target routes with the highest traffic levels. You can always experiment by switching preferred routes from one ISP to another if you get response-time complaints.

You may also need to balance the traffic coming back into your network. One common way to do this is to insert extra AS numbers in the advertisements as discussed above. This works best if you're advertising multiple, internal networks. You can then make the advertisements for one network look better than those for another and force the traffic back into your network via the corresponding ISP pipes.

Another approach is to limit the traffic on one ISP connection to that ISP--and all its customers--and let the rest of your traffic use the other ISP. This works well if much of your traffic rests with the customers of one of your ISPs. The approach also works well if you're using VPNs (virtual private networks) to connect your branch offices or partners on the same ISP. You can use the No-Export community feature to make sure your network isn't advertised beyond that ISP's network. This will take care of incoming traffic. You can also use the Local-Pref attribute to control outgoing traffic by making sure outgoing traffic is restricted by the ISP's ASN.

The drawback to this approach is that you sacrifice some redundancy. If the connection to your ISP that talks to the whole Internet goes down, your other connection can't take over unless you turn off the No-Export option. In this way, the only traffic that will know how to reach your network will still be restricted to that ISP's customers.

BGP provides a lot of control over the destiny of your Internet traffic. But be sure to work closely with your ISP when implementing these options. When you communicate with the Internet that intimately, you must make sure you don't cause any problems. Moreover, if you'll be accepting routes from the Internet, you'll need to invest in lots of memory and CPU for your routers. You should have at least 64 MB available, if you're accepting a full routing table from one ISP. You'll need to double that if you're accepting full routing tables from two ISPs in one router. Also, before you sign up with an ISP, be aware that there's no guarantee the ISP will even advertise a network that's been provided by another ISP. Make sure your new ISP can provide the redundancy and load-balancing you expect.

Anatomy of a Route Map

Below you can find an example configuration of two route maps and their associated commands as they appear in a Cisco router. Route maps associate BGP features or attributes with specific routes. They are applied on a per-neighbor basis (1), where they are referenced by name and told whether to apply the map to incoming (in) or outgoing (out) routing updates.

There can also be a series of route map statements (3) that are identified by their route map name. The statements include line numbers that determine the order in which those statements are read. Each statement has a condition established by the "match" command and/or an action established by the "set" command. The match command can refer to an access list (2) to identify a route based on its network address. A special "AS path access list" can be used to make a match based on the AS number. The AS path access list can use regular expressions to search for patterns.

Route map SET-LOCAL-PREF, highlighted in red, is applied to routing updates coming in from 10.1.4.2. Line 10 looks to the AS path access list to find updates coming from AS 20. If a match is found, a Local Preference of 300 is applied. If no match is found, however, the process proceeds to Line 20, which applies a Local Preference of 300 to the remaining routes.

Two route Maps and all their associated commands.

The NON-TRANSIT-FILTER route map, highlighted in blue, refers to access List 20 to permit networks that start with 172.25 (the internal network) and deny everything else. It prevents Internet advertisements from being readvertised back to the peer at the ISP.

Preparing for Multihoming

Step 1: Registration: If you want to participate in BGP routing on the Internet, you need to apply for a unique ASN (autonomous system number) at www.arin.net. Processing takes a few weeks and costs $500, plus an annual fee.

Step 2: Design: It's important for you to give some thought to how you'll integrate BGP with the rest of your network. Keep in mind that you'll want to run BGP only at the edge of your network. If you have multiple routers that are connected to the Internet, you can run BGP between them. This is commonly known as iBGP. However, whenever possible, you'll want to isolate your network from the large routing tables and consequently the instability that's common to the Internet. Instead, use default routes that send traffic destined for the Internet to one of your routers running BGP. To direct external traffic from your BGP routers back into your network, run your IGP (interior gateway protocol) on your BGP routers. Just don't redistribute the BGP routes into your IGP.

Step 3: Configuration: BGP is enabled with the "router" command, which references the ASN. The recommended "no synchronization" command lets BGP advertise the internal network without having to wait to hear updates from the internal network. The "network" command indicates which networks on the router are participating in BGP. The "neighbor" command establishes peering with a router in another AS. It's also used to activate other BGP options that are relevant to that neighbor.

Step 4: Basic BGP configuration on Cisco router:

What About Quality of Service?

One reason to add multiple Internet connections is to distribute load. If you think you can avoid this as long as you ensure that your most important traffic gets through, think again.

You can set up such assurances on a per-interface basis for traffic leaving your network. For example, with Cisco routers you can set up a "priority list" that will give preferences to different applications based on requested TCP ports. Other vendors have similar features in their equipment.

Traffic entering your network is another story. Your ISP controls such traffic, and unfortunately, most ISPs don't offer such services. But there are a few notable exceptions. Concentric Network Corp. (www.concentric.com) has announced it will provide varying levels of Quality of Service (QoS) with its VPN service. And Concert (www.concert.com), the AT&T and BT joint venture, has announced it will provide three priority levels using the emerging DiffServ standard for classifying packets based on the first six bits of the ToS (type of service) field found in the IP header.

DiffServ, outlined in RFC 2474 and 2475, is backward-compatible with an earlier scheme that uses the first three bits of the ToS field and offers 64 levels of service. However, DiffServ won't guarantee bandwidth. Another standard, RSVP (Resource Reservation Protocol), attempts to reserve a specific level of bandwidth from end to end. If the bandwidth is unavailable, it tries again later. RSVP's advantage is that once the bandwidth is reserved, there's no danger additional traffic will cause problems, because it will be simply refused. On the down side, RSVP doesn't scale very well with a large number of connections because the participating routers must set up and track the state of every connection. DiffServ probably will become the preferred ISP QoS implementation because it can aggregate all connections into the predefined levels of QoS.

Unfortunately, if you think it's taking a long time for ISPs to step up to the plate with QoS services, it's going to take a lot longer to get end-to-end QoS when crossing ISPs. The services mentioned above only will work for traffic that is kept within an ISP's network. An ISP cannot map QoS levels to another ISP and guarantee that they will be followed.

Questions or problems regarding this web site should be directed to Michael A.
Copyright © 2009 Compulink Business Systems. All rights reserved.
Last modified: 01/07/10.