Content 7.1 Troubleshooting the Application Layer 7.1.4 Isolating application layer problems Even though there may be IP connectivity between a source and a destination, problems may still exist for a specific upper-layer protocol such as FTP, HTTP, or Telnet. These protocols ride on top of the basic IP transport but are subject to protocol specific problems relating to packet filters and firewalls. It is possible that everything except mail will work between a given source and destination. Before troubleshooting at this level, it is important to establish whether IP connectivity exists between the source and the destination. If IP connectivity exists, then the issue must be at the application layer. The following list outlines possible issues:

A packet filter/firewall issue might have arisen for the specific protocol, data connection, or return traffic.
The specific service could be down on the server.
An authentication problem might have occurred on the server for the source or source network.
There could be a version mismatch or incompatibility with the client and server software.

Troubleshooting an upper-layer protocol connectivity problem requires understanding the process of the protocol. This information is usually found in the latest RFC for the protocol or on the developer web page. Questions that should be answered to make certain the functions of the protocol are understood include the following:

What IP protocols does the protocol use (TCP, UDP, ICMP, IGMP)?
What TCP or UDP port numbers are used by the protocol?
Does the protocol require any inbound TCP connections or inbound UDP packets?
Does the protocol embed IP addresses in the data portion of the packet?
Are the protocols being used on a client or a server?

If the protocol embeds IP addresses in the data portion of the packet and NAT has been configured anywhere along the path of the packet, the NAT gateway will need to know how to deal with that particular protocol or the connection will fail. NAT gateways typically change information in the data portion of a packet only when they have been specifically coded to do so. Some examples of protocols that embed IP addresses in the data portion of the packet are FTP, SQLNet, and Microsoft WINS. If there is a question regarding whether a firewall or router is interfering with the flow of data for a particular application or protocol, several steps can be taken to see what exactly is happening. These steps may not all be possible in every situation.

Move the client outside the firewall or address translation device.
Verify whether the client can connect to a server on the same subnet as the client.
Capture a network trace at the client LAN and on the LAN closest to the server or preferably, on the server LAN.
If the service is ASCII based, telnet to the port of the service from the router closest to the server, then work backward into the network toward the client.

Content 7.1 Troubleshooting the Application Layer 7.1.5 Identifying support resources Some application problems can be resolved by reading technical documentation at the software vendor or developer’s website. These sites also have patches and version updates that a troubleshooter can download to repair bugs or incompatibilities. When troubleshooting network problems, network administrators must know where to find information. Good sources of information include:

Standard organizations
Technical forums
Cisco Technical Assistance Center
Discussion groups

Content 7.1 Troubleshooting the Application Layer 7.1.6 Accessing support resources In most cases, network problems can be resolved without assistance from any outside technical support. However, some problems may seem to be too elusive and professional help is required. This is when Cisco Systems Technical Assistance Center (TAC) should be utilized. It is suggested the following be completed before calling Cisco (TAC):

Have the service contract number ready. TAC will ask for it.
Have a diagram of the network, or the affected portion of the network. Make sure all IP addresses and associated network masks or prefix lengths are listed.
List the steps already taken and their results compiled for the TAC engineer.
If the problem appears to be with only a few routers (fewer than four), capture the output from show tech command from these routers.

Dial-in or Telnet access also helps considerably in effective problem resolution.

Content 7.1 Troubleshooting the Application Layer 7.1.7 Correcting application layer problems Figure shows the steps for correcting application layer problems. Use the following steps:

Make a backup. Before proceeding, ensure that a valid configuration has been saved for any device on which the configuration may be modified. This provides for recovery to a known initial state.
Make initial hardware and software configuration changes. If the correction requires more than one change, make only one change at a time.
Evaluate and document the change and the results of each change. If the results of any problem-solving steps are unsuccessful, immediately undo the changes. If the problem is intermittent, wait to see if the problem occurs again before evaluating the effect of any change.
Verify that the change actually fixed the problem without introducing any new problems. The network should be returned to the baseline operation and no new or old symptoms should be present. If the problem is not solved, undo all the changes. If new or additional problems are discovered, modify the correction plan.
Stop making changes when the original problem appears to be solved.
If necessary, get input from outside resources. This may be a coworker, consultant, or Cisco Technical Assistance Center (TAC). On rare occasions a core dump may be necessary, which creates output that a specialist at Cisco Systems can analyze.
Once the problem is resolved, document the solution.

Content 7.2 Gathering Information on Application Layer Problems 7.2.1 Overview To make quick and accurate troubleshooting decisions, a network administrator must be able to get the right information at the right time. There are several tools available to help in this troubleshooting process. However, the best time to learn about these tools is not when a problem is encountered. The best time to explore and learn these tools is when the network is functioning correctly. This way network baselines can be established and recorded. When problems occur, administrators should refer to the normal baseline to identify inconsistencies more quickly. In short, an administrator must not only know about the tools, but they must also be able to recognize and decipher the pertinent information provided by the various tools. An administrator should be fluent with all the following tools:

Command line (UNIX, DOS, Cisco IOS)
Windows, UNIX, IOS utilities
Protocol Analyzers
Network Management Systems
System logs

The following section highlights some of the tools available and their typical use. It is assumed that the reader is somewhat proficient with most of these tools. Lab Activity Lab Exercise: Troubleshooting Problems at All Logical Layers In this lab, the student will use a troubleshooting methodology and Cisco commands to define, isolate, and correct issues.

Content 7.2 Gathering Information on Application Layer Problems 7.2.2 Common TCP/IP commands The TCP/IP protocol suite offers several commands to help troubleshoot Application Layer problems. Most of these commands should be very familiar while others may be new. Take time to fully understand and