Content 7.1 Troubleshooting the
Application Layer 7.1.4 Isolating application
layer problems Even though there may be IP connectivity
between a source and a destination, problems may still exist
for a specific upper-layer protocol such as FTP, HTTP, or
Telnet. These protocols ride on top of the basic IP transport
but are subject to protocol specific problems relating to
packet filters and firewalls. It is possible that everything
except mail will work between a given source and destination.
Before troubleshooting at this level, it is important to
establish whether IP connectivity exists between the source and
the destination. If IP connectivity exists, then the issue must
be at the application layer. The following list outlines
possible issues: - A packet filter/firewall issue might
have arisen for the specific protocol, data connection, or
return traffic.
- The specific service could be down on
the server.
- An authentication problem might have
occurred on the server for the source or source network.
- There could be a version mismatch or incompatibility with
the client and server software.
Troubleshooting an
upper-layer protocol connectivity problem requires
understanding the process of the protocol. This information is
usually found in the latest RFC for the protocol or on the
developer web page. Questions that should be answered to make
certain the functions of the protocol are understood include
the following: - What IP protocols does the protocol use
(TCP, UDP, ICMP, IGMP)?
- What TCP or UDP port numbers
are used by the protocol?
- Does the protocol require
any inbound TCP connections or inbound UDP packets?
- Does the protocol embed IP addresses in the data portion of
the packet?
- Are the protocols being used on a client
or a server?
If the protocol embeds IP addresses in
the data portion of the packet and NAT has been configured
anywhere along the path of the packet, the NAT gateway will
need to know how to deal with that particular protocol or the
connection will fail. NAT gateways typically change information
in the data portion of a packet only when they have been
specifically coded to do so. Some examples of protocols that
embed IP addresses in the data portion of the packet are FTP,
SQLNet, and Microsoft WINS. If there is a question regarding
whether a firewall or router is interfering with the flow of
data for a particular application or protocol, several steps
can be taken to see what exactly is happening. These steps may
not all be possible in every situation. - Move the
client outside the firewall or address translation device.
- Verify whether the client can connect to a server on
the same subnet as the client.
- Capture a network
trace at the client LAN and on the LAN closest to the server or
preferably, on the server LAN.
- If the service is
ASCII based, telnet to the port of the service from the router
closest to the server, then work backward into the network
toward the client.
Content 7.1
Troubleshooting the Application Layer 7.1.5
Identifying support resources Some application problems can
be resolved by reading technical documentation at the software
vendor or developer’s website. These sites also have patches
and version updates that a troubleshooter can download to
repair bugs or incompatibilities. When troubleshooting network
problems, network administrators must know where to find
information. Good sources of information include:
- Standard organizations
- Technical forums
- Cisco Technical Assistance Center
- Discussion
groups
Content 7.1 Troubleshooting
the Application Layer 7.1.6 Accessing support
resources In most cases, network problems can be resolved
without assistance from any outside technical support. However,
some problems may seem to be too elusive and professional help
is required. This is when Cisco Systems Technical Assistance
Center (TAC) should be utilized. It is suggested the following
be completed before calling Cisco (TAC): - Have the
service contract number ready. TAC will ask for it.
- Have a diagram of the network, or the affected portion of
the network. Make sure all IP addresses and associated network
masks or prefix lengths are listed.
- List the steps
already taken and their results compiled for the TAC engineer.
- If the problem appears to be with only a few routers
(fewer than four), capture the output from show tech
command from these routers.
Dial-in or Telnet
access also helps considerably in effective problem resolution.
Content 7.1 Troubleshooting the Application
Layer 7.1.7 Correcting application layer
problems Figure shows the steps for correcting application
layer problems. Use the following steps: - Make a
backup. Before proceeding, ensure that a valid configuration
has been saved for any device on which the configuration may be
modified. This provides for recovery to a known initial state.
- Make initial hardware and software configuration
changes. If the correction requires more than one change, make
only one change at a time.
- Evaluate and document the
change and the results of each change. If the results of any
problem-solving steps are unsuccessful, immediately undo the
changes. If the problem is intermittent, wait to see if the
problem occurs again before evaluating the effect of any
change.
- Verify that the change actually fixed the
problem without introducing any new problems. The network
should be returned to the baseline operation and no new or old
symptoms should be present. If the problem is not solved, undo
all the changes. If new or additional problems are discovered,
modify the correction plan.
- Stop making changes when
the original problem appears to be solved.
- If
necessary, get input from outside resources. This may be a
coworker, consultant, or Cisco Technical Assistance Center
(TAC). On rare occasions a core dump may be necessary, which
creates output that a specialist at Cisco Systems can analyze.
- Once the problem is resolved, document the solution.
Content 7.2 Gathering Information
on Application Layer Problems 7.2.1
Overview To make quick and accurate troubleshooting
decisions, a network administrator must be able to get the
right information at the right time. There are several tools
available to help in this troubleshooting process. However, the
best time to learn about these tools is not when a problem is
encountered. The best time to explore and learn these tools is
when the network is functioning correctly. This way network
baselines can be established and recorded. When problems occur,
administrators should refer to the normal baseline to identify
inconsistencies more quickly. In short, an administrator must
not only know about the tools, but they must also be able to
recognize and decipher the pertinent information provided by
the various tools. An administrator should be fluent with all
the following tools: - Command line (UNIX, DOS, Cisco
IOS)
- Windows, UNIX, IOS utilities
- Protocol
Analyzers
- Network Management Systems
- System
logs
The following section highlights some of the
tools available and their typical use. It is assumed that the
reader is somewhat proficient with most of these tools. Lab
Activity Lab Exercise: Troubleshooting Problems at All
Logical Layers In this lab, the student will use a
troubleshooting methodology and Cisco commands to define,
isolate, and correct issues.
Content 7.2
Gathering Information on Application Layer Problems
7.2.2 Common TCP/IP commands The TCP/IP protocol
suite offers several commands to help troubleshoot Application
Layer problems. Most of these commands should be very familiar
while others may be new. Take time to fully understand and