ITIL – IT Infrastructure Library – Best practice for IT service management.

FCAPS – Fault, Configuration, Accounting, Performance and security – ISO model for network maintenance.

TMN – Telecommunication Management Network – ITU-T adapted model of FCAPS.

PPDIOO – Cisco method. Prepare, plan, design, implement, operate, optimize.

**********************************************************************************

- If we have already provided username/password for FTP connections in the configuration, (using “ip ftp username/password” or “ip http client {username | password}“) we don’t need to specify credentials in “copy start ftp” command.

- HTTPs and SCP are more secure than HTTP/FTP which sends password in clear text.

**********************************************************************************

Archive configuration:

archive

                  path <flash>

                  write-memory

                  time-period <minutes>

- Configuration Rollback: using “configure replace <path which has new running-config>  list“

**********************************************************************************

NTP: “ntp server <ip>”

Daylight saving: “clock summer-time xxxx”

Logging: “logging console <>”. By default level-7(debug) to level-0(emergency) are stored.

“logging <ip>”. BY default all messages except level-7 are sent to Syslog server.

**********************************************************************************

Structured troubleshooting: Define problem, gather information, analyse, *eliminate*, propose hypothesis, test hypothesis, resolve problem.

“Shoot from the Hip”: Define problem, gather information, propose hypothesis, test hypothesis, resolve problem.

Approaches:

1. Top down
2. Bottom up
3. Divide and conquer
4. Follow the path
5. Spot the differences
6. Move the problem/components.

**********************************************************************************

Integrating Troubleshooting into Maintenance:

• Update-to-date documentation.
• Creating baseline for network performance. How much CPU% is normal?
• Has anything changed recently before the problem occurred

**********************************************************************************

• Use output filtering to identify problem. Eg: use “show ip route 1.1.1.1″ instead of “show ip route”. Note Default route is not displayed when using show ip route filtering. Or use “show ip route 1.1.1.0 255.255.255.0 longer-prefix” to see all routes above 1.1.1.0/24 networks.
• Filter show command outputs using {begin|include|exclude| section}.. Also we can use regular expression like “show ip route | include ^C” to show only connected routes.
• To redirect output to a file in flash/tftp, use “| tee” or “|redirect” command. Tee will display the output in screen and copy the content to remote file. ‘Redirect’ option will only copy the content to remote file.
• To append outputs to an existing file,use “| append” keyword.
• Extended ping for options like DF bit,repeat,source, size etc
• To check whether a TCP port is open use “telnet <IP> <port number>”. Eg: “telnet 1.1.1.1 80″ will check whether port 80(HTTP) is opened in 1.1.1.1
• Hardware diag commands: show inventory, show controllers, show platform, show diag, GOLD (Generic OnLine Diag), TDR (Time Domain Reflectometer)

**********************************************************************************

Following process in TS steps, will be benefited by the use of tool:

•  Define problem: syslog, event triggered via SNMP, EEM.
• Gather information: SPAN/RSPAN/ERSPAN. RSPAN cannot cross L3 boundry. Only pass via trunks.
• Analyze: by SNMP statistics or NetFlow usage. Netflow enabled via “ip flow ingress/egress” command. Replaces old “ip route-cache flow” command. “show ip cache flow” to check active flow pass via router. “ip flow-export” command to export the data from router to collector.
•  Test Hypothesis: configuration replace or rollback.

**********************************************************************************

Layer-2 forwarding checks:

show mac-address-table

show vlan

show interfaces switchport

show interfaces trunk

show platform forward  <<<< Info from TCAM.

traceroute mac <<<< traceroute to an MAC address, provided CDP is enabled.

**********************************************************************************

STP Check:

Note: Cost of the incoming interface is added, when forwarding the BPDU.

If there is any loop, only packets with DST MAC not available in the mac-address-table will be in loop.

show spanning-tree vlan xxx

show spanning-tree interface xxxxxxx

**********************************************************************************

• Difference between Router and Multilayer switches:
  • Routers support many interface/media types. Switches support almost only Ethernet.
  • Packet-switching throughput of router is less than MLS.
  • Routers support many features compared to MLS.
• Control plane troubleshooting is same for both routers and Multilayer switches.
• Date plane troubleshooting differs. “show ip cef, show adjacency, show arp” on Router. “show platform, show mls cef” on Multilayer switch.
• MLS can perform
  • Switching between VLANs… “vlan x” to create VLAN database.
  • Routing between VLANs… Using SVI. “int vlan x” and “ip routing” command required.
  • Routing between VLANs and Outer world:
    • Using SVI
    • Using “no switchport” on the interface and thereby making it as “Routing” port.
• On the routers, subinterfaces are used to connect to the downstream switches via trunk interface. Make sure the native VLAN is matched on both ends.

!

interface GigabitEthernet2/3.10

encapsulation dot1Q 10

ip address 10.10.10.254 255.255.255.0

no ip redirects

end

**********************************************************************************

Troubleshooting FHRP:

HSRP:

-  When the current active routers’ failure is by administrative action (by shutting down interface/reload of router), it sends ‘resign’ message that cause standby router to immediately take over the master ship. NO packet loss for 10 sec hold timer.

- If we add a router with priority greater than the current ‘active’ router and if pre-empt if enabled, this router will send “coup” message and takes over the mastership.

- Always recommended to have some delay before taking mastership so that active router converge its routing protocol and loads its RIB.

VRRP:

- Hello interval is 1 sec. Premption enabled by default. IETF standard. VIP can be same as interface IP.

-show {vrrp | glbp | hsrp } brief

**********************************************************************************

Troubleshooting Performance problems in MLS:

-User expectation, Business expectation and technical expectations are used as baseline to identify performance problem.

-Typical scenarios: Duplex mismatch, TCAM limitations, high CPU load.

**********************************************************************************

-To identify packet loss at interface level use: “show int xxx counters” and “show int xxxx counters errors”

• Align-Err:  Frames which do not have even number of octets and have CRC errors. Possible bad port, cable.
• FCS-Err:  Valid size but with Frame check sequence errors. Possible bad cable, NIC. Increases on the Full-duplex end, in case of duplex mismatch.
• Xmit-Err/Rcv-Err: Internal Tx and Rx buffers are full.
• UnderSize: Frames with less than 64 bytes but with Valid CRC.
• Single-Col/Multi-Col: Counters to indicate single/multiple collision occurs before transmitting a frame. Duplex-mismatch.
• Late-Col: Collision detected on a port late in transmission process. Possible Reasons: Duplex-mismatch or Ethernet cable that is too long. Increases on the half-duplex end, in case of duplex mismatch.
• Excess-Col: Packets dropped due to excessive collision. When a packet has collision for 16 times, this counter is incremented and the packet is dropped.
• Carri-Sen: Normal on half-duplex. When controller senses the wire and checks if it not busy.

– Runts: Frames less than 64 bytes and with bad CRC. Because of Bad cable or port.

– Giants: Frames Frames greater than 1518 with bad CRC. Reason: Mostly by bad NIC.

– IEEE 802.3 Frame size: 64 bytes to 1518 for non-jumbo Ethernet.

– Running half-duplex is better than duplex-mismatch.

**********************************************************************************

-Auto-MDIX:  Automatic Media-dependent interface crossover. It used to automatically detect whether straight/cross-over cable is required and automatically configures the interface.

-If speed and duplex auto-negotiation is disabled, then MDIX will also be disabled.”mdix auto” to enable MDIX. “show interfaces xxx transceiver properties” to check the MDIX.

**********************************************************************************

Troubleshooting TCAM problems:

• ‘Decision making logic’ in ‘forwarding hardware’ uses high-performance lookup memory called TCAM.
• Control-plane information like MAC-address table, routing table, PBR, QOS and ACL are programmed in TCAM.
• Frames cannot forward by TCAM, will be punt to CPU. If there is more packets punt to CPU, then it might affect throughput.
• Reasons for TCAM to punt packets to CPU:
  • Packets destined to CPU like telnet/SSH/SNMP.
  • MC/BC protocol control packets which sent to CPU in-addition to flooding.
  • If a feature is not supported. Eg: GRE.
• Due to TCAM limitations, some entries may not be written and hence ‘soft-forwarding’ is performed.
• To verify TCAM space on Cat 3560, use “show platform tcam utilization”. Check used/max column.
• TCAM space allocation depends on Switch database Manager (SDM) profile. [ Similar to CAM-profile in Force10]
• To check TCAM allocation failures for prefixes of specific length, use “show platform ip unicast counts”
• “sw forwarding” count in the output of “show controllers cpu-interface” indicates the number of packets punt to CPU.
• To avoid above problems: Perform Route summarization so that number of prefix fit the TCAM space. OR change SDM profile.

**********************************************************************************

Troubleshooting High CPU load:

• On switches, packets are switched by hardware. Hence CPU load and traffic load are not related. In low-range routers, there exists direct relationship.
• Check CPU process using “show process cpu sorted”. “CPU utilization for five seconds: x%/%y”. X represents time spent on process & interrupts. Y represents time spent on only interrupts.
• On switches, if the cycles spent by CPU for interrupts are more than 10%, we may need to investigate the cause as this implies CPU does packet-forwarding. (Interrupts – punt by TCAM)

**********************************************************************************

Troubleshooting Layer-3 connectivity:

• HDLC does not require any L3-L2 mapping.
• Data Structures used for Routing: RIB, FIB (doesnt have Protocol information), L3-L2 mapping table and Cisco Express Forwarding adjacency table.
• “show ip route” for RIB and “show ip cef {exact-route SRC DST}” for FIB table. “exact-route” option to identify which path the router will select in case of equal-cost load-balancing.
• “show ip arp / show frame map” for L3-L2 mapping table and “show adjacency detail” for adjacency table.

**********************************************************************************

Troubleshooting EIGRP:

• Data Structures has interface table (active interfaces list), neighbor table (all active EIGRP neighbors), topology table(to save all received routes)
• Note: Only successors are advertised to neighbors.  “show ip eigrp { interface | neighbor | topology}” . “debug eigrp packets” or “debug ip eigrp”. There is no “debug ip eigrp packet” command!
• “Neighbor not on common subnet” log message will be displayed if the received SRC IP of EIGRP hello is different from the configured network statement.

**********************************************************************************

Troubleshooting OSPF:

• Data Structures has interface table (active interfaces list), neighbor table (all active OSPF neighbors), LSDB- Link state database.
• “show ip ospf { interface | neighbor | database}” . “debug ip ospf {packets | events}”

**********************************************************************************

Troubleshooting Route Redistribution:

• If OSPF is redistributed into EIGRP, all OSPF routes in routing table in-addition to the ospf enabled interface networks are added into EIGRP topology table.
• For RIP, EIGRP, the default seed metric is “Unreachable”. Need to specify the metric for redistribution to happen.
• “show ip route x.x.x.x y.y.y.y” should show both “Advertised via OSFP/EIGRP” and “Advertised by OSPF/EIGRP”. If latter is not displayed, it means redistribution has been configured but the routes are not redistributed as expected.
• Add “subnet” option when you redistribute from any protocols to OSPF.
• Interface command to change the OSPF timers to default values: “default ip ospf hello-interval”.
• “debug ip ospf events” and “debug ip ospf adja” commands useful to identify why neighborship was failed to form.

**********************************************************************************

Troubleshooting BGP:

• Data Structure: Neighbor table (show ip bgp summary/neighbors) and BGP table (show ip bgp)
• Split-horizon is enabled. A router will not advertise a path back to the originating router, which it has selected as best path.
• Check “Advertised to update-group” in the output of show ip bgp, to verify whether we advertise BGP routes to peers.
• To check neighbor-> update-group relation, use “show ip bgp update-group”
• “debug ip bgp” to check adjacency state. No such command as “debug ip bgp adja”

**********************************************************************************

Troubleshooting Performance Problems on Routers:

• Some process responsible for high CPU: Arp input process (originating ARP), net background process(When an interface needs some buffer but nor available, this process create buffers from main buffer pool), IP backgroup process (to change interface status, IP, encapsulation), TCP timer process (TCP sessions on Router)
• “Show processes cpu history”. Graphical view about CPU usage for last 60 secs/60 min/72 hrs.
• “Show ip interface xxx” details about which switching method is enabled on that interface. “ip route-cache” enables Fast-switching (for unicast).
• CEF should be enabled for NBAR, Auto-Qos, MQC-FRTS, MPLS, QOS, CBWRED and other features to work. “ip route-cache cef” on interface mode.
• “Show ip cache” to display fast-switching cache. “show ip cef” to display CEF entries. “show {ip} cef adjacency <interface> <NH>” to see adjacency table.
• “show memory allocating-process totals” to check memory allocated to each process, Total/used memory. To check any memory leak: “Show process memory”
• When we notice “Input queue: 76/75(current/max)” in the output of “show interface”, it indicates memory leak problem. (Wedged Interface). Incoming packets are dropped on this interface. IOS bug. Upgrade/reload is the solutions.
• “Show buffers” might give clue about memory leak. “Free list” count will be very less compared to “total” count.
• “show diag” command to display DRAM memory available on routers’ linecard.

**********************************************************************************

Troubleshooting Security Features:

• Options to secure management plane, control plane and data plane.
• On some devices, “no service password-recovery” command is available which blocks successful recovery of configuration and password.
• Cisco IOS firewall. “ip inspect name <> tcp/udp/http” and then apply “ip inspect <> out” interface connecting to internet. “show ip inspect all”, “debug ip inspect”
• Zone based Firewall: zones are created and matched packets from one zone to another can be denied/passed/inspect. “show zone security”, “show zone-pair”
• AAA: “debug tacacs”, “debug aaa accounting”
• TACACS: Separates AAA as each process. Uses TCP. Encrypt entire packet. Limited accounting. It was developed by Cisco.
• RADIUS: Combines authentication and authorization. Uses UDP. Encrypt only password. Extensive accounting. Open Standard.
• Cisco Secure access Control System (ACS): Web-based GUI to authenticate users via AAA.
• RSA key can be generated using “crypto key generate rsa”. Need to save configuration to write this key in NVRAM. Can’t see in running config. Key is generated using hostname and domain name.

• Create “ifcfg-bond0” file in /etc/sysconfig/network-scripts and configure IP address, network mask and other port-channel parameters.

[root@server2 network-scripts]# vi ifcfg-bond0
[root@server2 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.10.1
NETWORK=192.168.10.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

• Configure individual interface such that “MASTER” is set to “bond0” and “SLAVE” is set.

[root@server2 network-scripts]# vim /etc/sysconfig/network-scripts/ifcfg-eth4
[root@server2 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth4
# Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
DEVICE=eth4
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[root@server2 network-scripts]#

• Make sure bonding module is loaded. If not add below two lines in /etc/modprobe.conf file. ”mode” determines bonding policy. By default it is “balance-rr” (round robin). For LACP configure mode as “802.3ad”. For active-backup configure mode as “active-backup”

[root@server2 network-scripts]# cat /etc/modprobe.conf | grep bond0
alias bond0 bonding
options bond0 miimon=100 mode=802.3ad
[root@server2 network-scripts]#

• Load the bonding module using “modprobe bonding” command if it is not already loaded. Restart the network service using “service network restart”. Check with “ifconfig bond0” or “cat /proc/net/bonding/bond0” command.

[root@server2 network-scripts]# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:18:FE:2E:36:6D
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::218:feff:fe2e:366d/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:89 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2572 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10920 (10.6 KiB)  TX bytes:315684 (308.2 KiB)
[root@server2 network-scripts]#

<on the other end- Force10 switch>

Force10#show int po brief

Codes: L – LACP Port-channel

LAG  Mode  Status       Uptime      Ports

L   1    L3    up           00:19:22    Gi 4/4     (Up)

Gi 4/5     (Up)

Force10#

<With active-backup mode. eth4/eth5 are bundled together. eth5 is primary active>

 [root@server2 ~]# cat /etc/modprobe.conf | grep bond
alias bond0 bonding
options bond0 miimon=100 mode=1 primary=eth5
[root@server2 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth5
Currently Active Slave: eth5
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth4
MII Status: up
Link Failure Count: 2
Permanent HW addr: 00:18:fe:2e:36:6d
Slave Interface: eth5
MII Status: up
Link Failure Count: 8
Permanent HW addr: 00:18:fe:2e:36:68

modprobe <module-name>   : to add this module

lsmod : to list loaded module

rmmod <module-name> : to remove a module

• Network devices may need an NTP server to sync its clock; so that all devices in the network display same time…We can configure a Linux host as an NTP server.
• Check whether NTP daemon is installed in Linux: [Tested in RedHat-3.4.6]

[root@server2 ~]# rpm -qa | grep ntp-
ntp-4.2.0.a.20040617-6.el4
[root@server2 ~]#

Procedure:

First we need to sync the linux host with stratum-0 or 1 NTP public servers (with NTP daemon disabled-to use ntp port: UDP 123). Once it is sync-ed, we can start the NTP daemon and it will respond to sync-request from local network devices.

• Check whether ntpd is disabled – Only then we can open UDP:123 port to sync with public NTP servers.

Use “service ntpd status” or “pgrep ntpd” or “netstat -a | grep ntp”. Last 2 commands will not have any outputs if NTP daemon is disabled.

• Check Public NTP servers are configured in “etc/ntp.conf”

[root@server2 ~]# cat /etc/ntp.conf | grep ^server
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server  127.127.1.0     # local clock
[root@server2 ~]#

• Even if we start the NTP service now, the linux host will not sync with public NTP servers if the difference between local time and NTP server time is huge. Hence manually force the linux host to update its time as given by NTP server. (NTP daemon has be disabled for the port to open)

[root@server2 ~]#date -s "Jan 01 00:00:00"
[root@server2 ~]# date
Sat Jan  1 00:00:18 PST 2011
[root@server2 ~]#
[root@server2 ~]# ntpdate 0.us.pool.ntp.org
28 Aug 13:17:46 ntpdate[27387]: step time server 74.118.152.85 offset 20693378.740392 sec
[root@server2 ~]# date
Sun Aug 28 13:17:48 PDT 2011
[root@server2 ~]#

We can see NTP logs in “/var/log/messages” folder.

• Now we can start the ntpd process using “service ntpd start” so that local linux host can act as NTP server for routers. Check whether the linux host is in sync with public NTP servers using “ntpq –p” command.
• “tcpdump –vv  -i eth0 port 123” to check incoming/outgoing NTP packets.
• Configuration on network device. In this example it is Force10 router;

Force10#show run ntp
!
ntp server 10.11.131.220
ntp update-calendar
Force10#
Force10#show clock detail
13:44:59.997 PDT Sun Aug 28 2011
Time source is NTP
Force10#show ntp status
Clock is synchronized, stratum 3, reference is 10.11.131.220
frequency is -64.000 ppm, stability is 407.836 ppm, precision is 4294967283
reference time is D2052746.38463000 (20:44:54.219 UTC Sun Aug 28 2011)
clock offset is -50.780340 msec, root delay is 0.08142 sec
root dispersion is 0.33315 sec, peer dispersion is 278.152 msec
peer mode is client

Note: NTP status will show clock sync-ed only after some minutes.

<NTP debug on Force10 when NTP sync occurs>

3d0h35m : NTP: rcv packet from 10.11.131.220

leap 3, mode 4, version 3, stratum 0, ppoll 16

rtdel 0000 (0.000000), rtdsp 00A2 (2.471924), refid 494E4954 (73.78.73.84)

ref 00000000.00000000 (6:28:16.000 UTC Thu Feb 7 2036)

org D20529E2.F6AAA000 (20:56:2.963 UTC Sun Aug 28 2011)

rec D20529E2.E71550CA (20:56:2.902 UTC Sun Aug 28 2011)

xmt D20529E2.E715F88F (20:56:2.902 UTC Sun Aug 28 2011)

inp D20529E2.F6D8D000 (20:56:2.964 UTC Sun Aug 28 2011)

3d0h35m : ntpDebug: receive: pkt 48 bytes

3d0h35m : ntpDebug: receive: his mode 4

Aug 28 20:56:03: %RPM0-P:CP %NTP-6-INVALID PKT: xntpd task, invalid pkt received, bad header

3d0h37m : NTP: rcv packet from 10.11.131.220

leap 0, mode 4, version 3, stratum 2, ppoll 16

rtdel 14BD (81.008911), rtdsp 10C6B (1048.507690), refid 44000E4C (68.0.14.76)

ref D20529FF.3F0E3CD9 (20:56:31.246 UTC Sun Aug 28 2011)

org D2052A43.BBA9F000 (20:57:39.733 UTC Sun Aug 28 2011)

rec D2052A43.A25ABC0E (20:57:39.634 UTC Sun Aug 28 2011)

xmt D2052A43.A25B530C (20:57:39.634 UTC Sun Aug 28 2011)

inp D2052A43.BBD74000 (20:57:39.733 UTC Sun Aug 28 2011)

3d0h37m : ntpDebug: receive: pkt 48 bytes

3d0h37m : ntpDebug: receive: his mode 4

3d0h37m : ntpDebug: clock_update(10.11.131.220)

3d0h37m : ntpDebug: synchronized to 10.11.131.220, stratum=2

References:

Public NTP servers: http://support.ntp.org/bin/view/Servers/NTPPoolServers

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch24_:_The_NTP_Server

http://linux.die.net/man/1/ntpd

Format: “tcpdump <options> <expressions>”

· By default, ‘tcpdump’ will dump traffic on first found lowest numbered interface from the list and continues to dump packets unless a stop signal is issued. (Ctr+C)

· Display all available interfaces using “tcpdump –D”

Most used options are:

-c <num> : Count option. To limit the number of packets captured by the filter

-i <iface> : to specify the Interface

-n : to suppress the Name conversion from well know addresses/DNS resolved hosts.

-q : Quick output without much protocol information.

-v / -vv : to display in Verbose

-t : without Timestamp

-r : to Read from a file

-w : to Write to a file

-e: to display link-layer (L2) Ethernet header in each line.

-A : to display full packet content in ASCII.

Expression Format:

{protocol} – {direction} – {type: with logical expression}

· Protocol: can be ether, ip, ip6, arp, tcp, udp etc

o If none specified, all protocols are included.

· Direction: src, dst, src or dst and src and dst

o If none specified, src or dst is assumed.

· Type: can be host, net , port and portrange

o If none specified, host is assumed

· Logical expressions: not, and, or

o not gets higher precedence. Both and,or take same precedence evaluating from left to right.

 Options available are:

o [src | dst] host <host> : To match src or destination with IP address matched ‘host’

o ether src [dst] <ehost> : To match src[destination] with mac-address same as ‘ehost’

o ether host <ehost> : to match either src or destination same as ‘ehost’.

o portrange port1-port2 : to match any (tcp or udp) ports in range port1 to port2

o less <length> : to match packets <= ‘length’

o greater <length> : to match packets >= ‘length’

o ip proto <protocol> : to match specific protocols/numbers. Some options are icmp,igmp,pim,vrrp,tcp,udp etc.

o ether proto <protocol> : to match specific ethertype. Some options are ip,ipv6,arp,iso

o [ether |ip ] multicast: to match L2/L3 multicast packets.

o [ether |ip] broadcast: to match L2/L3 broadcast packets.

o iso proto isis: to match ISIS PDUs

o vlan <vlan-id>: to match 802.1q packets. All vlans included if ‘vlan-id’ is not specified.

o mpls <label>: to match mpls packets.

Examples:

1. “tcpdump -t src host 10.16.151.206 && tcp 22” : To capture all SSH packets generated by a host 10.16.151.206

IP 10.16.151.206.ssh > 10.14.123.208.44248: P 940:1056(116) ack 1 win 14976
IP 10.16.151.206.ssh > 10.14.123.208.44248: P 1056:1172(116) ack 1 win 14976

2. “tcpdump -ttt -c 3 ip proto ospf” : To capture three OSPF packets with time format as delta to its previous(-ttt option). This should be useful to check whether we receive ospf packets every 10 seconds.

000000 IP 10.16.151.254 > ospf-all.mcast.net: OSPFv2, Hello, length: 44
10.000340 IP 10.16.151.254 > ospf-all.mcast.net: OSPFv2, Hello, length: 44
9. 999444 IP 10.16.151.254 > ospf-all.mcast.net: OSPFv2, Hello, length: 44

3. “tcpdump -evt -i eth0 ip src host 10.16.151.206 or ip proto ICMP and greater 2540”: To capture packet on interface “eth0” with L2 header (-e) and with verbose explanation (-v) and without timestamp (-t). Filter option is; those packets either can have SRC IP = 10.16.151.205 or ICMP packets but the size (whole packet size) should be greater than 2540 bytes. (evaluated from left to right as or,and has same precedence)

00:19:bb:2e:b7:1a (oui Unknown) > 00:01:e8:d5:9e:e2 (oui Unknown), ethertype IPv4 (0x0800), length 2543: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 2529) 
10.16.151.206 > 10.16.151.254: ICMP echo request, id 2642, seq 1, length 2509

4. “tcpdump -w capture_file dst host 10.16.151.206 and tcp dst port not 22”: To write the filtered packet capture to a file name “capture_file”. Filter option is; those packets should be destined to 10.16.151.206 and it shouldn’t be SSH packet.

5. “tcpdump -qr capture_file: To read the packet captured file “capture_file”

[root@linux-1 ~]# tcpdump -qr capture_file
reading from file capture_file, link-type EN10MB (Ethernet)
10:31:24.562468 IP 10.16.25.52.dzdaemon > 10.16.151.206.http: tcp 0
10:31:24.562651 IP 10.16.25.52.dzdaemon > 10.16.151.206.http: tcp 0
10:31:24.563159 IP 10.16.25.52.dzdaemon > 10.16.151.206.http: tcp 590

HTH

Port-based authentication:

• IEEE 802.1x standard
• Combination of AAA authentication and port security.
• Both switch and host needs to support 802.1x using Extensible Authentication Protocol over LANs (EAPOL), a L2 protocol. Default state of a switch port: unauthenticated.
• Configuration:
  • Enable AAA:
    • ‘aaa new-model’
  • Configure RADIUS server:
    • ‘radius-server host 1.1.1.1 key xxxx’
  • Configure dot1x to use radius host for authentication:
    • ‘aaa authentication dot1x default group radius’
  • Enable dot1x:
    • ‘dot1x system-auth-control;
  • Configure ports to use dot1x:
    • By default, when we enable dot1x, all ports are set to “forced-authorized” means any PC can start communication with network.
    • Configure, ‘dot1x port-control auto’ to make PC to negotiate with switch
  • To allow multiple host on a switch port;
    • ‘dot1x host-mode multihost’
  • Show dotx all

Spoofing attack:

• DHCP snooping:
  • When DHCP snooping is enabled, switch ports are categorized as trusted or untrusted.
  • DHCP servers are connected to trusted ports and all hosts are in untrusted port.
  • If the switch receives DHCP reply on untrusted ports, the switch drops the packet and moves the port to errdisable state. Hence, avoiding replies from rouge servers
  • Configuration:
    • Enable DHCP snooping:
      • ‘ip dhcp snooping’
      • VLANs where snooping has to be enabled;
        • ‘ip dhcp snooping vlan x’
      • Configure the port connected to server as trust
        • By default, all ports are untrusted.
        • (config-if)# ip dhcp snooping trust
      • Optional: Rate limit the DHCP request on untrusted port
        • (config-if)# ip dhcp snooping limit rate x
      • DHCP option-82 is enabled by default
        • [no] ip dhcp snooping information option
      • show ip dhcp snooping [binding]
• IP Source Guard:
  • To avoid address spoofing attack. Check made on end user packets that Source IP is same as assigned by DHCP server. Uses DHCP snooping binding table/static entry.
  • Check on:
    • Source IP should be same as in DHCP snooping table. Checked via IP ACL.
    • Source MAC should be same as learned and in snooping table using Port security
  • Configuration:
    • Static IP source binding:
      • ‘ip source binding <mac> vlan <id> <ip> interface <>’
      • Enable IP source guard on interface mode:
        • ‘(config-if)# ip verify source [port-security]’
        • By default, only source IP is checked.
        • For source MAC to check, use ‘port-security’ keyword.
      • ‘show ip verify source’
      • ‘show ip source binding’
• Dynamic ARP Inspection (DAI)
  • To avoid ARP spoofing by rouge host in same vlan.
  • The switch intercepts and inspects all ARP packets that arrive on an untrusted port; no inspection is done on trusted ports (which connects to another switch)
  • When an ARP reply is received on an untrusted port, the switch checks the MAC and IP addresses reported in the reply packet against DHCP snooping table or static entry.
  • By default, no check is done on ethernet MAC of the ARP packet.
  • Configuration:
    • Enable DAI on VLAN:
      • ‘(config)# ip arp inspection vlan x’
      • To configure trusted port:
        • ‘(config-if)# ip arp inspection trust’
      • To statically configure IP-MAC pair use ARP ACL:
        • ‘(config)#arp access-list <name>’
          • ‘permit ip host <ip> mac host <mac>’
          • ‘(config)# ip arp inspection filter <name> vlan <vlan> [static]’
        • By default, check is first done on ARP ACL table. If no hit, then DHCP binding table is checked. Add keyword ‘static’ to check only ARP ACL.
      • To validate the ethernet MAC of the ARP packet:
        • ‘(config)# ip arp inspection validate …’

VLAN Access list: (VACL)

• VACL are filters that directly can affect how packets are handled within a VLAN.
• Configured similar to route-map with match conditions and action items.
• Merged into TCAM table.
• Configuration:
  • ‘(config)# vlan access-map <name>
    • match {ip | mac } address <acl name>
    • action {drop | forward | redirect}
  • (config)# vlan filter <name> vlan-list <vlan>

Private VLAN

Securing VLAN trunks:

• Avoid DTP messages to be exchanged with end host. Configure ‘switchport mode access’ .
• VLAN hopping:
  • Crafted packet from one vlan can be passed to another VLAN provided following conditions occurs;
    • The host is connected to access port of a switch which can accept tagged packet
    • The switch has 802.1Q trunk with the host vlan as native vlan
  • Double tagged packet from the attacker is accepted by the switch and passes via trunk to another switch with outer native tag removed. This packet on reaching another switch can pass to the inner tagged vlan.
  • VLAN hopping cannot happen in new platform where access port blocks tagged packets.
  • VLAN hopping can be avoided by;
    • Set native vlan to unused vlan ID.
    • Prune native vlan from the trunk or configure switch to send native vlan packets tagged with native vlan ID. Command: ‘(config)#vlan dot1q tag native’

• PoE devices like IP phones receive power from the connected switch.
• The switch can provide power in two different methods:
  • Cisco Inline power (ILP): Cisco proprietary.
  • IEEE 802.3af: Based on IEEE standard.
• Switch should be capable of detecting PoE devices to offer power and bring the link UP.
• In 802.3af, switch applies small voltage between Tx and Rx line and resistance is measured to check whether current is drawn. A 25K Ohm resistance will be measured if PoE device is connected.
• There are 5 power classes that can be detected: 0 to 4. Class-0 being the default offers 15.4W.
• Class-1 device needs 4W and class-2 device needs 7W and Class-3 is optional 15.4W and class-4 device needs upto 50W (under IEEE 802.3at)
• In Cisco ILP, Power is supplied via data pair 2 and 3 (pins 1,2 and 3,6) at 48V DC.
• For 802.3af, power can be supplied as above or over data pairs 1 and 4.
• Switch uses CDP to detect power class in ILP. Switch can reduce the power on receiving CDP from connected cisco IP phone which has power required field.
• Configuration:
  • (config-if)# power inline  { auto [ max <>] | static max <> | never}
  • “never” to disable PoE on that interface.
  • “show power inline”

Voice VLAN:

• Cisco IP phone has 3-port switch setup. One to connected switch, one to connected PC and one to VoIP internal data.
• Voice data traffic should be allocated to separate vlan called ‘voice vlan’ and Qos functionality has to be implemented for voice vlan traffic.
• Switch configuration instructs how the connected Cisco IP phone should send data and voice traffic.
• Voice VLAN applicable only on access port and not on trunk. Qos should be enabled before VVLAN.
• (config-if)# switchport voice vlan <vlan id>
  • Data traffic is via native vlan (untagged)
  • Voice traffic is tagged with <vlan id>
  • Voice Qos set in 802.1p bits.
• (config-if)# switchport voice vlan dot1p
  • Data traffic is via native vlan (untagged)
  • Voice traffic is tagged with VLAN =0
  • Voice Qos set in 802.1p bits.
• (config-if)# switchport voice vlan untagged
  • Both Voice and data traffic via native vlan.
  • No 802.1p bits.
  • CDP exchange happens and the switch instruct IP phone to send traffic untagged.
  • (packets tagged with VLAN 1025)???
• (config-if)# switchport voice vlan none
  • Both Voice and data traffic via native vlan.
  • No 802.1p bits and no CDP/DTP exchange between switch and IP phone.
• ‘Show interface <type> switchport’ to view the voice and access VLAN.

Voice Qos:

• By default, Cisco phone uses IP precedence value of 5 for voice traffic and 3 for voice control traffic.
• Cisco IP phone can be considered as another switch and ‘trust boundary’ can be extended to phone.
• Configuration:
  • Enable Qos on multilayer switch: “mls qos”
  • On the interface, configure trust parameter: ‘mls qos trust { cos | ip-precedence | dscp}’
  • If we configure, ‘mls qos trust device cisco-phone’ above qos value is trusted only if a cisco IP phone is detected via CDP. Else, qos parameter is not trusted.
  • ‘switchport priority extend { cos <value> | trust}
    • ‘trust’ option extends the trust boundary to PC. Applications can send packets with Cos bits set and the IP phone trust those values and pass unchanged.
    • ‘cos <value>’ makes the IP phone to overwrite the Qos bits from the PC to the specified value. If the PC is untrusted, cos should be overwritten by 0.
    • By default, the phones’ PC port is untrusted and overwrites cos values to 0.
  • Configure ‘mls qos trust cos’ to switch uplink port.
  • ‘Show mls qos interface <type>’
• Auto-qos: Perform enabling qos, cos-dhcp mapping, ingress and egress queue, strict priority for voice egress traffic and trust boundaries.
  • (config-if)# auto qos voip { cisco-phone | cisco-softphone | trust }
    • ‘cisco-phone’ extends the trust boundary to the cisco IP phone detected by CDP.
    • ‘cisco-softphone’ . packets received with DSCP value of 24,26,46 are trusted.
    • ‘trust’: all packets received on the interface is trusted. Configured on switch uplink.
  • ‘Show auto qos interface <type>’

• Multilayer switches act as gateway for end users. First Hop Redundancy protocols (FHRP) are available to provide redundancy to end users.
• FHRP includes;
  • Hot Standby Router Protocol (HSRP)
  • Virtual Router Redundancy Protocol (VRRP)
  • Gateway Load Balancing Protocol (GLBP)

HSRP:

• HSRP is Cisco proprietary allows several routers to appear as single IP gateway address.
• Among the group of routers, one router is selected as ‘active’ and another is selected as ‘standby’ and all others are selected as ‘listen’
• All HSRP routers send Hello packets destined to 224.0.0.2 using UDP port 1985.
• A group number can be assigned to HSRP. Values from 0 to 255 (default being 0)
• HSRP groups are local significant to the interface.
• Router election:
  • Priority values (0 to 255) can be assigned to HSRP routers. Default value is 100. Highest: 255
  • Router with highest Priority will be ‘active’ for that group. If there is tie, router with highest interface IP address will win the election.
  • HSRP Devices progress their interface states as disabled, init, listen, speak, standby, active.
  • Only standby monitors hello message from active router.
  • Default hello and hold down timers are 3 and 10 respectively. Use same timers in all routers.
  • By default, preempt is disabled.
• Authentication:
  • Plain-text:
    • HSRP messages are sent with plain-text key. Default key being ‘cisco’
    • ‘(config-if)#standby 1 authentication cisco1’
  • MD5:
    • Can use either ‘key-string’ or ‘key-chain’ option.
    • ‘(config-if)#standby 1 authentication md5 {key-string | key-chain}’
• We can track uplink interface so that when the interface goes down, the priority is reduced by a value, 10 being default. “(config-if)#standby 1 track <interface> <decrement value>’
  • For successful takeover of master;
    • Another router should have highest priority.
    • ‘preempt’ should be configured on another router.
• HSRP address: Configure via ‘(config-if)#standby  <group> ip <address>”. Assign this IP address as default gateway to host. Virtual MAC address used for a specific group would be 00:00:0c:07:ac:xx where ‘xx’ is the HSRP group number.
• For load balance, configure two groups on an interface. Make the local router active for one group and standby for another group. Assign 2 configured IP address as default gateway to hosts.
• Configuration:
  • ‘(config-if)#standby  <group>  priority <>’
  • ‘(config-if)#standby  <group>  timers <hello> <hold>’
  • ‘(config-if)#standby 1 preempt delay { minimum | reload} ‘
    • Minimum <sec> : to force router to preempt after <sec> once the router is ready for active role.
    • Reload <sec> : to force router to wait for <sec> after it is reloaded.
  • “show standby” command to view the HSRP status.

VRRP:

• VRRP is standard based defined in RFC-2338. Same concept as HSRP.
• Active router is called as ‘master’ and all other routers are in ‘backup’ state.
• Priority values (1 to 254) can be assigned to VRRP routers. Default value is 100. Highest: 254
• VRRP group numbers: 0 to 255.
• Virtual Mac address is = 00: 00: 5e:00:01:xx where ‘xx’ is VRRP group number.
• VRRP sends hello message every 1 second destined to 224.0.0.18 using IP protocol: 112
• Backup router can learn advertisement interval from master. ‘vrrp <x> timers learn’
• Preempt enabled by default.
• Configuration:
  • ‘vrrp <group> priority’
  • ‘vrrp <group> ip <add>’
  • ‘vrrp <group> authentication <>’
  • ‘vrrp <group> preempt delay <>’
  • Show vrrp

GLBP:

• Cisco proprietary protocol that overcomes the limitation of load balancing in VRRP and HSRP.
• Multiple routers participate in forwarding packets from host to default gateway address.
• When client sends ARP request for default-gateway, GLBP replies with virtual MAC address of selected router in the group.
• All clients use same Virtual address but different MAC address.
• Active virtual Gateway: (AVG):
  • One router in the group is selected as AVG by election. Highest Priority or highest interface IP address.
  • AVG answers all ARP request for default-gateway from clients. Selection of a routers’ virtual MAC depends on load-balancing algorithm in use.
  • Upto four virtual MAC address (4 routers) can be used in a group at any time. Routers involved in forwarding the traffic is referred as ‘active virtual forwarder’ (AVF).
  • Other routers than 4 serve as backup AVF routers.
• GLBP group numbers can be from 0 to 1023. Priority can be 1 to 255. Highest: 255 and default:100
• Preempt disabled by default as in HSRP.
• AVG sends hello message to all GLBP routers and expects hello from each routers.
• Default: hello 3 seconds and holds 10 seconds. Other routers can learn these times from AVG.
• Active virtual forwarder: (AVF)
  • AVG assigns four routers as AVF along with their virtual MAC address.
  • Mac address will be in the form of 00:07:b4:xx:xx:yy
    • ‘xx:xx’ denotes six zero bits followed by 10-bit GLBP group number.
    • ‘yy’ denotes virtual forwarder number.
  • All AVF routers exchange hello packets with each other. If AVG fails to receive hello within hold timer, it assumes that AVF as failed and assigns the failed AVF function to another AVF router.  This router act as forwarding router for 2 MAC address (its own+failed routers’ mac)
  • AVG uses two timers to resolve this condition – single router handles 2 MAC addresses.
    • Redirect timer: Time when AVG will stop using the failed routers mac address in ARP reply. Default is 10 minutes
    • Timeout time: Time when AVG flushes the old mac address and the failed AVF ID from all GLBP peer. Clients which used old virtual mac address need to refresh its arp to lean new mac address for the default-gateway. Default is 4 Hours.
    • Can be changed by ‘glbp <group> timers redirect <redirect> <timeout>’
  • GLBP uses weighting function to determine which routers to become AVF. Each participating router will have 100 as maximum weight. When a specified interface goes down, the weight value is decremented by specified value (default: 10)
  • GLBP uses threshold to determine when a router can be AVF. If weight < lower threshold, the router should give up its AVF role. When weight> upper threshold, it can resume its  role
  • Create a track object as assign it to GLBP weighting configuration.
    • ‘(config)# track <number> interface <id> {line-protocol | ip-routing}’
    • ‘(config-if)# glbp <group> weighting <max value> lower <x> upper <x>’
    • ‘(config-if)# glbp <group> weighting track <number> decrement <value>’
  • Higher weight router cannot preempt the current AVF even if it has lower weight.
• Load-balancing:
  • AVG assigns the four virtual MAC address to the ARP request in following manner;
    • Round-robin: Default method. AVG uses mac addresses in round robin fashion in ARP reply. All AVF is expected to receive fairly equal amount of traffic.
    • Weighted: AVG uses a particular router depending on its interface weight function.
    • Host dependent: Each client receives the same virtual mac address in ARP reply.
  • ‘(config-if)# glbp <group> load-balancing { round-robin | weighted | host-dependent }’
• Configuration:
  • ‘(config-if)#glbp <group> priority <level>’
  • ‘(config-if)#glbp <group> preempt [delay]’
  • ‘(config-if)#glbp <group> timers [msec] <hello> [msec] <hold>’
  • ‘(config-if)#glbp <group> ip <address> [secondary]’
  • Show glbp [brief]

Hardware redundancy:

• Cat 4500, 6500 can accept two Supervisor modules, one acting as active and other remains in standby state. Standby supervisor boot up and initialize only to certain level depending on mode.
• Redundancy modes: Route Processor Redundancy (RPR);
  • When primary fails, standby reload every other modules and then initialize supervisor fn.
• Route Processor Redundancy plus (RPR+):
  • Standby initialize L2 and L3 fn without reloading other modules. Port status remains same.
• Stateful Switchover (SSO):
  • Standby is fully booted and initialized. Both startup and running config are synced
• Single-router Mode (SRM): Two route processors are being used buy only one active at a time.
• Dual-router Mode (DRM): Both route processors are active at a time.
• Commands:
  • (config)# redundancy
  • (config-red)#mode {rpr | rpr-plus | sso}
  • Show redundancy status
• Supervisor synchronization:
  • By default, the active supervisor synch both startup config and configuration register values with standby supervisor.
  • (config)# redundancy
  • (config-red)#main-cpu
  • (config-r-mc)# auto-sync { startup-config | config-register | bootvar}
  • To return to default, use “auto-sync standard”
• Nonstop Forwarding: (NSF)
  • Used along with SOO feature to quickly rebuild the RIB table from nsf-aware neighbor router
  • “bgp graceful-restart” under BGP configuration. “nsf” under eigrp, ospf and isis config mode.