[Mithrilhall]

I have a router that I'm monitoring/managing and the tcp-timeout was:

ip nat translation tcp-timeout 460


I had a complaint from a user that was being disconnected last night about every 7-8 minutes. This happed when they were playing WoW or using AIM. So I adjusted the timeout to:

ip nat translation tcp-timeout 2400

The CPU/Memory utilization on the router is still low and I'm keeping and eye on it.

I thought the NAT tcp-timeout setting only affected translations after a period of inactivity. I wouldn't think playing WoW would equate to an inactive connection and I'm pretty sure AIM sends a keep alive packet every so often.

[StumpedTechy]

No its not the period of inactivity that it resets -

It is in fact - to change the amount of time after which Network Address Translation (NAT) translations time out.

So if the translation is taking a long time you may in fact have to bump up the amount of time that it is set to retry. The question now becomes why is it taking so long to do the translation if the CPU usage is so low?

[Mithrilhall]

Someone recommended that I remove the line from the config but I'm wondering why the translation is timing out.

AIM (AOL Instant Messenger) sends out a keep alive packet from what I've read. So I don't know what's is causing it.

[O111111O]

What version of IOS are you running?

Do sh ver, give me major/minor/featureset.

[Mithrilhall]

cisco 2691 (R7000) processor (revision 0.1) with 105472K/25600K bytes of memory.
Processor board ID JMX0823L4TW
R7000 CPU at 160MHz, Implementation 39, Rev 3.3, 256KB L2 Cache
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
3 Serial network interface(s)
DRAM configuration is 64 bits wide with parity disabled.
55K bytes of non-volatile configuration memory.
31168K bytes of ATA System CompactFlash (Read/Write)

[O111111O]

That's well and good, it shows the hardware. Like I said, what feature set/major/minor rev? (The very first part of the sh ver that you cut out)

Example is below. This is a 3745 running IPBASE feature set 12.3(7)T


Cisco IOS Software, 3700 Software (C3745-IPBASE-M), Version 12.3(7)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2004 by Cisco Systems, Inc.
Compiled Sat 21-Feb-04 05:53 by eaarmas

ROM: System Bootstrap, Version 12.2(8r)T2, RELEASE SOFTWARE (fc1)

routerblah uptime is 44 weeks, 6 days, 19 hours, 22 minutes
System returned to ROM by power-on
System restarted at 15:32:54 EDT Thu Apr 14 2005
System image file is "flash:c3745-ipbase-mz.123-7.T.bin"

[O111111O]

By the way, the reason I'm asking is TCP FIN bug. Sum rev's of IOS would clear xlates immediately on TCP FIN.

Normally in IOS, TCP translations time out after 24 hours by default, unless a TCP Reset (RST) or TCP Finish (FIN) is seen in the TCP stream, in which case the translation times out after 1 minute.

Also, how many hosts do you have connected behind this router? I don't understand why your TCP timeout is so low. The default is typically fine unless you're running IP Inspect/have alot of hosts/are worried about memory. If you're worried about virii breakout running the router out of memory you could consider upgrading to 12.3T and using NAT rate limiting.

There's also a distinct possibility that you ARE simply seeing TCP FIN from destination host (HTTP does this), and your router is simply clearing the xlate like it should.

Try:

"ip nat translation finrst-timeout 300" (Bump up fin/rst timeout to 5 minutes)

[Rockn]

And you are allowing them to play these games on the network? It would be a better use of your time to tell them to do actual work and stop complaining about their games not working. It might be worth looking into now however if there are future legit apps that might not work.

[Mithrilhall]

Here is the show ver:

Cisco Internetwork Operating System Software 
IOS (tm) 2600 Software (C2691-IPBASE-M), Version 12.3(5c), RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 13-Apr-04 23:41 by cmong
Image text-base: 0x60008AF4, data-base: 0x60FFA000

ROM: System Bootstrap, Version 12.2(8r)T2, RELEASE SOFTWARE (fc1)

Router uptime is 12 weeks, 6 days, 4 hours, 41 minutes
System returned to ROM by reload at 09:12:01 EDT Fri Jun 10 2005
System restarted at 09:13:00 EDT Fri Jun 10 2005
System image file is "flash:c2691-ipbase-mz.123-5c.bin"

cisco 2691 (R7000) processor (revision 0.1) with 105472K/25600K bytes of memory.
Processor board ID JMX0823L4TW
R7000 CPU at 160MHz, Implementation 39, Rev 3.3, 256KB L2 Cache
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
3 Serial network interface(s)
DRAM configuration is 64 bits wide with parity disabled.
55K bytes of non-volatile configuration memory.
31168K bytes of ATA System CompactFlash (Read/Write)

BTW...this is for an apartment building...not work; so they can play games as they see fit.

Also, I would expect no more than 300 concurrent users.

[O111111O]

Well, to start with 12.3(5)c is very deferred. Upgrade to a minimum of 12.3(5)e. There's a CSC bug ID for 12.3(5)c that pertains to turning of ip nat tran timeout, which is why your router may have this.

Also, if you have anybody trying to use Vonage, that'll blow it up. Many many SIP issues with that release. (SIP fixup)


I would suggest the following, considering that you will have a fair user load. (What's your WAN? DS1, a couple of DS1's? Are you running IMA or MLPPP?)

1] Increase NAT TCP timeout to a day.**
** I would suggest upgrading to ED 12.3T release and using NAT rate limiting. This will help against a single host "DOS" if they're busy.

2] Do "sho ip nat tran detail" Will give you state table with source/destination NAT. See what's connected when this happens.

3] Create "Internet friendly" input ACL on your Ethernet interface facing your users. (Block RPC and other MS-BS from touching your WAN port outbound. Will help with virii outbreaks.)

4] Low latency/class based weighted fair queueing. Prioritize traffic based on port/protocol. NBAR can help with this. (I.E. Make sure there's always enough WAN pipe for jitter/latency sensitive traffic.)

5] Depending on your WAN load, there's also a possibility that TCP congestion control isn't working through NAT. You can turn this on in global as ip nat option.

-----

Example "Internet friendly" ACL, this will knock down alot of Microsoft poomp.

ip access-list extended eth-input-priv
deny tcp any any range 135 139
deny udp any any range 135 139
deny tcp any any eq 445
deny udp any any eq 445
permit ip any any
!
interface fastethernet 0/0
access-group eth-input-priv in

[Mithrilhall]

Currently running 3 T1s.

This is what I have for an ACL:

access-list 110 remark Block Dangerous Traffic from Internet
access-list 110 deny   tcp any any eq 135
access-list 110 deny   tcp any any eq 139
access-list 110 deny   tcp any any eq 445
access-list 110 deny   tcp any any eq 593
access-list 110 deny   tcp any any range 6881 6889
access-list 110 deny   tcp any any range 6346 6350
access-list 110 deny   tcp any any range 16881 16882
access-list 110 deny   tcp any any eq 1214
access-list 110 deny   udp any any eq 135
access-list 110 deny   udp any any eq netbios-ns
access-list 110 deny   udp any any eq netbios-dgm
access-list 110 deny   udp any any eq 445
access-list 110 deny   udp any any range 6881 6889
access-list 110 deny   udp any any range 16881 16882
access-list 110 permit tcp any any
access-list 110 permit udp any any
access-list 110 deny   53 any any
access-list 110 deny   55 any any
access-list 110 deny   77 any any
access-list 110 deny   pim any any
access-list 110 permit ip any any

[Mithrilhall]

Also, do you have any info or link on the CSC bug ID for 12.3(5)c as well as the SIP fixup pertaining to Vonage? We do have some Vonage users on the network that have complained about the service.

[O111111O]

So are your 3 T1's running in IMA or ML PPP? Or do you just have 3 /30's? With that many users jitter sensitive apps probably suck. If you're not currently, you'll want to bond those DS1's and use CBWFQ with RSVP/WRED on der Ethernet.

--------

Your ACL is input "from Internet". What I'm trying to help you with is NAT state table filling up because of outbound poop. Inbound ACL is ok, but you really need to think about "outbound". This protects the router (and you.)

----------

I'm curious, do you have a large CIDR for NAT pool that you're doing overload? SIP has issues accross PAT, so stateful inspection/fixup is usually needed to make that work.

Some that are applicable. Subsets of code are deferred for this very reason. (Supersceded by new ED rev.)

CSCdm05667 - Voice qual.
CSCsa47225 - MLPPP/PPP negotiation loop. (If you're running MLPPP.)
CSCed57814 - SIP/NAT setup. (You doing SIP inspect?)
CSCeh47763 - NAT causes router to generate ACKs for RST's
CSCef13633 - NAT causes malformed SIP contact header.

There are a bunch. Again, why the code is deferred. This happens a fair amount, ED code (especially "T" train) is "bleeding edge" stuff. Especially VOIP support.

To put it in perspective, allot of the core stuff I deal with runs "S" code, (Service Provider). 12.0(24)S is what I'm running in many places. As you can see, XM and other releases are up to 12.4 to support the myriad of things that customers want. Due to this, and the pile of different CPU/ASIC you see bugs.

In any case, before you beat your head against the wall - get out of deferred code.

[Mithrilhall]

This is what I have so far, and my Ts are bonded.

interface Serial0/0
   57 :  ip address xxx.xxx.xxx.xxx 255.255.255.252
   58 :  ip access-group 110 in
   59 :  ip nat outside
   60 :  ip load-sharing per-packet
   61 :  encapsulation ppp
   62 :  no ip route-cache
   63 :  no fair-queue
   64 :  service-module t1 timeslots 1-24
   65 :  service-module t1 remote-alarm-enable
   66 : !
   67 : interface FastEthernet0/1
   68 :  no ip address
   69 :  duplex auto
   70 :  speed auto
   71 : !
   72 : interface Serial0/1
   73 :  ip address xxx.xxx.xxx.xxx 255.255.255.252
   74 :  ip access-group 110 in
   75 :  ip nat outside
   76 :  ip load-sharing per-packet
   77 :  encapsulation ppp
   78 :  no ip route-cache
   79 :  no fair-queue
   80 :  service-module t1 timeslots 1-24
   81 :  service-module t1 remote-alarm-enable
   82 : !
   83 : interface Serial0/2
   84 :  ip address xxx.xxx.xxx.xxx 255.255.255.252
   85 :  ip access-group 110 in
   86 :  ip nat outside
   87 :  ip load-sharing per-packet
   88 :  encapsulation ppp
   89 :  no ip route-cache
   90 :  no fair-queue
   91 :  service-module t1 timeslots 1-24
   92 :  service-module t1 remote-alarm-enable
   93 : !

 153 : access-list 1 permit xxx.xxx.xxx.xxx 0.0.1.255
  154 : access-list 88 permit xxx.xxx.xxx.xxx
  155 : access-list 88 permit xxx.xxx.xxx.xxx
  156 : access-list 90 permit xxx.xxx.xxx.xxx 0.0.0.255
  157 : access-list 99 permit xxx.xxx.xxx.xxx
  158 : access-list 99 permit xxx.xxx.xxx.xxx
  159 : access-list 99 permit xxx.xxx.xxx.xxx
  160 : access-list 99 permit xxx.xxx.xxx.xxx
  161 : access-list 99 permit xxx.xxx.xxx.xxx
  162 : access-list 99 permit xxx.xxx.xxx.xxx 0.0.0.255
  163 : access-list 99 deny   any log
  164 : access-list 110 remark Block Dangerous Traffic from Internet
  165 : access-list 110 deny   tcp any any eq 135
  166 : access-list 110 deny   tcp any any eq 139
  167 : access-list 110 deny   tcp any any eq 445
  168 : access-list 110 deny   tcp any any eq 593
  169 : access-list 110 deny   tcp any any range 6881 6889
  170 : access-list 110 deny   tcp any any range 6346 6350
  171 : access-list 110 deny   tcp any any range 16881 16882
  172 : access-list 110 deny   tcp any any eq 1214
  173 : access-list 110 deny   udp any any eq 135
  174 : access-list 110 deny   udp any any eq netbios-ns
  175 : access-list 110 deny   udp any any eq netbios-dgm
  176 : access-list 110 deny   udp any any eq 445
  177 : access-list 110 deny   udp any any range 6881 6889
  178 : access-list 110 deny   udp any any range 16881 16882
  179 : access-list 110 permit tcp any any
  180 : access-list 110 permit udp any any
  181 : access-list 110 deny   53 any any
  182 : access-list 110 deny   55 any any
  183 : access-list 110 deny   77 any any
  184 : access-list 110 deny   pim any any
  185 : access-list 110 deny   ip xxx.xxx.xxx.xxx 0.0.1.255 any
  186 : access-list 110 permit ip any any
  187 : access-list 111 remark Block Dangerous Traffic from TenantNet
  188 : access-list 111 deny   tcp any any eq 135
  189 : access-list 111 deny   tcp any any eq 139
  190 : access-list 111 deny   tcp any any eq 445
  191 : access-list 111 deny   tcp any any eq 593
  192 : access-list 111 deny   udp any any eq 135
  193 : access-list 111 deny   udp any any eq netbios-ns
  194 : access-list 111 deny   udp any any eq netbios-dgm
  195 : access-list 111 deny   udp any any eq 445
  196 : access-list 111 permit tcp any any
  197 : access-list 111 permit udp any any
  198 : access-list 111 deny   53 any any
  199 : access-list 111 deny   55 any any
  200 : access-list 111 deny   77 any any
  201 : access-list 111 deny   pim any any
  202 : access-list 111 permit ip any any
  203 : !

[O111111O]

Hi,

Your DS1's aren't bonded. You're using per-packet cef load sharing. Very much not to be confused with inverse multiplexing.

What you have are NxDS1 interfaces, CEF table on your router will forward outbound packets per-packet with the CEF adjacency table.

To put it bluntly, per-packet load sharing is horrible. This is crap that Cisco puts as requirments in IE labs to try and make things like MPLS TE labels map accross it. (It's a *****) With per-packet load sharing it fires each packet out blindly and forces the end host to re-assemble the packet. In many instances packets arrive at the destination out of order and you'll see huge variances in packet delivery time (Jitter). Also, with out of order packets the host may simply drop them. (I.E. your AIM issue.)

On top of all that, is your ISP also using per-packet load sharing? Probably not. Default is per-destination. What this means is that your loadsharing is simplex. Outbound from you may "balanced" across three DS1's, but inbound from your ISP is destination balanced. The end result is that something like an FTP will max out a single DS1.

It's probably reason #1 that any users passing SIP/H.323 traffic are complaining.

If your ISP is flexible (hopefully they are) I would suggest you have them move you to something that's truly inverse multiplexing. In your case; Multilink PPP.

With any inverse-multiplexing on Cisco devices, you'll have a virtual interface (in this case Multilink 1) with an IP address, and your physical interfaces will have no IP but will be inserted into a channel-group.

The router essentially has a heuristic queuing engine for each packet. In essence it truly round-robins each packet across "n" given interfaces in the multilink group. In your case, since you have 3 DS1's you will receive true 4.5Mbps bi-directional throughput.

In summary;

NAT w/ your code is an issue. I would suggest using default timeouts.
Access-list 110 is ok, but it's not stateful. You'll drop certain connections (whole conversation there)
per-packet load sharing with CEF is bad! Make it go away! Your users will be so much happier!

I would tackle the multiplexing of your DS1's first.

PPP multilink example below: (Your ISP HAS TO DO THIS AS WELL.)

interface multilink1
ip address xxx.xxx.xxx.xxx 255.255.255.252
ppp encapsulation
ppp multilink
multilink-group 1
ip access-group 110 in
multilink min-links 3 mandatory **not really needed, but I like it.
ip nat outside

interface S0/n (where "n" is the interface number.)
no ip address
encapsulation ppp
no fair-queue
no cdp enable
ppp multilink
multilink-group 1