Subject: [USR-TC] ARC not releasing IP addresses

From: "MacDonald, James" <[email-masked]>
Date: Thu, 15 Aug 2002 14:39:34 +0000

Hi All, 

We have 1 hyperarc that seems to have a problem where it can not release IPs
previously assigned properly. It sometimes does and sometimes doesn't ...
eventually the arc  runs out of IP addresses to assign and spews the
following error:

Aug 15 09:02:12 9L102PDI-H1 At 13:01:46, Facility "IP", Level "CRITICAL"::
ip_fwd_get_opt: no IP address available for dynamic address assignment 

As you can see, the IP addresses (162 of them) are completely used up and
there are only 7 users online. We don't think this is a code specific issue
as we run this code on all ARCs (about 250 of them) and this is the only one
getting these errors ... it may be hardware related, but in my experience
these types of issues are generally code ... so, I'm at a cross-roads so to
speak. Has anyone else run into this and found a workaround?

BTW: we can find only 2 ways to correct the issue, rebooting or removing the
IP Pools and adding them back ... we generally reboot as that is much
quicker and less error prone.

System Version: V5.0.83/Non-Encr

9L102PDI-H1> list ip pools

IP ADDRESS POOLS
Name                Address        Size InUse State   Route      Unused
Priority Status
9L102-H1-POOL1   216.123.143.17/29 6    6     PUBLIC  AGGREGATE    0    1
ACTIVE
9L102-H1-POOL2   216.123.140.65/27 30   30    PUBLIC  AGGREGATE    0    1
ACTIVE
9L102-H1-POOL3    216.123.132.1/25 126  126   PUBLIC  AGGREGATE    0    1
ACTIVE
9L102PDI-H1> list conNECTIONS 

CONNECTIONS

                                                Start       Start
IfName          User Name            Type   DLL  Date        Time
slot:1/mod:4    xxxxxx  		DIALIN PPP      14-AUG-2002 21:14:41

slot:1/mod:5    xxxxxx		DIALIN PPP      15-AUG-2002 05:13:22  
slot:1/mod:11  xxxxxx		DIALIN PPP      15-AUG-2002 13:55:43  
slot:1/mod:13  xxxxxx		DIALIN PPP      15-AUG-2002 03:28:04  
slot:1/mod:16  xxxxxx		DIALIN PPP      15-AUG-2002 14:04:17  
slot:1/mod:19  xxxxxx		DIALIN PPP      15-AUG-2002 10:19:32  
slot:1/mod:21  xxxxxx		DIALIN PPP      15-AUG-2002 08:59:12  
9L102PDI-H1> 

Thanks,

James MacDonald, CCNP
Senior Platform Manager, IP
Network Operations
AT&T Canada Corp.
370 King St W., 6th Floor
Toronto, ON M5V1J9
Ph: 416-204-5793
Fx: 416-204-5744
Emergency: 416-204-5700 (NMC Hotline)
Email: [email-masked]

---

From: V <[email-masked]>
Date: Fri, 16 Aug 2002 00:05:37 +0000

Do you have 'hint assigned'  enabled?

-V
----- Original Message -----
From: "MacDonald, James" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, August 15, 2002 9:39 AM
Subject: [USR-TC] ARC not releasing IP addresses

> _______________________________________________
> USR-TC mailing list
> [email-masked]
> http://mailman.xmission.com/cgi-bin/mailman/listinfo/usr-tc

---

From: Aaron Nabil <[email-masked]>
Date: Fri, 16 Aug 2002 15:19:23 +0000

I'm running 3 to 1 now, where have other people started seeing problems
at?

---

From: Todd Bertolozzi <[email-masked]>
Date: Fri, 16 Aug 2002 16:54:40 +0000

I have many chassis setup dual arc (7 dsp's per arc on each chassis).

I have not noticed any issues with this.  Guess it depends on how much stuff
the arc is doing and how much memory it has.  You can definitely go more
then 3 though.

Todd

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Aaron Nabil
Sent: Friday, August 16, 2002 11:19 AM
To: [email-masked]'
Subject: [USR-TC] What is a conservative dsp to arc ratio?

---

From: "Chris A. Kalin" <[email-masked]>
Date: Fri, 16 Aug 2002 17:21:29 +0000

At how many ARCs and DSPs would we need to switch to 130A power supplies?

Would having two 70A supplies serve the same purpose as having one 130A,
other than the obvious fact that if one of them were to go down, the chassis
would be very underpowered.  :)

Thanks!

Chris Kalin

---

From: Tom Williams <[email-masked]>
Date: Fri, 16 Aug 2002 17:21:43 +0000

If you are running a full chassis 7 to 1 is a good number, but I have
customers who do not want to use more then 1 arc, and they run 10 to 1 with
128Meg on the ARc, seems to work fine it all depends on the RAM, and what
the arc is going to be doing.

Tom

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Todd Bertolozzi
Sent: Friday, August 16, 2002 11:55 AM
To: [email-masked]
Subject: RE: [USR-TC] What is a conservative dsp to arc ratio?

Todd

---

From: Todd Bertolozzi <[email-masked]>
Date: Fri, 16 Aug 2002 17:29:20 +0000

The following are 3com's recommended specs for maintaining redundancy
while running dual power supplies units or PSU's.  If you are running 2
PSU's with the
following recommendations you will not have any issues if one of them fails.

45 Amp =    6 DSP's, 1 HiperArc, and 1 NMC
70 Amp =   10 DSP's, 1 HiperArc, and 1 NMC
130 Amp =  You can fully load the chassis

These are what 3com recommends....you can go a bit more but then you're not
doing what the equipment vendor recommends.

Todd

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Chris A. Kalin
Sent: Friday, August 16, 2002 1:22 PM
To: [email-masked]
Subject: Re: [USR-TC] What is a conservative dsp to arc ratio?

Thanks!

Chris Kalin

---

From: Mike Greene <[email-masked]>
Date: Fri, 23 Aug 2002 23:15:54 +0000

Just was reviewing the "2 ARC" discussion on the list (I'm finally about to 
do this after all these years) and smiled at the reliability issue on the 
ARC's and NMC's.

Just for grins I logged into one of my ARC's running 4.2.32 and lookie here:

SYSTEM DESCRIPTION
System Descriptor:
3Com Corporation HiPer Access Router Card Built on Aug 16 1999 at 17:19:37.
Object ID:                                1.3.6.1.4.1.429.2.19
System UpTime:                            413d 07:07:16
System Version:                           V4.2.32

413 days!

4.2.32 has been by far the best code I've run on my ARC's, 5.2.102 is 
looking pretty good too, but still not enough time to say for sure.  No 
memory leaks yet though, which is better than 5.2.9!!

- Mike

------------------------------------------------
Rock Island Communications, Inc.  (360)-378-5884
http://www.rockisland.com/  San Juan Islands, WA
------------------------------------------------

---

From: Support <[email-masked]>
Date: Mon, 26 Aug 2002 20:50:29 +0000

Hi.  My MRTG graph (http://www.inet2000.com/mrtg/3comtc.modem.html) shows
period with no modem use, however we've had no complaints of the system
dying for any period of time and the call durations on some of our currently
online users are as high as 47 hours for this call (Yes, that's a camper and
another issue....)

We're new to MRTG (as you can see from the graphs) and expect that this is a
MRTG configuration issue (perhaps a longer timeout???) and not a real modem
pool problem.  Any help would be appreciated greatly.

Todd Chamberlain
[email-masked]

---

From: Lewis Bergman <[email-masked]>
Date: Mon, 26 Aug 2002 21:38:38 +0000

On Monday 26 August 2002 03:49 pm, Support wrote:
> dying for any period of time and the call durations on some of our
> currently online users are as high as 47 hours for this call (Yes, that's a
> camper and another issue....)
One of the options mrtg has is to reflect a null response as zero rather than 
the same as the last non-null response mrtg received. Maybe this is something 
to look into. More to the point would be to fix any network issues that are 
causing this timeout.
-- 
Lewis Bergman
Texas Communications
4309 Maple St.
Abilene, TX 79602-8044
915-695-6962 ext 115

---

From: Support <[email-masked]>
Date: Mon, 26 Aug 2002 23:21:40 +0000

Thanks Lewis.  I'm not certain it's a timeout at all - I was just guessing
that there were actually people online during that time, just that MRTG
didn't know it.  I'm basically guessing that it didn't get a response
quickly enough and records this as zero.

I'm not aware of any network problems, but frankly that's using the
'everything seems fast and seems to work correctly' diagnostic tool.  We've
been running this modem pool for a couple month's and haven't had any
substantial problems reported by users, and I'm pretty sure if we were
offline for 45 minutes or so, 4 or 5 times yesterday, we'd have heard about
it.  :-)

So, that means there really is no Modem Pool problem, just a MRTG problem -
or at least some sort of reporting problem.  The other reports on that same
pool...

http://www.inet2000.com/mrtg/3comtc.pub.html
http://www.inet2000.com/mrtg/3comtc.ram.html
http://www.inet2000.com/mrtg/modems.htm

don't seem to have any gaps in the data, so it doesn't seem that MRTG has a
hard time talking to it in other cases....

So, basically, I'm a newbie on MRTG and I don't really know what to ask,
other than 'help.' :-)

---

From: Craig Gittens <[email-masked]>
Date: Wed, 28 Aug 2002 15:12:34 +0000

I would try changing the MAX users setting to 26 or 28. It can be found in
the MRTG config file. I have had problems where if the max threshhold is
surpassed (could it be counting a telnet as a user?) my MRTG doesn't read it
as zero and just repeats the last reading. (Using it for traffic not user
count tho) There are other causes. Do you have a firewall between the MRTG
box and the 3Com? What type of box are you running MRTG from? I have found
that running MRTG from an NT box is flaky sometimes.

This is my first post. yay! We bought a TC unit a while ago..not even sure
what model it is..and I haven't set it up yet...hoping to get it done pretty
soon...so I will have lots of stupid questions. I did notice that there is a
lot of helpful stuff in the headers of the list email so I will try those
links first....

Craig.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Support
Sent: Monday, August 26, 2002 7:19 PM
To: [email-masked]
Subject: Re: [USR-TC] MRTG Help

---

From: Support <[email-masked]>
Date: Wed, 28 Aug 2002 16:14:25 +0000

OK.  Thanks Craig.  I'm proud to be the recipient of your first post!  I'll
try these things.  If you check the graph, it certainly does seem to be only
a problem when that DSP card fills up, so it may well be the Max user thing.
I'll check that, thanks.

There is no firewall or anything, they're just sitting next to each other
plugged into a 3COM PS40 Switching Hub.

It is running on an NT machine, but we're running lots of Perl things
running on NT machines.  That's an idea, but I doubt it's the issue, since
all the other MRTG graphs seem OK.  Of course, I've only been MRTGing for a
week and I have no clue.  (those two statements are totally separate facts.)

Todd Chamberlain
[email-masked]

---

From: Support <[email-masked]>
Date: Wed, 28 Aug 2002 17:00:19 +0000

Actually, I think this is probably it - thank you.  There is an option
"unknaszero" (which I assume means to put any unknown response as a zero,
and I should have a "unknown as last" instead.  I'd rather not artificially
put that max at 26, so do you happen to know what the correct option is for
unknown as last value?  Maybe 'unknaslast'?

---

From: Craig Gittens <[email-masked]>
Date: Wed, 28 Aug 2002 17:25:12 +0000

I don't I am still working on MRTG 2.8 something. :) However putting it at
26 could be a temporary thing just to see if it is the problem.

Craig.

-----Original Message-----
Sent: Wednesday, August 28, 2002 12:56 PM

---

From: Support <[email-masked]>
Date: Thu, 29 Aug 2002 00:26:17 +0000

Craig, thank you.  This turned out to be the problem - at least so far it
looks good.  I read the MRTG manual (NO!) and the 'unknaszero' option is
exactly that, an option.  If you leave it off, it'll automatically do a
'unknown as last value' sort of situation.  I simply removed that option and
now I left the MAX at 24 (which reflect the modems in that pool) and we look
set.  You can see the difference from last evening to so far this evening
with no dropout to zero's so far

http://www.inet2000.com/mrtg/3comtc.modem.html

Thanks again, Craig.

Todd Chamberlain

----- Original Message -----
From: "Craig Gittens" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, August 28, 2002 11:23 AM
Subject: RE: [USR-TC] MRTG Help

> > surpassed (could it be counting a telnet as a user?) my MRTG doesn't
read
> it
> > soon...so I will have lots of stupid questions. I did notice that there
is
> a
> > don't seem to have any gaps in the data, so it doesn't seem that MRTG
has
> a
> > > One of the options mrtg has is to reflect a null response as zero
rather
> > than
> > > to look into. More to the point would be to fix any network issues
that
> > are

---

From: Craig Gittens <[email-masked]>
Date: Thu, 29 Aug 2002 12:33:02 +0000

I'm glad that fixed it Todd. However I must say that I would not be happy
with that solution. Why is it possibly getting a reading over 24 or an
unknown reading? That would bug the h311 out of me. For example, suppose you
get a reading of 6 and then get that flat 6 reading for the next two hours
because it suddenly jumped to 25 or some bug kicked in. You have no reliable
stats. Yes, your graphs looks prettier now but not necessarily correct. If
you notice you have quite a few flat lines in it which indicate it did it's
unknown number for at least 15-20 minutes. That is an indication that
something is wrong.

Craig.

-----Original Message-----
Sent: Wednesday, August 28, 2002 8:23 PM

Thanks again, Craig.

Todd Chamberlain

----- Original Message -----
From: "Craig Gittens" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, August 28, 2002 11:23 AM
Subject: RE: [USR-TC] MRTG Help

read
is
has
rather
that

---

From: Support <[email-masked]>
Date: Thu, 29 Aug 2002 15:31:04 +0000

I agree, sort of...  Because all my other graphs seem OK (at least
there is no compelling - glaring reason to think they are wrong) my
gut reaction is that the MRTG is working OK and communicating
with the 3Com pool Ok and everthing, and it's just adding one user
for a telnet session, or it's seeing the last user not released yet
while the next user is already connected or something.

I had assumed that the Pool was reporting a telnet session or
something as an extra active connection, so that when the card
was full with 24 real people online, it was reporting 25 to MRTG
and MRTG saw that as an unknown value.  Maybe this is a bad
assumption.  In any case, a guy shouldn't assume things.  (you
know what they say about AssUMeing things.)

I had also thought that setting Max Modems to 26 or 28 wasn't a
good idea since that would put a 26 or 28 along the left side of the
graph, and would throw all the Percentages out, but that may be the
better thing, at least to test.  This way we can see if the peak modems
in use are 25 or 26 or 27 and see how far out the reported numbers
are.  If my average users are 16, I want to know that it IS actually 16
and not really 14 plus two telnet sessions.

Thanks, I've done that and we'll wait a while and see how it looks.  :-)
http://www.inet2000.com/mrtg/3comtc.modem.html

Todd Chamberlain

----- Original Message -----
From: "Craig Gittens" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, August 29, 2002 6:30 AM
Subject: RE: [USR-TC] MRTG Help

> > > I would try changing the MAX users setting to 26 or 28. It can be
found
> in
> > > > > Hi.  My MRTG graph
(http://www.inet2000.com/mrtg/3comtc.modem.html)
> > > shows

---

From: Lewis Bergman <[email-masked]>
Date: Thu, 29 Aug 2002 21:16:47 +0000

On Thursday 29 August 2002 07:30 am, Craig Gittens wrote:
> unknown reading? That would bug the h311 out of me. For example, suppose
> you get a reading of 6 and then get that flat 6 reading for the next two
> hours because it suddenly jumped to 25 or some bug kicked in. You have no
> reliable stats. Yes, your graphs looks prettier now but not necessarily
> correct. If you notice you have quite a few flat lines in it which indicate
> it did it's unknown number for at least 15-20 minutes. That is an
> indication that something is wrong.
The real solution of course is to buy enough ports so that one or two telnet 
users doesn't mean your graphs are totally whacked. These things are chep 
enough now, just buy another.
-- 
Lewis Bergman
Texas Communications
4309 Maple St.
Abilene, TX 79602-8044
915-695-6962 ext 115

---

From: Support <[email-masked]>
Date: Fri, 30 Aug 2002 16:44:46 +0000

Hey, do you have a deal with my Telco on C-T1 lines too?  :-)  The HiperDSP
cards are cheap, but the Telco pricing isn't.  Actually, I have two HDSP
cards, but only have one C-T1 right now.

Some of the pricing here is based on escalation the more you have, so we
have to combine some ports on T1, some on BRI and some on Analog Centrex
lines.  For example, if you call up the telco and want 1 BRI line (so two
telephone or modem channels) the BRI line will be $74.50 each (so $37.25 per
line).  As you would normally expect, prices get a bit cheaper for 2 lines
or 5 lines, so that they eventually get down to $62.50 per BRI (just $31.25
per modem line) for 10 BRI lines.  However, if you want 11+ lines, suddenly
ALL the lines (including the first 10) go up to $150 per line.  SaskTel says
this is due to trunk loading (Yea, that's a load alright) but it's obviously
to keep competative ISP's in the red.  10 lines cost $500 / month, but 11
lines would cost $1,650 / month.  That's a pretty expensive 11th line!

The same thing with Centrex lines - they start off affordable, but if you
get to many in the same hunt group, the price escalates to make it
unaffordable.  So, we have 10 BRI lines (20 modems), 16 Centrex lines and a
T1 (24 modem ports) for 60 ports.

---

From: Lewis Bergman <[email-masked]>
Date: Fri, 30 Aug 2002 18:00:06 +0000

And I thought we had it bad with South Western Bell. They are anti
compettitive and illegally bundeling regulated and non-regulated services
but they aren't that blatent.

What country are you in?

---

From: Support <[email-masked]>
Date: Fri, 30 Aug 2002 20:03:03 +0000

Guess.  The government is totally charge and squashes any innovation by the
private individual.  Russia of course.   :-)

No sorry, actually Saskatchewan, Canada.  SaskTel is our Government run
Telco.
Todd Chamberlain