Subject: Re: [USR-TC] UST TC1000 and 'Level 3 alarm'

From: Steve Johnson <[email-masked]>
Date: Sun, 10 Feb 2002 18:48:27 +0000

(I meant to send this to the list last week and accidently sent it to just 1 
person)

Yeah the PRIs look clean.

As we are investigating this more, it looks like the ARCs are not accepting 
the calls.  What will happen is the arc will dump all calls on the box, then 
the first channel of the first PRI will take a call and then drop it, and 
continue to do this, while this 1 call is in progress any calls after that 
get the all circuits are busy message.

So I'm wondering if this is something weird with the ARCs.

Now I have 4 Boxes for 1 POP 2 boxes for another, on Monday all 4 boxes at 
this POP had this problem, almost in sequential order, it may just seem like 
it was in order because of the hunting.  So now I'm wondering if there is a 
DoS for hyper arcs that no one is aware of yet, and that I am being hit by 
them.  

It is real strang that 5 our 8 boxes have had this problem, all running the 
same firm ware on the ARCs, NMCs and DSPs...

Also I considered the possibility that someone got ahold of my SNMP strings 
and is monkeying around with the boxes, so today I changed all the 
communities.

I'm still at a loss tho, and the CEO is getting upset with the issue. 

-Steve

On Tuesday 05 February 2002 12:56 pm, you wrote:
> Steve Johnson wrote:
> > I've blamed the CO and they tell me its us.. I also found that reseting
> > the D chanels or reset the card (which resets the d in the process)
> > clears the problem
>
> You forget the first three rules of telecom school. Blame the CPE, blame
> the CPE, blame the CPE.
>
> I went through a problem with WorldCom were my Total Control was showing
> bi-polar errors. Get the WorldCom techs on the phone, sir there's
> nothing wrong with that T1, nothing wrong with our switch, nothing wrong
> with our network,  blah blah blah blah.
>
> Moved the circuits to another provider and bingo, bi-polar errors went
> away.
>
> Have you checked the lines for errors? Sometimes a switch will take down
> the D channel if the line is not clean.
>
> -Ron

---

From: Charles Sprickman <[email-masked]>
Date: Sun, 10 Feb 2002 17:52:04 +0000

On Sun, 10 Feb 2002, Steve Johnson wrote:

What are all the software revs again?  Are these DSPs or quads or both?

> get the all circuits are busy message.

When this is happening, what does a "list int" on the arc show?  Is
everything up/up?  If not, does a "list chassis" show that the arc owns
all the cards it should?

When did this start?  Is this a new installation, new software?

> DoS for hyper arcs that no one is aware of yet, and that I am being hit by

Do you know what type of switch the telco is using?  Lucent or Nortel?
Maybe if you post that info we can do some comparisons on the PRI setup.

> same firm ware on the ARCs, NMCs and DSPs...

All the same telco?

As to the hacking/DoS possibilities, I'd guess that if the ARC was being
hammered by something, you'd see some sort of lag or weirdness when
telnetted into the arc.  I assume you've rebooted lately?  We're running
4.2.32, and every 3-4 months we have to reboot the arcs or they start
doing all sorts of odd things...

Charles

> _______________________________________________
> USR-TC mailing list
> [email-masked]
> http://mailman.xmission.com/cgi-bin/mailman/listinfo/usr-tc
>

---

From: Steve Johnson <[email-masked]>
Date: Mon, 11 Feb 2002 01:36:14 +0000

> everything up/up?  If not, does a "list chassis" show that the arc owns

I hadn't done that, next failure I will do that and post the info..

> When did this start?  Is this a new installation, new software?

About a week ago, and no this install has been around for years, and the
last firmware upgrade was done about 6 or so months ago.  This all started
about 1 week ago, and no changes have been made on our end.

> Maybe if you post that info we can do some comparisons on the PRI setup.

5ESS, Is that Nortel?  I'm not sure..

> > same firm ware on the ARCs, NMCs and DSPs...

No.  4 on PacBell and 1 on ATG.  Both are 5ESS switches.

> doing all sorts of odd things...

Sometimes the only fix this this problem is a reboot.. So as
of today 4 boxes needed rebooted for this problem.

Thanks for the help =) I'll keep you posted next time it fails.

-Steve

-- 
Steve Johnson <[email-masked]>              Sonic.net, Inc.
System Administrator                            300 B Street
707-522-1000 (voice)                            Santa Rosa, CA 95404
707-547-2199 (fax)                              http://www.sonic.net