Subject: [USR-TC] "the harvard problem"

From: [email-masked]
Date: Thu, 07 Mar 2002 16:42:40 +0000

Ok, here's my candidate for weird problem of the year.  I'll try to be
brief, but please don't confuse that with a lack of having tried a billion
things to figure out what's going on.  :)

Our users are unable to connect to a VERY SMALL number of IP hosts when
they connect to our HiperDSP/HiperARC Chassis.  For instance, it seems
that we cannot open www.harvard.edu in a web browser.  Don't confuse this
with an inability to open any web site, far over 99% seem to work just
fine, including www.law.harvard.edu, www.hup.harvard.edu,
www.med.harvard.edu... the list goes on and on (by the way, we have no
special affiliation with harvard (like a peering arrangement), from our
point of view they are just a random web server).  In fact, I'm only sure
of three Web servers on the whole internet that we have problems with (I'm
sure there are more, but those are the only ones I know).  Those three
display the same symptoms every time: a web browser hangs on "host
contacted, waiting for reply..."  And never gets through.

Ok, to make sure you believe this is weird, we can access these sites no
problem from workstations on the same ethernet as the TC Chassis.  We can
access the sites no problem from dialup connections to an old PortMaster
with analog lines in the back (also on the same ethernet).  And in fact I
created a special user in the hiperarc, dialed up, logged in as that user,
and had no problems.  I thought it might be related to the IP address
(like the web server being unable to reverse lookup the address), but no,
I can connect using a specified IP address using an account set up in the
hiperarc, and it works fine, but if I get the same IP address after RADIUS
authentication, no dice.

SO, the obvious candidate for the problem is the RADIUS server, which is
cistron, just upgraded because of a security problem (making it look even
more suspicious).  My problem is, I have no idea what kind of information
the RADIUS server could be sending to the HiperArc to cause this problem.
I mean, I couldn't cause this problem if I wanted to!  I couldn't even
cause anything similar to this problem!  Furthmore, log information in the
"detail" file looks essentially identical regardless of how the connection
is authenticated...  I've got those, as well as output from "monitor
radius" and the like, if anyone really wants to dig deep.  But naturally,
what I'm really hoping for is just that someone will say "OH, it sounds
like a such-and-such kind of problem, try that".

Thanks for even reading about my woes, and especially for any thoughts you
may be able to share...

-- 
David Ernst
HoosierNet, Inc.

---

From: Dave Lajoie <[email-masked]>
Date: Thu, 07 Mar 2002 17:05:19 +0000

Dave,

	I have had a problem 'like' this, may or may not be the same issue
but solution for me was to change MTU setting from 512 to 1514 and it
cleared up the issue. This was some time ago and am not quite sure how to
address setting changes for MTU setting in cistron but you could check to
see what it is assigning for MTU size. Its a long shot and just off the
top of my head, hope it helps.

Dave Lajoie
RA Administrator
North Country Internet Access
E-Mail: [email-masked]

---

From: Robert <[email-masked]>
Date: Thu, 07 Mar 2002 17:21:26 +0000

I'm guessing this is more likely due to your network configuration. Are you
assigning global IPs? Are you subnetting part of a class C? Does it happen
with all dialup connetions, or just slow (below 33.6) ones?

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Thursday, March 07, 2002 11:42 AM
Subject: [USR-TC] "the harvard problem"

> _______________________________________________
> USR-TC mailing list
> [email-masked]
> http://mailman.xmission.com/cgi-bin/mailman/listinfo/usr-tc

---

From: Jeff Garvas <[email-masked]>
Date: Thu, 07 Mar 2002 10:36:03 +0000

Hey folks,

I'm happily using ICRADIUS, a hack of cistron with MySQL for the backend.

We've been hacking an application to communicate directly with the MySQL 
backend to populate user/password information.

My question is rather simple for those of you familiar with the 3COM/USR 
RADIUS Dictionary.

Is there any Attribute I can set with a text message or an appropriate way 
to deny authentication and/or connectivity while indicating to the caller 
that their account has been deactivated?

We've tried creating our own field of "DEACTIVATED" (to make it obvious to 
the lower end technicians in the log file) but we came across a funny 
problem.

If you remove the Password attribute and replace it with "DEACTIVATED" the 
system will complain that the attribute is unknown, and then try each 
RADIUS server (one after the other, going back and forth) six times, until 
it gives up.

If you put the Password attribute back in with a "munged" password, it 
complains about the attribute DEACTIVATED once, and then hangs up on them 
because the password doesn't match.

Is there a way to send any more information easily by just modifying the 
Attributes in RADIUS, or is my only practical (and easily implemented) 
solution to just munge the password field?

-Jeff Garvas

---

From: Jeff Mcadams <[email-masked]>
Date: Thu, 07 Mar 2002 17:51:34 +0000

Also sprach Dave Lajoie
>cleared up the issue. This was some time ago and am not quite sure how
>to address setting changes for MTU setting in cistron but you could
>check to see what it is assigning for MTU size. Its a long shot and
>just off the top of my head, hope it helps.

Indeed...reading through the description, I was thinking the same
thing...that it was an MTU issue.

Indeed, I seem to remember that in a previous discussion that turned out
to be an MTU issue, www.harvard.edu was even mentioned as a troublesome
site!

Anyway...for some more information on the subject...this is rather
detailed and I suspect that many of you understand all of this...but for
any that don't...this should explain how MTU settings on a dial-up link
can become a hard problem to track down like this.

MTU setting with RADIUS authentication is a bit on the odd side.  That's
because MTU (actually, in PPP MRU is negotiated, not MTU) is negotiated
during LCP, which runs *before* any authentication protocols happen.

So the deal is that the customer connects, MTU is negotiated to 1500
most likely (or possibly 1514, 1524, something like that, but at least
1500) during LCP, then you get the authentication information (from PAP
or CHAP or whatever), send off a RADIUS authentication request, get the
reply which tells the Arc to set the MTU (note...not MRU) to a smaller
value (576 is a common one).

OK...here's where it gets tricky.  The MRU negotiated in PPP from the
customer is likely 1500 or higher...meaning that the customer's computer
is saying, "don't send me any packets larger than 1500 bytes" (or
whatever value is negotiated).  When the RADIUS authentication response
comes back, it's telling the Arc "don't send any packets over this link
larger than 576 bytes" (or, again, whatever the value is set to).
So...the Arc can comply with the RADIUS response value without violating
the value negotiated in LCP, so there is no reason to go back and
re-open LCP to renegotiate a new value.  So now we have the Arc set to
not send any packets larger than 576 bytes, and the customer's computer
set to not receive anything larger than 1500 bytes, which won't be a
problem 'cause the Arc won't ever send it anything larger than 576
bytes.  The trick is, the customer doesn't know about the 576 byte limit
that the Arc has in place (and in theory, it shouldn't *need* to know).

So, what does all of this mean?  Well, when the customer tries to
connect to www.harvard.edu, the TCP SYN packet that the customer's
computer sends out likely has the Maximum Segment Size option set (or it
may not, the result will be the same) to 1500 bytes ('cause the
customer's computer thinks it can receive packets up to 1500 bytes
'cause that's the MRU value negotiated in PPP).  So the Harvard web
server either immediately sends packets up to 1500 bytes in size, or
very quickly ramps the packet size up to 1500 bytes, but certainly
larger than 576 bytes.  The Harvard web server also almost assuredly
sets the "Don't Fragment" bit in the IP header of the packets its
sending out.

So these packets that are larger than 576 bytes, but not larger than
1500 bytes traverse the Internet, get to your Arc, and the Arc says, "I
can't send this packet across this link because I have the MTU set to
576".  So the Arc drops the packet on the floor, and generates an ICMP
Destination Unreachable - Fragmentation needed but DF (Don't Fragment)
bit set message and sends it back to the IP address for Harvard's web
server.  This ICMP message would tell Harvard's web server that there is
a link somewhere along the path of the packet that it sent out that
can't handle packets larger than 576 bytes...ideally, Harvard's web
server would get this packet back and resend the data, but break it up
into smaller chunks that will fit into 576 byte packets.

The problem is that Harvard is either blocking these ICMP packets at
some sort of packet filter or firewall in front of their web server, or,
more likely (since you're not having problems with other web servers
within Harvard's network) www.harvard.edu is some sort of load-balanced
cluster of web servers that cannot handle ICMP messages like this
correctly.  Regardless of the cause, the ICMP message that your Arc is
sending out is never getting back to the Harvard web server, so it never
learns that it needs to send smaller packets, so it keeps on sending the
larger ones, which your Arc keeps dropping on the floor, so no data gets
through at all.

Solution...remove the MTU settings from the RADIUS server's user config,
or set it to a value of 1500 or higher (1500 is the MTU value of
ethernet, so setting it higher doesn't really get you anything).
Another possibility is to let Harvard know that they either have a
firewall or a load-balancer on their web servers that are breaking Path
MTU Discovery and that they should fix it.  The problem with the second
solution is that Harvard may not be willing to fix it, and you almost
assuredly will run into this problem with other sites on the 'net and
its probably just easier to set your MTU's higher than trying to fight a
crusade like this on the 'net.  :)

>> Ok, to make sure you believe this is weird, we can access these sites
>> no problem from workstations on the same ethernet as the TC Chassis.
>> We can access the sites no problem from dialup connections to an old
>> PortMaster with analog lines in the back (also on the same ethernet).
>> And in fact I created a special user in the hiperarc, dialed up,
>> logged in as that user, and had no problems.

I've pulled this part of your message out to point out why all of these
things worked when the RADIUS authenticated user on the Arc didn't.

Workstations on the ethernet obviously have not link with an MTU of 576
between them and the Harvard web server.

Old PortMaster ComOS code (as well as the 3Com NETServer code that was
derived from it) didn't pay any attention to the MTU values returned in
a RADIUS authentication response as they had already negotiated MRU
during LCP and didn't have any support for changing it in the middle of
a session.

A user created in the Arc's local user table most likely didn't have the
MTU setting that the user record in your RADIUS server had, so the MTU
remained set to 1500 and traffic flowed normally.
-- 
Jeff McAdams                            Email: [email-masked]
Head Network Administrator              Voice: (502) 966-3848
IgLou Internet Services                        (800) 436-4456

---

From: [email-masked]
Date: Thu, 07 Mar 2002 19:16:45 +0000

Yes.  Thank you.  All of those who guessed "MTU", you were correct.  The
default entry in the new Cistron Radius was indeed set to 576, I changed
it to 1500 and all appears to be better.  They oughta rethink that
default.  :)

Thanks very much, folks!

David

-- 
David Ernst
HoosierNet, Inc.

---

From: Paul Farber <[email-masked]>
Date: Thu, 07 Mar 2002 21:17:03 +0000

I ran into a similiar problem... I had just changed the RADIUS attribute 
Framed-MTU from 1500 to 576.

TC equipment would not get to several web sites (www.iwon.com, 
www.psecu.com, www.weather.com etc) but our patton gear worked fine (both 
use the same auth/acct server).

Since 2 /24's were assigned to the TC's and 1 /24 to the Pattons I figured 
that we may have been filtered for some reson.  Called some admins at 
PSECU (I bank there) and they verified thier ACLs, not the problem.

RESET THE FRAMED-MTU TO 1500 AND THE TC'S WORKED TO ALL SITES AGAIN.

Not sure if it is a fragmentation bit bug, a RADIUS bug in the ARC code 
or a setting on the ARC that I needed to change... I just reset the MTU 
size in the USERS file and it all worked again.

-- 
Paul Farber
Farber Technology
[email-masked]
Ph  570-628-5303
Fax 570-628-5545

---

From: Mike Andrews <[email-masked]>
Date: Fri, 08 Mar 2002 00:03:26 +0000

Sure would be nice if we could just cluebat all the people that run
websites that have that problem.  Point them all to a site like

Hint to firewall administrators: DO NOT BLINDLY BLOCK ALL ICMP TRAFFIC.
It's a dumb idea.  Some ICMP is essential.  This exact problem is why.

See http://www.worldgate.com/~marcs/mtu/ for a good read on the subject.

Mike Andrews * [email-masked] * [email-masked] * http://www.bit0.com
VP, sysadmin, & network guy, Digital Crescent Inc, Frankfort KY
Internet access for Frankfort, Lexington, Louisville and surrounding counties
It's not news, it's Fark.com.    "With sufficient thrust, pigs fly just fine"

---

From: Jeff Mcadams <[email-masked]>
Date: Fri, 08 Mar 2002 02:09:20 +0000

Also sprach Paul Farber
>size in the USERS file and it all worked again.

Its not really a bug...per-se.  More a botched configuration really...at
least for firewall configs that block all ICMP.  As Mike mentioned,
block *all* ICMP is a "Bad Thing(tm)".  ICMP was defined as a protocol
for a reason, and the healthy functioning of TCP in particular is
dependant upon it in several ways.
-- 
Jeff McAdams                            Email: [email-masked]
Head Network Administrator              Voice: (502) 966-3848
IgLou Internet Services                        (800) 436-4456

---

From: Jeff Mcadams <[email-masked]>
Date: Fri, 08 Mar 2002 02:11:16 +0000

Also sprach Mike Andrews

>It's a dumb idea.  Some ICMP is essential.  This exact problem is why.

The problem is that its not always just firewalls that cause the
problem.  Load-balancing software and appliances (BigIP, Cisco
LocalDirector, etc.) are notorious for not handling ICMP Fragmentation
needed but DF set messages correctly.  Those are a bit harder to deal
with than firewall configs.  :/
-- 
Jeff McAdams                            Email: [email-masked]
Head Network Administrator              Voice: (502) 966-3848
IgLou Internet Services                        (800) 436-4456