[kwlug disc.] Tough server debug question

[Insurance Squared Inc.]

I've got a server issue that I've been unable to fix - worse I don't 
even know where to really start without spending days and days testing.  
Linux forums didn't get it either :).

What's happening as best I can tell is that apache is leaving threads 
open.  I have a database program that is a black box program (from a 
vendor).  I have a php program that does a few things, the main part of 
which is calling the black box program.  Apache is calling the php 
program with a bunch of inputs (from a POST form on websites).  So it 
goes webpage-->apache-->php script-->black box cgi.

Occassionally the apache threads stay open.  Maybe only one or two a 
day, but it's cumulative.  The threads never shut down and they burn 
almost 100% of CPU resources.  End result is I have to reboot apache 
every day or two or the server bogs down.

This is bizarre because I don't think apache would normally leave a 
thread open like that.  And I don't recall any changes to the php script 
or the cgi program recently. 

Other symptoms that don't really help me diagnose the problem:
- when I edit the php script with nano on the server and page down, it 
gives me a glibc error and exits.  Works fine when editing in vi.  Like 
I've got an ugly character or something in the script.  But again, no 
updates to it in a long time.
- the php script and the black box program accept a variety of inputs.  
The threads always seem to lock up when a specific set of variables is 
sent to it (i.e. there's a set A of inputs and set B, apache leaves the 
thread open only when set B of inputs are received).  But I don't think 
it's always when set b is called. Not confirmed, but I think it's 
actually only a small number of times when set B is used that it locks up.

I honestly don't expect a concise answer as to where the problem is :).  
But I'm really stumped as to where to even start looking.  Anyone have 
any thoughts on what I should be looking at to isolate or pin down 
what's going on?