(racoon 807) racoon fails to get the subjectName of the peers certificate

[Roland Dirlewanger <Roland.Dirlewanger@dr15.cnrs.fr>]

Hi,

I'm using ipsec-tools 0.3.3 on a Linux Fedora Core 2 box. I tried to 
setup racoon in order to accept road-warriors connexions from Windows XP 
or 2000 clients. The clients authenticate themselves using X509 
certificates.

When I connect from a Windows host, the ISAKMP-SA cannot be established. 
It fails with the following message :

2004-10-14 21:55:09: ERROR: 15556:error:0D0680A8:asn1 encoding 
routines:ASN1_CHECK_TLEN:wrong [three more lines follow]
2004-10-14 21:55:09: ERROR: failed to get subjectName

I tried to find out why the certificates sent by the Windows boxes 
cannot be decoded by racoon. I added the following code to the function 
mem2x509 in crypto_openssl.c in order to dump the client certificate 
into a file /tmp/cert :
       int fd = creat("/tmp/cert", 0x644);
       write(fd, cert->v, cert->l);
       close(fd);
and tried again.

I watched through the resulting /tmp/cert file using "openssl asn1parse 
-inform der < /tmp/cert". I noticed that /tmp/cert does not containg an 
X509 certificate but a PKCS#7 object which is in fact, the certificate 
chain leading to the certificate of the Windows machine.

It seems that racoon is really expecting a single certificate sent by 
the peer and cannot handle a PKCS#7 certificate chain. Note that I been 
been using the very same certificates in order to setup road-warrior 
connexions from Windows to FreeSwan for a more than a year.

Is there something I can configure either on WIndows or on racoon.conf  
in order  to make racoon work with my certificates ?

Thank you very much in advance.

Roland.

--
Roland Dirlewanger <Roland.Dirlewanger@dr15.cnrs.fr>
CNRS - Délégation Aquitaine et Poitou-Charentes
Esplanade des Arts et Métiers - BP 105
33402 TALENCE CEDEX
FRANCE

Tél: 05 57 35 58 52, Fax: 05 57 35 58 01