Thursday, June 6, 2013

ASP.Net Webservice slow to start

This may be the smallest needle ever found... in the largest field of haystacks you can imagine.  It took us two months to track down... let me describe it to you:

IIS 7.5
Dot net framework 4.5
Windows Server 2008 R2 Web
ESXi 5.0

We run a web service from Pitney Bowes that does address validation and gives us back insurance taxing information.  Pretty important and maybe the only consolidated source of that info from one web service.  They are solid,  but a few months ago we found that when the service hadn't been used in a while (called via WCF from our program), the initial request would take 60-90 seconds to complete. After that,  sub-second response times.

Do a google search on something like [ wcf slow first startup] and you will find post after post of people having the problem,  and coming up with workarounds. No one was really fixing the problem though.  In our case,  we had recently virtualized the servers so of course a lot of fingers were pointing that way.

We also looked into DNS -- you know, maybe we weren't resolving very quickly and had to wait for a secondary server to give us the answer. Ran the best practice analyzer, made a few changes... nope.

We looked into the IIS web farm and modified the recycle times... blah, blah, blah. Nope.

We looked into changing default proxy settings (we don't use a proxy) in web.config. nope.  Interestingly enough we were blasted with Microsoft's sneaky push of IE10, that happens to control the "proxy" part of things. Not the problem.

Then I happened across a related post that talked about a setting that might help us out.  It ends up that back before 2008 microsoft starting using their server named (certificate revocation list) to verify data... but that server didn't exist anymore.  This post shows some config settings to get around it.  READ IT -- it's good. [here]

Since these are production boxes that don't like config changes too often,  we took a different approach.  By adding a static entry to our etc/hosts file on the IIS server, we didn't need the config change and the problem went away.

We added
Problem resolved. 

OK -- this is one of those WTH moments.  or maybe it's an OMG moment.  Whatever. Now on a side note,  our Exchange 2010 server will exhibit the same characteristics sometimes so we added that same static DNS entry.

So everything is better.  The service starts up first and every time in sub second response. Pitney Bowes had no idea what to do. Microsoft had no idea what to do. Bloggers had no ideas... but a little IT shop in Venice, FL figured it out.