IIS Website performance issues under load

[schuits]

This is a long shot but here goes.

We have a RESTFul website on IIS thats been running fine until last Thursday.
And then suddenly became very slow when multiple users are accessing it. Timeout issues all the time.
1.2ms requests in the test environment are taking 29ms or more in Live, that's an API call that returns 6 records only.

This is what we've done:
- Log into the website from the server hosting it. Speed issues persist. This indicates to me it's not a network issue.
- Requests on the database itself are fine. This indicates the database is performing normally.
- Services and other jobs running on the web server are fine. This indicates it's not a database connection issue.
- Rolled back to the previous version of the website. Nothing changed. This indicates the website code itself is not the issue.
- Monitored CPU, Memory and Network on the server and infrastructure. Everything is nominal.
- Recycled the app pool and checked the settings.
- Created a copy of the website on a seperate app pool. When a single user logs on the site if super fast.
We then switched all users to this copy and the speed went down to a crawl.
- Restricted website access to only company internal users. The site is slightly faster due to the limited number of users, but not as fast as normal.
- Checked for any software updates that were done, there were none.
- Increased the allowed threads per CPU, no effect.
- Rebooted the server, checked for viruses, checked for DDos attacks. Checked the drives are defragged.
- ReIndexed the database and rebooted. Even though the database is clearly not the issue.

It really looks like the website/IIS are where the issue lies, only we don't know what.

Does anyone have and tips?

 

[Solitude]

I had something like that a year or so back. It was caused by an external library. Once I removed it, everything was fine. I don't know why it caused it since everything looked normal but once a lot of users hit the site it slowed down to a crawl. Luckily the library was added just before it all started so it was easy to find the problem on my side.

 

[schuits]

I had something like that a year or so back. It was caused by an external library. Once I removed it, everything was fine. I don't know why it caused it since everything looked normal but once a lot of users hit the site it slowed down to a crawl. Luckily the library was added just before it all started so it was easy to find the problem on my side.

Thanks. I can see how that would cause a problem. Unfortunately we haven't added any, and even did a rollback of the latest release to ensure it wasn't a code change causing the problem.

 

[schuits]

Ok for now we have a workaround in place.
We've denied internal access to the site via HTTP and the IP address. Or HTTP and the site name.
All users have to access the external site via HTTPS.

Something with the HTTP access is causing problems which we'll need to investigate.

I think a drink or two are in order now.