Response message: Non HTTP response message: Too many open files

[Nikolay_Miroshnichenko]

Hi,
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem
is. Whether it the server declines or JMeter or JVM problem. Looking at
the error messages in responses (see below) I'd say that JVM (or rather
OS, Debian Lenny 64) meets its limits...

*****

Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files

*****
And a valid answer looks like this:

****

Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK

*****

Thanks.

Nikolay Miroshnichenko

[Deepak Shetty]

Hi
It looks like your server has reached its limit . you probably need to
change ulimit on the server. When your requests fail you can probably check
the number of open sockets on the server (using netstat or equivalent). .
there are other OS related parameters related to how long the socket should
be kept open even after it is closed which affect OS behavior

regards
deepak

On Tue, Oct 13, 2009 at 2:45 PM, Nikolay_Miroshnichenko

Post by Nikolay_Miroshnichenko
Hi,
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem is.
Whether it the server declines or JMeter or JVM problem. Looking at the
error messages in responses (see below) I'd say that JVM (or rather OS,
Debian Lenny 64) meets its limits...
*****
Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files
*****
****
Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
*****
Thanks.
Nikolay Miroshnichenko
---------------------------------------------------------------------

[Nikolay_Miroshnichenko]

Post by Deepak Shetty
Hi
It looks like your server has reached its limit . you probably need to
change ulimit on the server. When your requests fail you can probably check
the number of open sockets on the server (using netstat or equivalent). .
there are other OS related parameters related to how long the socket should
be kept open even after it is closed which affect OS behavior
regards
deepak
On Tue, Oct 13, 2009 at 2:45 PM, Nikolay_Miroshnichenko

Post by Nikolay_Miroshnichenko
Hi,
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem is.
Whether it the server declines or JMeter or JVM problem. Looking at the
error messages in responses (see below) I'd say that JVM (or rather OS,
Debian Lenny 64) meets its limits...
*****
Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files
*****
****
Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
*****
Thanks.
Nikolay Miroshnichenko
---------------------------------------------------------------------

So it is a server problem, not a test generator.
Thank you!

[Deepak Shetty]

Id verify it by checking the server logs (or number of open sockets)
Im not sure what the jmeter error log would look like if it was the jmeter
that ran out of file descriptors (for the sockets)

If you cant check the server logs/dont have access to it , an easy to verify
is run the same test (but with only half the number of threads) on two
client machines i.e. run jmeter from too different machines at the same
time. If its a server issue then you should see the same errors.



On Tue, Oct 13, 2009 at 3:02 PM, Nikolay_Miroshnichenko

Post by Nikolay_Miroshnichenko

Post by Deepak Shetty
Hi
It looks like your server has reached its limit . you probably need to
change ulimit on the server. When your requests fail you can probably check
the number of open sockets on the server (using netstat or equivalent). .
there are other OS related parameters related to how long the socket should
be kept open even after it is closed which affect OS behavior
regards
deepak
On Tue, Oct 13, 2009 at 2:45 PM, Nikolay_Miroshnichenko
Hi,

Post by Nikolay_Miroshnichenko
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem is.
Whether it the server declines or JMeter or JVM problem. Looking at the
error messages in responses (see below) I'd say that JVM (or rather OS,
Debian Lenny 64) meets its limits...
*****
Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files
*****
****
Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
*****
Thanks.
Nikolay Miroshnichenko
---------------------------------------------------------------------

So it is a server problem, not a test generator.
Thank you!
--
Best Regards
Nikolay Miroshnichenko
---------------------------------------------------------------------

[sebb]

Post by Deepak Shetty
Id verify it by checking the server logs (or number of open sockets)
Im not sure what the jmeter error log would look like if it was the jmeter
that ran out of file descriptors (for the sockets)

Exactly as shown ...

Post by Deepak Shetty
If you cant check the server logs/dont have access to it , an easy to verify
is run the same test (but with only half the number of threads) on two
client machines i.e. run jmeter from too different machines at the same
time. If its a server issue then you should see the same errors.

Good advice.

Post by Deepak Shetty
On Tue, Oct 13, 2009 at 3:02 PM, Nikolay_Miroshnichenko

Post by Nikolay_Miroshnichenko

Post by Deepak Shetty
Hi
It looks like your server has reached its limit . you probably need to
change ulimit on the server. When your requests fail you can probably check
the number of open sockets on the server (using netstat or equivalent). .
there are other OS related parameters related to how long the socket should
be kept open even after it is closed which affect OS behavior
regards
deepak
On Tue, Oct 13, 2009 at 2:45 PM, Nikolay_Miroshnichenko
Hi,

Post by Nikolay_Miroshnichenko
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem is.
Whether it the server declines or JMeter or JVM problem. Looking at the
error messages in responses (see below) I'd say that JVM (or rather OS,
Debian Lenny 64) meets its limits...
*****
Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files
*****
****
Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
*****
Thanks.
Nikolay Miroshnichenko
---------------------------------------------------------------------

So it is a server problem, not a test generator.
Thank you!
--
Best Regards
Nikolay Miroshnichenko
---------------------------------------------------------------------

[sebb]

Post by Deepak Shetty

Post by Deepak Shetty
Hi
It looks like your server has reached its limit . you probably need to
change ulimit on the server. When your requests fail you can probably

check

Post by Deepak Shetty
the number of open sockets on the server (using netstat or equivalent). .
there are other OS related parameters related to how long the socket

should

Post by Deepak Shetty
be kept open even after it is closed which affect OS behavior
regards
deepak
On Tue, Oct 13, 2009 at 2:45 PM, Nikolay_Miroshnichenko

Post by Nikolay_Miroshnichenko
Hi,
we're testing a web site under 1000 users (threads) load.
Currently our servers hold about 500 simultaneous users with great
latencies.
With more users the requests fail. But I'm not sure where the problem

is.

Post by Deepak Shetty

Post by Nikolay_Miroshnichenko
Whether it the server declines or JMeter or JVM problem. Looking at the
error messages in responses (see below) I'd say that JVM (or rather OS,
Debian Lenny 64) meets its limits...
*****
Thread Name: NewUsers 1-664
Sample Start: 2009-10-10 22:09:31 MSD
Load time: 0
Latency: 0
Size in bytes: 1279
Sample Count: 1
Error Count: 1
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Too many open files

"Non HTTP" means that the error is detected by JMeter, not generated
by the server.

Post by Deepak Shetty

Post by Deepak Shetty

Post by Nikolay_Miroshnichenko
*****
****
Thread Name: NewUsers 1-778
Sample Start: 2009-10-10 22:08:33 MSD
Load time: 58159
Latency: 58130
Size in bytes: 611787
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
*****
Thanks.
Nikolay Miroshnichenko

---------------------------------------------------------------------
So it is a server problem, not a test generator.

No, it's a problem on the JMeter host - it has run out of sockets.

Either make more sockets available, or use fewer threads on that host
(and some more on another host).

Post by Deepak Shetty
Thank you!
--
Best Regards
Nikolay Miroshnichenko
---------------------------------------------------------------------