HTTPHostConnectionException in jmeter, but I do see a request served successfully in access log

[Manish Sapariya]

Hi,
I am completely baffled by this.

I see following exception for some request. However the request
is successfully served as per the access log of httpd.
======================

org.apache.http.conn.HttpHostConnectException: Connection to
http://arabic.gslab.com refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:286)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1088)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1077)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 12 more

======================

My jmeter setup:
- distributed with one gui and other 10 nodes.
- Each node runs 50 users.
- GUI node do not run the test.
- CentOS 64 bit, using java 64 bit
- The time is synchronized on all machines.

I would appreciate any pointers to debug this any further.

Thanks and Regards,
Manish
kPoint wins *Global eLearning
Award<http://www.kpoint.com/kpoint-wins-learntech-global-e-learning-award/>
* in “Learning Technologies Solution”!

[guru]

Hi Manish, I am wondering if you have fixed this issue because I am facing
similar issue.

We triggered 250 create class events and only 50 events got success and 200
failed showing various error in jmeter like:

a) org.apache.http.conn.HttpHostConnectException: Connection to
https://www.abc.com refused

b) javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at
com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(Unknown
Source)

c) org.apache.http.NoHttpResponseException: The target server failed
to respond

at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)




--
View this message in context: http://jmeter.512774.n5.nabble.com/HTTPHostConnectionException-in-jmeter-but-I-do-see-a-request-served-successfully-in-access-log-tp5718190p5718199.html
Sent from the JMeter - User mailing list archive at Nabble.com.

[sebb]

Post by Manish Sapariya
Hi,
I am completely baffled by this.
I see following exception for some request. However the request
is successfully served as per the access log of httpd.
======================

Are you sure it's the same connection?
That does not seem possible.

If the connection is refused, how can the server handle it?

Post by Manish Sapariya
org.apache.http.conn.HttpHostConnectException: Connection to
http://arabic.gslab.com refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:286)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1088)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1077)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 12 more
======================
- distributed with one gui and other 10 nodes.
- Each node runs 50 users.
- GUI node do not run the test.
- CentOS 64 bit, using java 64 bit
- The time is synchronized on all machines.
I would appreciate any pointers to debug this any further.

Double (triple) check which connections are succeeeding and which are failing.

Try using non-GUI mode on the client node.

I'm suprised you need 10 servers for a total of 500 users.
Depending on the test plan, a single node can handle that.

But perhaps you want to test from multiple IPs.

It might be worth trying a single non-GUI test with 500 users and no servers.

Post by Manish Sapariya
Thanks and Regards,
Manish
kPoint wins *Global eLearning
Award<http://www.kpoint.com/kpoint-wins-learntech-global-e-learning-award/>
* in “Learning Technologies Solution”!

[Manish Sapariya]

Sebb,
Thanks for the probing questions and sorry for the delay in reply.
Somehow I did not notice this reply.

Post by Manish Sapariya
I see following exception for some request. However the request

Post by Manish Sapariya
is successfully served as per the access log of httpd.
======================

Are you sure it's the same connection?
That does not seem possible.
If the connection is refused, how can the server handle it?

That's what I could not figure out.

Post by Manish Sapariya
Double (triple) check which connections are succeeeding and which are
failing

I did cross check it with the unique identified in the request and made sure
that its the only request.

Post by Manish Sapariya
Try using non-GUI mode on the client node.
I'm suprised you need 10 servers for a total of 500 users.
Depending on the test plan, a single node can handle that.

My test is data heavy. Its a test to simulate a video tutorials
viewing test. I forgot to mention that the test plan have 3 thread
groups running in parallel. So it makes it 150 threads in single
instance.

I am still trying to chase the problem. In order to understand
which connection is being rejected, I am trying to capture data
packets, but its huge data. So I am trying to capture only syn,
syn-ack and possible rst packets. But to do any kind of co-relation
I need to know what local ports jmeter used.

However I am not sure if jmeter can log the local port it used, or
better yet, when it shows failed request with any kind of exception,
can it also log local port for that socket?

I tried to check the properties files, but could not find exactly what
I was looking for. Turning the complete debugging for httpclient
may be too much, and may also slow down the test itself.

Any inputs?
Thanks,
Manish

[Shmuel Krakower]

Strange issue, specially that your get both timeout and connection refused
exceptions at the same time.

Few questions/ideas:
1. You mentioned you are using few servers to run the JMeter test. Is this
error shown on specific host or every time on a different host.
2. What happen if you run only a partial load with a single JMeter host? Is
the error shown too?
3. Have you tried to switch to another HTTP Implementation on the samplers?
4. To log I would suggest you to use the Simple Data Writer and log all
data related with failing samples only.
5. Is it possible you simply overloading your SUT (System Under Test)? Did
you look at it during load test?



Shmuel Krakower.
www.Beatsoo.org - re-use your jmeter scripts for application performance
monitoring from worldwide locations for free.

Post by Manish Sapariya
Sebb,
Thanks for the probing questions and sorry for the delay in reply.
Somehow I did not notice this reply.

Post by Manish Sapariya
I see following exception for some request. However the request

Post by Manish Sapariya
is successfully served as per the access log of httpd.
======================

Are you sure it's the same connection?
That does not seem possible.
If the connection is refused, how can the server handle it?

That's what I could not figure out.

Post by Manish Sapariya
Double (triple) check which connections are succeeeding and which are
failing

I did cross check it with the unique identified in the request and made sure
that its the only request.

Post by Manish Sapariya
Try using non-GUI mode on the client node.
I'm suprised you need 10 servers for a total of 500 users.
Depending on the test plan, a single node can handle that.

My test is data heavy. Its a test to simulate a video tutorials
viewing test. I forgot to mention that the test plan have 3 thread
groups running in parallel. So it makes it 150 threads in single
instance.
I am still trying to chase the problem. In order to understand
which connection is being rejected, I am trying to capture data
packets, but its huge data. So I am trying to capture only syn,
syn-ack and possible rst packets. But to do any kind of co-relation
I need to know what local ports jmeter used.
However I am not sure if jmeter can log the local port it used, or
better yet, when it shows failed request with any kind of exception,
can it also log local port for that socket?
I tried to check the properties files, but could not find exactly what
I was looking for. Turning the complete debugging for httpclient
may be too much, and may also slow down the test itself.
Any inputs?
Thanks,
Manish

[Manish Sapariya]

Thanks Shmuel.
Please see my answers inline.

Post by Shmuel Krakower
1. You mentioned you are using few servers to run the JMeter test. Is this
error shown on specific host or every time on a different host.

Not on specific host.

Post by Shmuel Krakower
2. What happen if you run only a partial load with a single JMeter host? Is
the error shown too?

The problem I saw only when I run my tests full throttle.

Post by Shmuel Krakower
3. Have you tried to switch to another HTTP Implementation on the samplers?

No. I will try out when I run the same test again.

Post by Shmuel Krakower
4. To log I would suggest you to use the Simple Data Writer and log all
data related with failing samples only.

Will try this out.

Post by Shmuel Krakower
5. Is it possible you simply overloading your SUT (System Under Test)? Did
you look at it during load test?
As I mentioned, I see this problem when I run full throttle.

I also tried capturing packets using tcpdump, but I could not find easy way
to capture only relevant http connections which are failing.

I will post to this thread if I make any in roads.

Regards,
Manish

Post by Shmuel Krakower
Shmuel Krakower.
www.Beatsoo.org - re-use your jmeter scripts for application performance
monitoring from worldwide locations for free.

Post by Manish Sapariya
Sebb,
Thanks for the probing questions and sorry for the delay in reply.
Somehow I did not notice this reply.

Post by Manish Sapariya
I see following exception for some request. However the request

Post by Manish Sapariya
is successfully served as per the access log of httpd.
======================

Are you sure it's the same connection?
That does not seem possible.
If the connection is refused, how can the server handle it?

That's what I could not figure out.

Post by Manish Sapariya
Double (triple) check which connections are succeeeding and which are
failing

I did cross check it with the unique identified in the request and made sure
that its the only request.

Post by Manish Sapariya
Try using non-GUI mode on the client node.
I'm suprised you need 10 servers for a total of 500 users.
Depending on the test plan, a single node can handle that.

My test is data heavy. Its a test to simulate a video tutorials
viewing test. I forgot to mention that the test plan have 3 thread
groups running in parallel. So it makes it 150 threads in single
instance.
I am still trying to chase the problem. In order to understand
which connection is being rejected, I am trying to capture data
packets, but its huge data. So I am trying to capture only syn,
syn-ack and possible rst packets. But to do any kind of co-relation
I need to know what local ports jmeter used.
However I am not sure if jmeter can log the local port it used, or
better yet, when it shows failed request with any kind of exception,
can it also log local port for that socket?
I tried to check the properties files, but could not find exactly what
I was looking for. Turning the complete debugging for httpclient
may be too much, and may also slow down the test itself.
Any inputs?
Thanks,
Manish

[Flavio Cysne]

I have had similar problems at work.
Monitor your environment Network Bandwidth (network cards, interfaces and
switches), it may be being overcharged at some point.
How many threads available in web/application server's thread pool? And
application server connections pool? And DB max connections?