Bug? Jmeter only tries one addr in multi-addr DNS round robin cluster

[Chris Shenton]

I'm a jmeter newbie but looked through the docs, FAQ, and mail
archives. Running jmeter on FreeBSD-4.10 with jdk-1.4.2.

My target is a set of three app server boxes running Tomcat-4.1. I've
implemented a cluster with Filip Hanik's back-port of the TC5
clustering code.

http://cvs.apache.org/%7Efhanik/

I'm trying to test this and make sure it works for us on TC4.

It seems to work fine in simple browser tests. I have a DNS name
"cluster" which has three A records, one for each node's address.

***@Palimpsest(347> dnsip cluster.saic.hq.nasa.gov
198.116.138.39 198.116.138.106 198.116.138.56

I point Mozilla at this and login, click around, etc -- it returns to
the same node each time. I disable tomcat on that node. Upon
subsequent clicks, Mozilla goes to one of the other addresses
associated with the cluster's DNS name. So this works fine.

But when I configure Jmeter with the same DNS name, it appears that it
looks up the name at startup, only saves one of the three addresses,
and never tries the others. When re-running the same test, it never
goes to a different host. If I disable tomcat on the node it's
probing during a run, it continues to try to connect to the dead node
and never goes to another node. It only goes to a different node if I
actually stop then restart jmeter.

This makes it impossible for jmeter to test DNS round robin based
clusters. (In my final deployment I'll have a HW load balancer, but
don't have one in my development environment.)

I think Jmeter should be smart enough to cycle through addresses, or
have each thread randomly pick an address from the list DNS provides,
or prefer one and switch to another if the preferred one is
unavailable (TCP Connection Refused is a pretty good sign of that!).

Am I missing something? Is this expected behavior? Desired behavior?

Thanks.

[Paul Leclerc]

Chris,
Don't forget that by default, Java caches IP addresses for DNS
lookups. See this link for more information:
http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html and look
at both networkaddress.cache.ttl and sun.net.inetaddr.ttl



On Wed, 10 Nov 2004 16:41:06 -0500, Chris Shenton

Post by Chris Shenton
I'm a jmeter newbie but looked through the docs, FAQ, and mail
archives. Running jmeter on FreeBSD-4.10 with jdk-1.4.2.
My target is a set of three app server boxes running Tomcat-4.1. I've
implemented a cluster with Filip Hanik's back-port of the TC5
clustering code.
http://cvs.apache.org/%7Efhanik/
I'm trying to test this and make sure it works for us on TC4.
It seems to work fine in simple browser tests. I have a DNS name
"cluster" which has three A records, one for each node's address.
198.116.138.39 198.116.138.106 198.116.138.56
I point Mozilla at this and login, click around, etc -- it returns to
the same node each time. I disable tomcat on that node. Upon
subsequent clicks, Mozilla goes to one of the other addresses
associated with the cluster's DNS name. So this works fine.
But when I configure Jmeter with the same DNS name, it appears that it
looks up the name at startup, only saves one of the three addresses,
and never tries the others. When re-running the same test, it never
goes to a different host. If I disable tomcat on the node it's
probing during a run, it continues to try to connect to the dead node
and never goes to another node. It only goes to a different node if I
actually stop then restart jmeter.
This makes it impossible for jmeter to test DNS round robin based
clusters. (In my final deployment I'll have a HW load balancer, but
don't have one in my development environment.)
I think Jmeter should be smart enough to cycle through addresses, or
have each thread randomly pick an address from the list DNS provides,
or prefer one and switch to another if the preferred one is
unavailable (TCP Connection Refused is a pretty good sign of that!).

[Chris Shenton]

Post by Paul Leclerc
Don't forget that by default, Java caches IP addresses for DNS
http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html and
look at both networkaddress.cache.ttl and sun.net.inetaddr.ttl

That looked very promising but it doesn't seem to be helping.
In $JAVA_HOME/jre/lib/security.java.security I set

networkaddress.cache.ttl=1

and see it querying my dnscache often, whereas before it queried it
only once. That's good. However, it's only trying one of the addresses
in DNS for that name.

The name I configure jmeter to test is "cluster" which evaluates to
two different IPs both with 1s TTL; I have it use 5 threads, and loop
5 times, with a 30-second ramp-up-time. I do see my DNS being queried
repeatedly for that name, but jmeter only hitting one.

Interestingly, quitting and re-running the "jmeter" startup script has
no effect: it keeps returning to the same target host address. My
dnscache is documented to return a list of addresses in random order
for each query. The law of probabilities says it should get them both
50% of the time, this is about 10 out of 10 times it gets the same
one. (Is it suspicious it's the lower of the two addrs?).



Also, if I shut down the tomcat on the target it's picked, jmeter
appears to hang. Is it not written to try other addresses in the list
of addresses it got for the target's name? I don't have the code here
to look. Perhaps this wasn't a design goal.

Any thoughts? I can't test the robustness of my tomcat-4 session
clustering if my test tools won't try alternate addresses.

Thanks again.

[Peter Lin]

hmmm, I'm just guessing here. First to make sure i understand.

1. you can verify your DNS returns random IP in the cluster
2. when a tomcat instance is shut down jmeter hangs. this supports the
idea that jmeter is using the same IP
3. restarting jmeter or tomcat doesn't appear to make JMeter go to
random server in the cluster

I can think of a couple ways to verify if it is the JVM caching the IP
is causing it. Write a simple class to get the IP address 100 times
and see which IP it resolves to. It may very well be that the JVM
isn't flushing out the old IP and is instead just using the old one.

I take it your test plan points to something like cluster.mydomain.com.

peter



On Fri, 19 Nov 2004 19:53:37 -0500, Chris Shenton

Post by Chris Shenton

Post by Paul Leclerc
Don't forget that by default, Java caches IP addresses for DNS
http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html and
look at both networkaddress.cache.ttl and sun.net.inetaddr.ttl

That looked very promising but it doesn't seem to be helping.
In $JAVA_HOME/jre/lib/security.java.security I set
networkaddress.cache.ttl=1
and see it querying my dnscache often, whereas before it queried it
only once. That's good. However, it's only trying one of the addresses
in DNS for that name.
The name I configure jmeter to test is "cluster" which evaluates to
two different IPs both with 1s TTL; I have it use 5 threads, and loop
5 times, with a 30-second ramp-up-time. I do see my DNS being queried
repeatedly for that name, but jmeter only hitting one.
Interestingly, quitting and re-running the "jmeter" startup script has
no effect: it keeps returning to the same target host address. My
dnscache is documented to return a list of addresses in random order
for each query. The law of probabilities says it should get them both
50% of the time, this is about 10 out of 10 times it gets the same
one. (Is it suspicious it's the lower of the two addrs?).
Also, if I shut down the tomcat on the target it's picked, jmeter
appears to hang. Is it not written to try other addresses in the list
of addresses it got for the target's name? I don't have the code here
to look. Perhaps this wasn't a design goal.
Any thoughts? I can't test the robustness of my tomcat-4 session
clustering if my test tools won't try alternate addresses.
Thanks again.
---------------------------------------------------------------------

[Chris Shenton]

Post by Peter Lin
hmmm, I'm just guessing here. First to make sure i understand.
1. you can verify your DNS returns random IP in the cluster
2. when a tomcat instance is shut down jmeter hangs. this supports the
idea that jmeter is using the same IP
3. restarting jmeter or tomcat doesn't appear to make JMeter go to
random server in the cluster
I take it your test plan points to something like cluster.mydomain.com.

Exactly correct. I can see how #1 and #2 might be caused by an
implementation that only looks at the first IP in a list, but #3
baffles me.

Post by Peter Lin
I can think of a couple ways to verify if it is the JVM caching the IP
is causing it. Write a simple class to get the IP address 100 times
and see which IP it resolves to. It may very well be that the JVM
isn't flushing out the old IP and is instead just using the old one.

Good idea, thanks.

[hui]

you can un-check "Use KeepAlike"