[AWS] Further info on the weird "AWS is returning too much data" issue

[Thomas Løcke]

On Sun, Jan 23, 2011 at 6:59 PM, Pascal Obry <pascal at obry.net> wrote:
>> The numbers returned by ApacheBench were correct, and grepping for
>> "200 46" in the log file also returned exactly 500_000 lines. I
>> repeated this three times, for a total of 1_500_000 hits. No errors
>> found. Everything matched.
>
> You mean ApacheBench, node.js and AWS have all the same data, right?


No, I mean that the data logged by my AWS server matches the results
given by ApacheBench, so if ApacheBench reports 500_000 completed
requests and 23_000_000 bytes transferred, then that is also what I'm
seeing in the AWS access log file.


>> So it seems that when concurrent hits go up, AWS start to mess up somewhere.
>
> When you talk about concurrency you have 2 or 5 servers on AWS side, but
> how many clients are sending requests?


I have one AWS server with the Max_Connection configuration parameter set at 50.

I have one instance of ApacheBench running on a different host. This
one instance can act as 1 or more clients, like so:

    $ ab -n 50000 -c 1 URL

Here ApacheBench request URL 50000 times using 1 client.

    $ ab -n 50000 -c 5 URL

And here ApacheBench request URL 50000 times using 5 concurrent
clients, ie. 10000 requests per client.

AWS fails when more than one client is hammering it at the same time.
I've not been able to reproduce the problem as long as there's only
one client hitting my AWS server.


>> Is there anything I can do to help track down this issue?
>
> I'm sure I know what is happening here :) I bet that if you have 1 more
> server on AWS server side than clients sending data all will be ok. Can
> you confirm?


I've already tested this, and the problem persists.

If I setup ApacheBench to run two simultaneous clients, like this

    $ ab -n 50000 -c 2 URL

And my AWS server is configured with Max_Connection set to 5, then it
still happens. Not very often, but it happens.

The more clients, the worse the problem is.

If I setup my AWS server with Max_Connection 1, then I can no longer
reproduce the problem, no matter how many clients that are accessing
the server.

If I setup my AWS server with Max_Connection 2, then the problem show
up as soon as I access the server with more than 1 client.

So this problem seems to be dependant on two things:

    1. AWS must be configured with Max_Connection > 1
    2. There must be > 1 clients accessing the AWS server at the same time

:o)
Thomas Løcke