[AWS] Further info on the weird "AWS is returning too much data" issue

[Thomas Løcke]

Today I expanded on my AWS vs node.js tests with the inclusion of a
bunch of tests where ApacheBench was running on a remote host.

This merely served to make AWS look even better. The newly added
graphics to the wiki page is a clear testament to that:

http://wiki.ada-dk.org/index.php/AWS_vs_node.js#The_Tests

But these new tests also drove home the fact that not all is well in
AWS land: It is returning too much data, whereas node.js is
consistently returning exactly what it is supposed to.

So naturally I did some more tests.

First I did a simple test against two of my Apache servers, and
despite them both being rather slow compared to AWS and node.js, they
both returned the exact same amount of bytes on every run.

With this knowledge, I felt it was safe to conclude that ApacheBench
was not to blame. If both node.js and Apache are working as intended,
then clearly something is amiss with AWS?

So I enabled logging on my AWS server, and started benchmarking it
again. First run was 500_000 hits with a concurrency of 1.

The numbers returned by ApacheBench were correct, and grepping for
"200 46" in the log file also returned exactly 500_000 lines. I
repeated this three times, for a total of 1_500_000 hits. No errors
found. Everything matched.

Then I tried with a concurrency of 2, and then things went bad.
Instead of getting 23_000_000 bytes worth of HTML transferred for
500_000 requests, I got 23_000_046. I checked the AWS logfile, and
there it was: 500_001 lines found.

I tried running it a few more times, and with a concurrency of 2, it
appears to fault 50/50. Sometimes it works, sometimes it doesn't.

Then I went up to a concurrency of 5, and now things really got bad. I
consistently got more bytes than requested. At its worst I got 9338
bytes too many (203 pages). This was with a concurrency of 100 and
500_000 hits.

So it seems that when concurrent hits go up, AWS start to mess up somewhere.

It appears as if when AWS is very busy, it sometimes roll together
two, or more, requests into one. It is as if multiple tasks are
writing to the same socket at the same time, occasionally. That would
explain why ApacheBench counts the proper amount of requests served,
but get the wrong amount of bytes. ApacheBench is oblivious to the
actual data it is getting, it simple counts the request as served and
the bytes returned.

Is there anything I can do to help track down this issue?

:o)
Thomas Løcke