OnlineTester: Benchmarking

Tagged:  •    •    •    •    •    •  

This post describes a session measuring the behavior of an Iliad web application using AJAX request. Memory consumption and run time were investigated for different numbers of simultaneous users.

Spoiler


Memory usage has drastically improved since the first public test session, it is down by about 50%. There is still some room for improvement, but the changes of the last two weeks have been impressive. As far as speed and stability is concerned, the application performs very well, even under (relatively) heavy load.

Intro


I've described in some detail how I developed my first web appplication based on GNU Smalltalk and Iliad. The first public test run was conducted during July 2009 with 5 groups of about 25 young German high-school (Gymnasium) students, who responded very well to the new testing tool.
The tool itself had a few problems, which were promptly analyzed and fixed by Nico and especially Paolo, who spent quite some time on analyzing test images and discovering some bugs in different pieces of the software along the way.
Once the patches were applied, I ran some benchmarks with increasing complexity. I started off using Ruby's mechanize tool, which is quite convenient for interacting with web pages, but finally I had to resort back to the good ol' net/http library, since I could not figure out
how to produce an AJAX-like request within mechanize limits.

The test setup


The following script was used to simulate successively larger amounts of simultaneous users.
require 'mechanize'
require 'net/http'
require 'uri'

# test on localhost
BASE="http://127.0.0.1:4080/"

# run a complete session with a very fast students
def run(name)
  Thread.start( name ) { |n|
    agent = WWW::Mechanize.new
    rw = agent.get( "#{BASE}onlinetest" )
    rw.encoding = "UTF-8"
    load_assets(agent, rw)
    form = rw.forms.first
    input = form.fields.first
    input.value = n
    tw = agent.submit( form )
    tw.encoding = "UTF-8"
    load_assets(agent, tw)
    complete_test(agent, tw)  # *** toggled ***
  }
end

# real browsers load js, css and img, too
def load_assets(agent, page)
  page.search( '//head/script' ).each do |s|
    script = agent.get( BASE + s["src"] )
  end
  page.search( "//head/link[@rel='Stylesheet']" ).each do |s|
    css = agent.get( BASE + s["href"] )
  end
  page.search( "img" ).each do |s|
    image = agent.get( BASE + s["src"] )
  end
end

# do what Iliad would cause the browser to do when clicking on radio buttons
# the test user alwas chooses option 3 ... randomize it later for nice reports
def complete_test(agent, page)
  page.search( "form[@class='antwort']" ).each do |f|
    choice = f.search( "input[@type='radio']" )[ 2 ]
    token = f.search( "input[@type='hidden']" )[ 0 ]
    params = { "_token" => token["value"], choice["name"] => choice["value"] }
    url = URI.parse( BASE+f["action"] )
    request = Net::HTTP::Post.new( url.path )
    request.form_data = params
    request[ "Cookie" ] = agent.cookies.first
    request[ "Accept" ] = "application/json, text/javascript, */*"
    response = Net::HTTP.new(url.host, url.port).start { |http|
      http.request(request)
    }
    # puts( response.body.inspect )
  end
end

# run  tests "in parallel"
threads = Array.new
$stdout.puts( "starting" )
uid = Time.now.to_f.to_s
1.upto( ARGV.first.to_i ) { |i| threads << run( "user_#{uid}_#{i}" ) }
$stdout.puts( "waiting" )
threads.each { |t| t.join }
$stdout.puts( "done" )

This script was called from a simple shell wrapper:
for i in 10 20 30 40 50 60 70 80 90 100 
do
  sleep 3
  make nut 
  time ruby mech.rb $i 
done

The line "make nut" kills a previously running server, recreates the image with the application code from scratch, loads the test and finally starts the Swazoo server.

The responses


The script above loads the dynamically generated registration page (1 kB) including 6 linked files (278 kB) and performs the login, upon which the 77 kB large test page is generated and retrieved, again with all linked files, which amounts to 17 requests and a total of 648 kB static files. All these are served through Swazoo from and to localhost.
I ran two batches of benchmarks with 10,20,...,100 simultaneous users, the first only loaded the complete test page, the second completed the test as quickly as (synchronously) possible.
For this second batch, the script then sent the same requests to the server, that a web browser would have sent, if the user clicked on a radio button once per question. The responses created here are between 800 and 900 bytes, depending on how much text is associated with the widget.

The timings


While the first batch run shows how Swazoo would keep up with an initial "onslaught" of clean-cache clients, the second batch run is more of a test of Iliad's speed, as there were 95 questions per test, i.e. there are 95 dynamically generated JSON-responses per user.
  n | dt 1 | dt 2
 ---+------+------
 10 |    7 |   19
 20 |   14 |   40
 30 |   22 |   64
 40 |   29 |   82
 50 |   38 |  110
 60 |   45 |  132
 70 |   54 |  160
 80 |   62 |  190
 90 |   73 |  205 connection timeout in 1 thread
100 |   82 |  230 connection timeout in 1 thread

Resident memory usage


gst-remote always claimed about 830 MB virtual memory and that amount did not change during test sessions. More interesting to watch is the resident memory usage, as reported by
atop -m -a -P PRM 1 | grep gst-remote

I took the difference from the inital and final RSIZE footprint and rounded to MB. Since the results with the wrapper shown above were surprising ...
  n | dRSIZE 1 | dRSIZE 2
 ---+----------+----------
 80 |      108 |     193  
 90 |      126 |     206  
100 |      113 |     227  

... so I decided to aid measurements by using explicit garbage collections:
for i in 10 20 30 40 50 60 70 80 90 100 
do
  sleep 3 
  make nut 
  gst-remote --eval="ObjectMemory globalGarbageCollect" 
  sleep 3 
  time ruby mech.rb $i 
  gst-remote --eval="ObjectMemory globalGarbageCollect" 
  sleep 3 
done

And while I was at it, I also added another run, where the user changes his mind, i.e. the form returned in JSON is submitted again.
  n | dRSIZE 0 | dRSIZE 1 | dRSIZE 2
 ---+----------+----------+----------
 10 |      14  |      21  |      25 
 20 |      25  |      47  |      48 
 30 |      41  |      74  |      66 
 40 |      55  |      87  |     104 
 50 |      65  |     114  |     121 
 60 |      70  |     124  |     155 
 70 |      87  |     167  |     179 
 80 |     106  |     193  |     200 
 90 |     106  |     too  |    late     
100 |     114  |      at  |   night

Oh, well... I declare memory measurements to be difficult. And I have not even looked closely at the development of RSIZE over time within a single run... I saw one session where RSIZE stayed at 141 MB for 30 s before climbing to 170 MB within 6 s.

Coda


As already stated in the spoiler above:

There is still some room for improvement, but the changes of the last two weeks have been impressive. As far as speed and stability is concerned, the application performs very well, even under (relatively) heavy load. Many thanks again to Paolo and Nico for taking the time to provide great tools.

Great thing!!!!! I'm sure it will come handy once we get to some basic working version.

For memory testing, you might want to use "pmap -d", as I'm a bit surprised by the 830Mb figure. I get a much less expensive map: 338Mb overall, with 10Mb data and 28Kb shared.
I'm starting from a gst -i and executing:

"Global garbage collection... done"
Loading package Sockets
Loading package Iconv
Loading package Sport
Loading package Iliad-Core
Loading package Magritte
Loading package Iliad-More-UI
Loading package Iliad-More-Magritte
Loading package XML-SAXDriver
Loading package Iliad-More-Examples
Loading package Swazoo
Loading package Iliad-More
Loading package Iliad-Swazoo
Loading package XML-SAXParser
Loading package XML-DOM
Loading package XML-XMLParser
"Global garbage collection... done, heap grown"
Loading package Iliad
Loading package XML-XMLNodeBuilder
Loading package Iliad-Localization
Loading package XML
Loading package Ambaradan-Localization
Loading package Ambaradan-Application
Loading package Ambaradan
"Global garbage collection... done"
Starting server ...
gst-remote server started.
Restarting Swazoo! startOn: 3000...

Probably top isn't very accurate.

... as pmap shows that gst really has reserved that much (virtual) memory:

 $ pmap -q 8978
 8978:   gst-remote --port=12345 --server -I ot.im
 ...
 00002b0af8d3d000 258928K -----    [ anon ]
 ...
 00002b0b09517000 523260K -----    [ anon ]
 ...

Maybe you're running a 32-bit gst?

Yes, it's occupied in the address space but not used (MAP_NORESERVE). The idea is that any mmap (for example from libc or from big objects) will never end up right after the end of the OOP table or of the object memory.

84354000-84356000 r-xp 00000000 fd:01
84356000-84555000 ---p 00002000 fd:01
84557000-845ff000 r-xp 00000000 fd:01
845ff000-847fe000 ---p 000a8000 fd:01

User login