LBaaS Performance Benchmarking

We played around with different tools to benchmark our LBaaS (Haproxy) performance, few of them being Tsung, iperf, ab, etc. After some discussion and investigation, we boiled down to providing 2 metrics to our customers for LBaaS:

1. Requests/second
2. Throughput

To get optimal performance off our apache servers and LBaaS, we did some TCP optimizations on apache server and the hypervisor where our LBaaS resides

# /etc/sysctl.conf
# Increase system file descriptor limit
fs.file-max = 100000

# Discourage Linux from swapping idle processes to disk (default = 60)
vm.swappiness = 10

# Increase ephermeral IP ports
net.ipv4.ip_local_port_range = 10000 65000

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 1

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# If your servers talk UDP, also up these limits
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

# Disable source routing and redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0

# Log packets with impossible addresses for security
net.ipv4.conf.all.log_martians = 1

We also observed during our tests that the default Haproxy limits the maximum concurrent connections on the loadbalancer to around 4000. We modified the haproxy.cfg file to increase the open file limit and the maxconn to cater to large number of concurrent connections:

global
    daemon
    user nobody
    group nogroup
    ulimit-n 200000
    maxconn 50000

We conducted different tests with Apache Benchmarking tool with different payload sizes, the standard "It works" index.html page which is 177 Bytes, 1KB, 10KB and 100KB. We also tweaked the number of parallel/concurrent connections while measuring the performance. Following tables explained the results we achieved. We were running 16 vCPU, 30GB Large Apache2 servers for our testing purpose.

HTTP Testing:

1) Requests/second to apache server(across 1 million requests and 3 iterations):

2) Requests/second through LBaaS with 2 backend servers (across 1 million requests and 3 iterations):

SSL Testing:

1) HTTPS Requests/second to apache server (across 1 million requests and 3 iterations):

2) HTTPS requests/second through LBaaS with 2 backend servers (across 1 million requests and 3 iterations):

Throughput Testing:

With LBaaS sitting in middle of client and server, there is definitely some impact on the maximum throughput we can get over the wire. Following are the throughputs we observed:

1) Throughput between client and server without LBaaS (measured across 30s of iperf traffic):

2) Throughput between client and server with LBaaS (measured across 30s of iperf traffic):

We did some tweaking with haproxy config (haproxy.cfg) and figured out that increasing the number of haproxy processes increases throughput significantly. The config change we did was:

global
    daemon
    user nobody
    group nogroup
    ulimit-n 200000
    maxconn 50000
    nbproc 4

Please note that this config change is still not in the code, we would require some change in vrouter agent to handle this new config.