Pythian Blog: Technical Track

Oracle RAC: Network Performance with Jumbo Frames

by Jared Still on May 14, 2020 12:00:00 AM

Introduction

When working with Oracle RAC, it's strongly advised to use Jumbo Frames for the network that provides the interconnect between nodes. As the nodes tend to send a lot of blocks back and forth across this private network, the larger the size of the packets, the fewer of them there are to send. The usual block size for an Oracle database is 8192 bytes. The standard MTU (Maximum Transmission Unit) for IP frame size is 1500 bytes. Sending an 8k Oracle block requires assembling six chunks of data to create a frame or packet. However, if Jumbo Frames are used (9000 bytes), the entire block fits neatly into a single frame or packet. ----- Note: Viewing the mechanical effects of MTU in action requires a fair amount of effort to setup a SPAN or Port Mirror. Use that port to capture the traffic from the wire. This is not being done for this test. Why this explanation? Because, as shown below, the packet size will be ~8k, even though the MTU is set to 1500. Because we cannot see the effects of MTU directly on the client or server, these effects are inferred from other data. "Frame" and "packet" are terms that seem to be used interchangeably. However, they are context-sensitive. That is, they occupy different layers of the OSI model. ----- On with the story... Recently, I was working with a two Node Oracle RAC system that runs in a VMWare ESXi 6.5 environment. It was thought that due to the optimizations being performed by VMWare in the virtual network stack, that Jumbo Frames were unnecessary. However, that does not seem to be the case. After some testing of throughput using both the standard 1500 byte MTU and 9000 byte Jumbo Frame MTU, the larger MTU size resulted in a 22% increase in throughput speed. Why did that happen? Well, keep reading to find out.

The Test Lab

Though the VMWare testing was done on Oracle Linux 7.8, the following experiments are being performed on Ubuntu 20.04 LTS. As there was no need to run Oracle, Ubuntu works just fine for these tests. Following are the two servers created: 
ubuntu-test-mule-01: tcp test client - 192.168.1.226
 ubuntu-test-mule-02: tcp test server - 192.168.1.227
Some version information: 
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '^(NAME=|VERSION=)' /etc/os-release
 NAME="Ubuntu"
 VERSION="20.04 LTS (Focal Fossa)"

Network Configuration

Other than having different IP addresses, ubuntu-test-mule-01 and ubuntu-test-mule-02 are set up exactly the same way. Because this version of Ubuntu uses netplan to configure the interfaces, we modified the /etc/netplan/00-installer-config.yaml file to configure the two test interfaces. The interfaces used for the testing are enp0s8 and enp0s9. Then, netplan apply was used to enable the changes. 
root@ubuntu-mule-01:~/perl-sockets/packet-test# cat /etc/netplan/00-installer-config.yaml
 # This is the network config written by 'subiquity'
 network:
  ethernets:
  enp0s3:
  dhcp4: true
  enp0s8:
  dhcp4: false
  addresses: [192.168.154.4/24]
  mtu: 9000
  enp0s9:
  dhcp4: false
  addresses: [192.168.199.35/24]
  version: 2
The results: 
# ip a
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
 link/ether 08:00:27:b8:5c:dc brd ff:ff:ff:ff:ff:ff
 inet 192.168.1.227/24 brd 192.168.1.255 scope global enp0s3
 valid_lft forever preferred_lft forever
 inet6 fe80::a00:27ff:feb8:5cdc/64 scope link
 valid_lft forever preferred_lft forever
 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
 link/ether 08:00:27:ab:d5:44 brd ff:ff:ff:ff:ff:ff
 inet 192.168.154.5/24 brd 192.168.154.255 scope global enp0s8
 valid_lft forever preferred_lft forever
 inet6 fe80::a00:27ff:feab:d544/64 scope link
 valid_lft forever preferred_lft forever
 4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
 link/ether 08:00:27:22:fc:a2 brd ff:ff:ff:ff:ff:ff
 inet 192.168.199.36/24 brd 192.168.199.255 scope global enp0s9
 valid_lft forever preferred_lft forever
 inet6 fe80::a00:27ff:fe22:fca2/64 scope link
 valid_lft forever preferred_lft forever

Configure the TCP Test

Some time ago, I put together some Perl scripts for network throughput testing. There were several reasons for this, including:
  • You can copy and paste the code if necessary.
  • It is easy to modify for different tests.
The following was run on each of the test mule servers: # git clone https://github.com/jkstill/perl-sockets.git

On the ubuntu-mule-02 Server:

The following changes were made to server.pl: First, disable the dot reporting. By default a "." is printed every 256 packets that are received. You can disable this using the following line: my $displayReport = 0; # set to 0 to disable reporting Now, set the listening addresses. Default code: 
my $sock = IO::Socket::INET->new(
  #LocalAddr => '192.168.1.255', # uncomment and edit adddress if needed
  LocalPort => $port,
  Proto => $proto,
  Listen => 1,
  Reuse => 1
 ) or die "Cannot create socket: $@";
We changed this to reflect the network interfaces that would be used for the testing on the server-side. 
my $sock = IO::Socket::INET->new(
  LocalAddr => '192.168.154.5', # MTU 9000
  #LocalAddr => '192.168.199.36', # MTU 1500
  LocalPort => $port,
  Proto => $proto,
  Listen => 1,
  Reuse => 1
 ) or die "Cannot create socket: $@";
The appropriate interface was used for each test.

On the ubuntu-mule-01 Client:

Some test data was created. The use of /dev/urandom and gzip makes it unlikely that you can perform any compression. This is something I learned to do for quick throughput tests using ssh, as ssh compresses data. It's probably not necessary in this case, but then again, it doesn't hurt to ensure the test data is non-compressible. 
root # cd perl-sockets/packet-test
 root # dd if=/dev/urandom bs=1048576 count=101 | gzip - | dd iflag=fullblock bs=1048576 count=100 of=testdata-100M.dat
 root # dd if=/dev/urandom bs=1048576 count=1025 | gzip - | dd iflag=fullblock bs=1048576 count=1024 of=testdata-1G.dat
 
 root@ubuntu-mule-01:~/perl-sockets/packet-test# ls -l testdata-1*
 -rw-r--r-- 1 root root 104857600 May 11 16:05 testdata-100M.dat
 -rw-r--r-- 1 root root 1073741824 May 11 16:04 testdata-1G.dat

The Driver Script

We used the packet-driver.sh script to run each of the tests from the client-side. This script simply runs a throughput test 23 times in succession, using the specified MTU size. 
#!/usr/bin/env bash
 
 : ${1:?Call with 'packet-driver.sh <SIZE> '!}
 : ${mtu:=$1}
 
 if ( echo $mtu | grep -vE '1500|9000' ); then
  echo Please use 1500 or 9000
  exit 1
 fi
 
 declare -A localHosts
 declare -A remoteHosts
 
 localHosts[9000]=192.168.154.4
 localHosts[1500]=192.168.199.35
 
 remoteHosts[9000]=192.168.154.5
 remoteHosts[1500]=192.168.199.36
 
 blocksize=8192
 testfile=testdata-1G.dat
 
 cmd="./client.pl --remote-host ${remoteHosts[$mtu]} --local-host ${localHosts[$mtu]} --file $testfile --buffer-size $blocksize"
 
 for i in {0..22}
 do
  echo "executing: $cmd"
  $cmd
 done

Perform the Tests

For each test, we enabled the correct interface in server.pl, and then the started the server. For the client-side, we called the packet-driver.sh script with the required MTU size. The MTU size passed on the command line determines which interface is used on the client-side.

1500 MTU

On the server-side, make sure the address in server.pl is set for the 1500 byte MTU interface. Then, start the server: 
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '\s+LocalAddr' server.pl
 LocalAddr => '192.168.199.36', # MTU 1500
 
 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./server.pl | tee mtu-1500.log
 Initial Receive Buffer is 425984 bytes
 Server is now listening ...
 Initial Buffer size set to: 2048
On the client-side, run packet-driver.sh: 
root@ubuntu-mule-01:~/perl-sockets/packet-test# ./packet-driver.sh 1500
 executing: ./client.pl --remote-host 192.168.199.36 --local-host 192.168.199.35 --file testdata-1G.dat --buffer-size 8192
 
 remote host: 192.168.199.36
 port: 4242
 bufsz: 8192
 simulated latency: 0
 
 bufsz: 8192
 Send Buffer is 425984 bytes
 Connected to 192.168.199.36 on port 4242
 Sending data...

9000 MTU

On the server-side, make sure the address in server.pl is set for the 9000 byte MTU interface. Then, start the server: 
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '\s+LocalAddr' server.pl
 LocalAddr => '192.168.154.5', # MTU 9000
 
 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./server.pl | tee mtu-9000.log
 Initial Receive Buffer is 425984 bytes
 Server is now listening ...
 Initial Buffer size set to: 2048
Now, run packet-driver.sh on the client: 
root@ubuntu-mule-01:~/perl-sockets/packet-test# ./packet-driver.sh 9000
 executing: ./client.pl --remote-host 192.168.154.5 --local-host 192.168.154.4 --file testdata-1G.dat --buffer-size 8192
 
 remote host: 192.168.154.5
 port: 4242
 bufsz: 8192
 simulated latency: 0
 
 bufsz: 8192
 Send Buffer is 425984 bytes
 Connected to 192.168.154.5 on port 4242
 Sending data...

Reporting

When all tests are complete, use packet-averages.pl to calculate the averages across all tests per MTU size. 
root@ubuntu-mule-02:~/perl-sockets/packet-test# ./packet-averages.pl < mtu-1500.log
 key/avg: Bytes Received 1073733637.000000
 key/avg: Avg Packet Size 7898.147391
 key/avg: Packets Received 135948.304348
 key/avg: Average milliseconds 0.043824
 key/avg: Avg Megabytes/Second 172.000000
 key/avg: Avg milliseconds/MiB 5.818500
 key/avg: Total Elapsed Seconds 6.850447
 key/avg: Network Elapsed Seconds 5.958098
 
 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./packet-averages.pl < mtu-9000.log
 key/avg: Bytes Received 1073733637.000000
 key/avg: Avg Packet Size 7519.793478
 key/avg: Packets Received 142790.217391
 key/avg: Average milliseconds 0.033652
 key/avg: Avg Megabytes/Second 213.165217
 key/avg: Avg milliseconds/MiB 4.692753
 key/avg: Total Elapsed Seconds 5.495095
 key/avg: Network Elapsed Seconds 4.805343
The average Total Elapsed Seconds for the 9000 MTU tests is only 80% of the time required for the 1500 MTU tests. From these results, it appears as if using Jumbo Frames is a pretty clear winner, even in a virtualized environment. This result might seem somewhat surprising, as the tests are not sending any data over a physical wire. In this case, the "network" is only composed of VirtualBox host network adapters. So, why then are Jumbo Frames still so much faster than the standard 1500 MTU size?

Performance Profiling

This time, we'll run a single test for each MTU size. We'll use the perf profiler to gather process profile information on the client-side. First, the 1500 MTU size: 
perf record --output perf-mtu-1500.data ./client.pl --remote-host 192.168.199.36 --local-host 192.168.199.35 --file testdata-1G.dat --buffer-size 8192
Now the 9000 MTU size: 
perf record --output perf-mtu-9000.data ./client.pl --remote-host 192.168.154.5 --local-host 192.168.154.4 --file testdata-1G.dat --buffer-size 8192
Let's see some reports from perf.

1500 MTU

root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-1500.data --stats | grep TOTAL
 TOTAL events: 28648
 
 root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-1500.data | head -20
 # To display the perf.data header info, please use --header/--header-only options.
 #
 #
 # Total Lost Samples: 0
 #
 # Samples: 28K of event 'cpu-clock:pppH'
 # Event count (approx.): 7137500000
 #
 # Overhead Command Shared Object Symbol
 # ........ ....... .................. ....................................
 #
 56.35% perl [kernel.kallsyms] [k] e1000_xmit_frame
 21.00% perl [kernel.kallsyms] [k] e1000_alloc_rx_buffers
 9.71% perl [kernel.kallsyms] [k] e1000_clean
 2.22% perl [kernel.kallsyms] [k] __softirqentry_text_start
 1.74% perl [kernel.kallsyms] [k] __lock_text_start
 0.61% perl [kernel.kallsyms] [k] copy_user_generic_string
 0.58% perl [kernel.kallsyms] [k] clear_page_rep
 0.39% perl libpthread-2.31.so [.] __libc_read
 0.30% perl [kernel.kallsyms] [k] do_syscall_64

9000 MTU

root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-9000.data --stats | grep TOTAL
 TOTAL events: 25259
 root@ubuntu-mule-01:~/perl-sockets/packet-test#
 root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-9000.data | head -20
 # To display the perf.data header info, please use --header/--header-only options.
 #
 #
 # Total Lost Samples: 0
 #
 # Samples: 25K of event 'cpu-clock:pppH'
 # Event count (approx.): 6290500000
 #
 # Overhead Command Shared Object Symbol
 # ........ ....... .................. ........................................
 #
 64.15% perl [kernel.kallsyms] [k] e1000_xmit_frame
 16.92% perl [kernel.kallsyms] [k] e1000_alloc_jumbo_rx_buffers
 9.03% perl [kernel.kallsyms] [k] e1000_clean
 1.79% perl [kernel.kallsyms] [k] __softirqentry_text_start
 1.66% perl [kernel.kallsyms] [k] __lock_text_start
 0.41% perl [kernel.kallsyms] [k] clear_page_rep
 0.39% perl [kernel.kallsyms] [k] copy_user_generic_string
 0.26% perl [kernel.kallsyms] [k] do_syscall_64
 0.24% perl libpthread-2.31.so [.] __libc_read
A conclusion we might draw from these reports: When using an MTU of 9000, the test program spent more time sending data ( e1000_xmit_frame) and less time in overhead. Note that 16.92% of the time was spent in allocating Jumbo-sized frames through e1000_alloc_jumbo_rx_buffers in the MBU 9000 test, versus 21% of the time required in the MTU 1500 test for e1000_alloc_rx_buffers. The reason for the performance increase, in this case, seems to be this: The use of Jumbo Frames simply requires less work for the server. Rather than assembling six 1500 byte frames into a packet to accommodate our 8192-byte packet, Jumbo Frames can get it all in one frame. Though these tests were run using servers virtualized with VirtualBox, the results are quite similar to those seen in servers running in VMWare ESXi. The fact that the servers are virtual does not reduce the need to ensure that RAC nodes get the fastest possible throughput on the private network used for the interconnect... And that means using Jumbo Frames.

No Comments Yet

Let us know what you think

See all

See all

Subscribe by email
Follow us
Love Your Data
Pythian Logo

© 2024 All rights reserved.