VMware VSphere 4 Manuel d'installation et d'exploitation

VMware vSphere™ 4 Fault Tolerance:

Architecture and Performance

WHITE PAPER

VMWARE WHITE PAPER

Table of Contents

1. VMware Fault Tolerance Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.DeterministicRecord/Replay................................................................... 3

1.2.FaultTolerance................................................................................. 3

1.3.VMwarevLockstepInterval..................................................................... 4

1.4.TransparentFailover............................................................................4

2. Performance Aspects and Best Practice Recommendations. . . . . . . . . . . . . . . . . . . . . . 5

2.1.FTOperations:TurningOnandEnabling........................................................ 5

2.2.ResourceConsumption ........................................................................ 5

2.3.SecondaryVirtualMachineExecutionSpeed.................................................... 5

2.4.I/OLatencies................................................................................... 6

2.5.NetworkLink .................................................................................. 6

2.6.NICAssignmentsforLoggingTraffic............................................................ 6

2.7.VirtualMachinePlacement..................................................................... 6

2.8.DRSandVMotion.............................................................................. 7

2.9.TimerInterrupts................................................................................ 7

2.10. Fault Tolerance Logging Bandwidth Sizing Guideline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.FaultTolerancePerformance ..................................................7

3.1.SPECjbb2005 .................................................................................. 7

3.2.KernelCompile................................................................................8

3.3.NetperfThroughput ...........................................................................9

3.4.NetperfLatencyBoundCase................................................................... 9

3.5.FilebenchRandomDiskRead/Write ...........................................................10

3.6.Oracle11g....................................................................................11

3.7.MicrosoftSQLServer2005....................................................................12

3.8.MicrosoftExchangeServer2007...............................................................13

4. VMware Fault Tolerance Performance Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

5.Conclusion..................................................................14

AppendixA:BenchmarkSetup .................................................15

StorageArray.....................................................................................15

PrimaryandSecondaryHosts.....................................................................15

ClientMachine....................................................................................15

AppendixB:WorkloadDetails ..................................................16

SPECjbb2005 .....................................................................................16

KernelCompile...................................................................................16

Netperf...........................................................................................16

Filebench.........................................................................................16

Oracle11g—Swingbench........................................................................16

MSSQL2005—DVDStoreBenchmark ............................................................17

Exchange2007—Loadgen.......................................................................18

VMWARE WHITE PAPER

VMware® Fault Tolerance (FT) provides continuous availability to virtual machines, eliminating downtime and disruption — even in

the event of a complete host failure. This whitepaper gives a brief description of the VMware FT architecture and discusses the

performance implication of this feature with data from a wide variety of workloads.

1. VMware Fault Tolerance Architecture

The technology behind VMware Fault Tolerance is called VMware® vLockstep. The following sections describe some of the key aspects

of VMware vLockstep technology.

1.1. Deterministic Record/Replay

Deterministic Record/Replay is a technology introduced with VMware Workstation 6.0 that allows for capturing the execution of a

running virtual machine for later replay. Deterministic replay of computer execution is challenging since external inputs like incoming

network packets, mouse, keyboard, and disk I/O completion events operate asynchronously and trigger interrupts that alter the code

execution path. Deterministic replay could be achieved by recording non-deterministic inputs and then by injecting those inputs at

the same execution point during replay (see Figure 1). This method greatly reduces processing resources and space as compared to

exhaustively recording and replaying individual instructions.

Figure 1. Event Injection during Replay

Disk I/O Timer Event

In order to eciently inject the inputs at the correct execution point, some processor changes were required. VMware collaborated

with AMD and Intel to make sure all currently shipping Intel and AMD server processors support these changes. See KB article

1008027 for a list of supported processors.

VMware currently supports record/replay only for uniprocessor virtual machines. Record/Replay of symmetric multi-processing (SMP)

virtual machines is more challenging because in addition to recording all external inputs, the order of shared memory access also has

to be captured for deterministic replay.

1.2. Fault Tolerance Logging Traffic

Figure 2 shows the high level architecture of VMware Fault Tolerance.

VMware FT relies on deterministic record/replay technology described above. When VMware FT is enabled for a virtual machine (“the

primary”), a second instance of the virtual machine (the “secondary”) is created by live-migrating the memory contents of the primary

using VMware® VMotion™. Once live, the secondary virtual machine runs in lockstep and eectively mirrors the guest instruction

execution of the primary.

The hypervisor running on the primary host captures external inputs to the virtual machine and transfers them asynchronously to the

secondary host. The hypervisor running on the secondary host receives these inputs and injects them into the replaying virtual machine

at the appropriate execution point. The primary and the secondary virtual machines share the same virtual disk on shared storage, but

all I/O operations are performed only on the primary host. While the hypervisor does not issue I/O produced by the secondary, it posts

all I/O completion events to the secondary virtual machine at the same execution point as they occurred on the primary.

VMWARE WHITE PAPER

Figure 2. High Level Architecture of VMware Fault Tolerance

VMware

APP

VMware

APP

Primary

Record

Client

Shared Storage

FT Logging Trac

ACKs

Secondary

Replay

The communication channel between the primary and the secondary host is established by the hypervisor using a standard

TCP/IP socket connection and the trac owing between them is called FT logging trac. By default, incoming network trac and

disk reads at the primary virtual machine are captured and sent to the secondary, but it is also possible to make the secondary virtual

machine read disk I/O directly from the disk. See KB article 1011965 for more information about this alternative mode.

1.3. VMware vLockstep Interval

The primary virtual machine’s execution is always ahead of the secondary with respect to physical time. However, with respect to

virtual time, both the primary and secondary progress in sync with identical execution state. While the secondary’s execution lags

behind the primary, the vLockstep mechanism ensures that the secondary always has all the information in the log to reach the same

execution point as the primary. The physical time lag between the primary and secondary virtual machine execution is denoted as

the vLockstep interval in the FT summary status page.

Figure 3: vLockstep Interval in the FT Summary Status Page

The vLockstep interval is calculated as a moving average and it assumes that the round-trip network latency between the primary

and secondary hosts is constant. The vLockstep interval will increase if the secondary virtual machine lacks sucient CPU cycles to

keep up with the primary. Under this circumstance, whenever the primary virtual machine becomes idle (for example while waiting

for an I/O completion) the secondary will catch up and the vLockstep interval will reduce. If the vLockstep interval is consistently

high, the hypervisor may slow the primary virtual machine to let the secondary catch up.

1.4. Transparent Failover

FT ensures that there is no or data or state loss in the virtual machine when the failover happens. Also, after a failover, the new

primary will perform no I/O that is inconsistent with anything previously issued by the old primary. This is achieved by ensuring that

the hypervisor at the primary commits to any externally visible action, such as network transmits or disk writes, only after receiving an

acknowledgement from the secondary that it has received all the log events preceding that event.

VMWARE WHITE PAPER

2. Performance Aspects and Best Practice Recommendations

This section describes the performance aspects of Fault Tolerance with best practices recommendations to maximize performance.

For operational best practices please refer to the VMware Fault Tolerance Recommendations and Considerations on VMware

vSphere 4 White Paper.

2.1. FT Operations: Turning On and Enabling

There are two types of FT operations that can be performed on a virtual machine: Turning FT on or o, and enabling or disabling FT.

The performance implications of these operations are as follows:

“Turn On FT” prepares the virtual machine for FT.

• WhenFTisturnedon,devicesthatarenotsupportedwithFTarepromptedforremoval,andthevirtualmachinesmemory

reservation is set to its memory size to prevent ballooning or swapping.

• Useofprocessor’shardwareMMUfeature(AMDRVI/IntelEPT)resultsinnon-determinismandthereforeitisnotsupported

with FT. When FT is turned on for a virtual machine, hardware MMU feature is disabled for that virtual machine. However, virtual

machines that don’t have FT turned on can take advantage of hardware MMU on the same host.

• TurningonFTwillnotsucceedifthevirtualmachineispoweredonandisusingHardwareMMU.Inthiscase,thevirtualmachine

first needs to be either powered off, or migrated to a host that does not have hardware MMU. Similarly turning off FT on a

powered-on virtual machine would not make the virtual machine automatically use hardware MMU; the virtual machine would

need to be powered off and powered back on or migrated to a host that supports hardware MMU for the changes to take effect.

Please see KB article 1008027 for more information on which guest OS and CPU combination requires power on/off operations

for changes to take effect.

“Enable FT” operation enables Fault Tolerance by live-migrating the virtual machine to another host to create a secondary

virtual machine.

• Sincelive-migrationisaresource-intensiveoperation,limitingthefrequencyofenable/disableFToperationsisrecommended.

• Thesecondaryvirtualmachineusesadditionalresourcesonyourcluster.ThereforeiftheclusterhasinsufficientCPUormemory

resources, the secondary will not be created.

When “Turn on FT” operation succeeds for a virtual machine that is already powered on, it automatically creates a new secondary

virtualmachine.Soithasthesameeectas“EnablingFT”.

2.2. Resource Consumption

The additional resource requirements for running a virtual machine with Fault Tolerance enabled are as follows:

• CPUcyclesandmemoryforrunningthesecondaryvirtualmachine

• CPUcyclesforrecordingontheprimaryhostandreplayingonthesecondaryhost

• CPUcyclesforsendingFTloggingtrafficfromtheprimaryhostandreceivingitonthesecondary

• NetworkbandwidthfortheFTloggingtraffic

Record and replay may consume dierent amounts of CPU depending on the event being recorded and replayed, and as a result,

slight dierences in the CPU utilization of the primary and the secondary virtual machines is common and can be ignored.

2.3. Secondary Virtual Machine Execution Speed

As explained in section 1.3, the hypervisor may slow down the primary virtual machine if the secondary is not keeping up pace with

the primary. Secondary virtual machine execution can be slower than the primary for a variety of reasons:

• ThesecondaryhosthasaCPUwithasignificantlylowerclockfrequency

• Powermanagementisenabledonthesecondaryhost,causingtheCPUfrequencytobescaleddown

• ThesecondaryvirtualmachineiscontendingforCPUwithothervirtualmachines

VMWARE WHITE PAPER

To ensure that the secondary virtual machine runs as fast as the primary, it is recommended that:

• ThehostsintheFTclusterarehomogenous,withsimilarCPUmake,model,andfrequency.TheCPUfrequencydifferenceshould

not exceed 400 MHz.

• Boththeprimaryandsecondaryhostsusethesamepowermanagementpolicy.

• CPUreservationissettofullforcaseswherethesecondaryhostcouldbeoverloaded.TheCPUreservationsettingontheprimary

applies to the secondary as well, so setting full CPU reservation ensures that the secondary gets CPU cycles even when there is

CPU contention.

2.4. I/O Latencies

All incoming network packets to the primary, and all disk reads at the primary, are immediately sent to the secondary. However, as

explained in section 1.4, network transmits and disk writes at the primary are held until the secondary acknowledges that all events

that precede the packet transmit or disk write. As a result, the round-trip network latency between the primary and the secondary

aectstheI/Olatencyofdiskwritesandnetworktransmitoperations.SincetheroundtriplatencyinaLANenvironmentisusuallyin

the order of a few hundred microseconds, and disk I/O latencies are usually on the order of a few milliseconds, this delay does

not impact disk write operations. One may, however, notice delays in network ping responses if the response time is shown in

microseconds. For best performance, it is recommended that the round-trip network latency between the primary and secondary

host be less than 1 millisecond.

2.5. Network Link

Since the primary and secondary virtual machines proceed in vLockstep, the network link between the primary and the

secondary host plays an important role in performance. A Gigabit link is required to avoid congestion. In addition, higher bandwidth

network interfaces generally have lower transmission latency. If the network is congested and the primary host is not able to send

trac to the secondary (i.e. when the TCP window is full), then the primary virtual machine will make little or no forward progress. If

the network connection between the primary and secondary hosts goes down, either the current primary or the current secondary

virtual machine will take over, and the other virtual machine will die.

2.6. NIC Assignments for Logging Traffic

FT generates two types of network trac:

• Migrationtraffictocreatethesecondaryvirtualmachine

• FTloggingtraffic

MigrationtrachappensovertheNICdesignatedforVMotionanditcausesnetworkbandwidthusagetospikeforashorttime.

SeparateanddedicatedNICsarerecommendedforFTloggingtracandVMotiontrac,especiallywhenmultipleFTvirtual

machinesresideonthesamehost.SharingthesameNICforbothFTloggingandVMotioncanaecttheperformanceofFTvirtual

machines whenever a secondary is created for another FT pair or a VMotion operation is performed for any other reason.

VMwarevSwitchnetworkingallowsyoutosendVMotionandFTtractoseparateNICswhilealsousingthemasredundantlinksfor

NICfailover.SeeKB article 1011966 for more information.

Adding multiple uplinks to the virtual switch does not automatically result in distribution of FT logging trac. If there are multiple FT

pairs, then trac could be distributed with IP-hash based load balancing policy, and by spreading the secondary virtual machines to

dierent hosts.

2.7. Virtual Machine Placement

FT logging trac is asymmetric: the bulk of the trac ow happens from the primary to the secondary hosts and the secondary host

only sends back acknowledgements. If multiple primary virtual machines are co-located on the same host, they could all compete for

thesamenetworkbandwidthontheloggingNIC.Idlevirtualmachinesconsumelessbandwidth,butI/O-intensivevirtualmachines

can consume a lot of network bandwidth. It can be helpful to place the primary of one FT pair and the secondary of another FT pair

onthesamehosttobalancethetracontheFTloggingNIC.

VMWARE WHITE PAPER

It is recommended that FT primary virtual machines be distributed across multiple hosts and, as a general rule of thumb, the number

of FT virtual machines be limited to four per host. In addition to avoiding the possibility of saturating the network link, it also reduces

the number of simultaneous live migrations required to create new secondary virtual machines in the event of a host failure.

2.8. DRS and VMotion

DRS takes into account the additional CPU and memory resources used by the secondary virtual machine in the cluster, but

DRS does not migrate FT enabled virtual machines to load balance the cluster. If either the primary or secondary dies, a new

secondary is spawned and is placed on the candidate host determined by HA. The candidate host determined by HA may not be an

optimal placement for balancing, however one can manually VMotion either the primary or the secondary virtual machines to a

dierent host as needed.

2.9. Timer Interrupts

Though timer interrupts do not signicantly impact FT performance, all timer interrupt events must be recorded at the primary

and replayed at the secondary. This means that having a lower timer interrupt rate results in a lower volume of FT logging trac.

The following table illustrates this.

Guest OS Timer interrupt rate Idle VM FT trac

RHEL5.064-bit 1000 Hz 1.43 Mbits/sec

SLES10SP232-bit 250 Hz 0.68 Mbits/sec

Windows2003DatacenterEdition 82 Hz 0.15 Mbits/sec

Where possible, lowering the timer interrupt rate is recommended. See KB article 1005802 for more information on how to reduce

timer interrupt rates for Linux guest operating systems.

2.10. Fault Tolerance Logging Bandwidth Sizing Guideline

As described in section 1.2, FT logging network trac depends on the number of non-deterministic events and external inputs that

need to be recorded at the primary virtual machine. Since the majority of this trac usually consists of incoming network packets

and disk reads, it is possible to estimate the amount of FT logging network bandwidth (in Mbits/sec) required for the virtual machine

using the following formula:

FT logging bandwidth ~= [ (Average disk read throughput in Mbytes/sec * 8) + Average network receives (Mbits/sec) ] * 1.2

In addition to the inputs to the virtual machine, this formula reserves 20 percent additional networking bandwidth for recording

non-deterministic CPU events and for the TCP/IP headers.

3. Fault Tolerance Performance

This section discusses the performance characteristics of Fault Tolerant virtual machines using a variety of micro-benchmarks and

real-life workloads. Micro-benchmarks were used to stress CPU, disk, and network subsystems individually by driving them to

saturation. Real life workloads, on the other hand, have been chosen to be representative of what most customers would run and

they have been congured to have a CPU utilization of 60 percent in steady state. Identical hardware test beds were used for all the

experiments, and the performance comparison was done by running the same workload on the same virtual machine with and

without FT enabled. The hardware and experimental setup details are provided in the Appendix. For each experiment, the trac on

theFTloggingNICduringthesteadystateportionoftheworkloadisalsoprovidedasareference.

3.1. SPECjbb2005

SPECjbb2005isanindustrystandardbenchmarkthatmeasuresJavaapplicationperformancewithparticularstressonCPUand

memory. The workload is memory intensive and saturates the CPU but does little I/O. Because this workload saturates the CPU and

generates little logging trac, its FT performance is dependent on how well the secondary can keep pace with the primary.

VMWARE WHITE PAPER

Figure 4. SPECjbb2005 Performance

100

120

FT Disabled

FT Enabled

SPECjbb2005

Relative performance (%)

RHEL 5 64-bit, 4GB

FT trac:

1.4 Mbits/sec

3.2. Kernel Compile

This experiment shows the time taken to do kernel compile, which is both CPU and MMU intensive workload due to forking of many

parallel processes. As with the previous experiment, CPU is 100 percent utilized and thus FT performance is dependent on how well

the secondary can keep pace with the primary. This workload does some disk reads and writes, but generates no network trac.

Besides timer interrupt events, the FT logging trac includes the disk reads. As seen in theFigure 5, the performance overhead of

enabling FT was very small.

Figure 5. Kernel Compilation Performance

100

150

200

250

300

350

FT Disabled

FT Enabled

Kernel Compile

(lower is better)

Seconds

SLES 10 32-bit, 512MB

FT trac:

3 Mbits/sec

VMWARE WHITE PAPER

3.3. Netperf Throughput

Netperfisamicro-benchmarkthatmeasuresthethroughputofsendingandreceivingnetworkpackets.Inthisexperimentnetperf

was congured so packets could be sent continuously without having to wait for acknowledgements. Since all the receive trac

needs to be recorded and then transmitted to the secondary, netperf Rx represents a workload with signicant FT logging trac.

As shown inFigure 6, with FT enabled, the virtual machine received 890 Mbits/sec of trac while generating and sending 950 Mbits/

sec of logging trac to the secondary. Transmit trac, on the other hand, produced relatively little FT logging trac, mostly from

acknowledgement responses and transmit completion interrupt events.

Figure 6. Netperf Throughput

100

200

300

500

400

600

700

800

900

1000

FT Disabled

FT Enabled

Netperf throughput

Mbits/sec

Receives Transmits

FT trac Rx: 950 Mbits/sec

FT trac Tx: 54 Mbits/sec

3.4. Netperf Latency Bound Case

In this experiment, netperf was congured to use the same message and socket size so that outstanding messages could only be

sent one at a time. Under this setup, the TCP/IP stack of the sender has to wait for an acknowledgment response from the receiver

before sending the next message and, thereby, any increase in latency results in a corresponding drop in network throughput.

Noteinrealityalmostallapplicationssendmultiplemessageswithoutwaitingforacknowledgement,soapplicationthroughputis

not impacted by any increase in network latency. However since this experiment was purposefully designed to test the worst-case

scenario, the throughput was made dependent on the network latency. There are not any known real world applications that would

exhibit this behavior. As discussed in section 1.4, when FT is enabled, the primary virtual machine delays the network transmit until

the secondary acknowledges that it has received all the events preceding the transmission of that packet. As expected, FT enabled

virtual machines had higher latencies which caused a corresponding drop in throughput.

VMWARE WHITE PAPER

Figure 7. Netperf Latency Comparison

100

200

300

500

400

600

700

800

900

1000

FT Disabled

FT Enabled

Netperf - latency sensitive case

Mbits/sec

Receives Transmits

FT trac

Rx: 500 Mbits/sec

Tx: 36 Mbits/sec

3.5. Filebench Random Disk Read/Write

Filebench is a benchmark designed to simulate dierent I/O workload proles. In this experiment, lebench was used to generate

randomI/Osusing200workerthreads.Thisworkloadsaturatesavailablediskbandwidthforthegivenblocksize.EnablingFTdidnot

impact throughput; however, at large block sizes, disk read operations consumed signicant networking bandwidth on the

FTloggingNIC.

Figure 8. Filebench Performance

1000

2000

3000

5000

4000

6000

7000

8000

9000

FT Disabled

FT Enabled

Filebench

IOPS

2KB read 64KB read 2KB write 64KB write 2KB read/write

FT trac

2K read: 155 Mbits/sec

2K write: 3.6 Mbits/sec

64K read: 1400 Mbits/sec

64K write: 1.96 Mbits/sec

2K read/write: 34 Mbits/sec

Autres manuels pour vSphere 4

Table des matières

Autres manuels VMware Logiciel

VMware

VMware VCM 5.3 - CONFIGURATION MANAGER SECURITY ENVIRONMENT... Guide rapide

VMware

VMware VCENTER CONVERTER EN-000158-02 Manuel utilisateur

VMware

VMware VCENTER CONFIGURATION MANAGER 5.3 - SOFTWARE REQUIEREMENTS... Guide rapide

VMware

VMware VCLOUD DIRECTOR Manuel utilisateur

VMware

VMware VSHIELD APP 1.0.0 UPDATE 1 - API Mode d'emploi

VMware

VMware VIEW 4.5 Manuel

VMware

VMware CLOUD DIRECTOR 1.0 Manuel

VMware

VMware VSHIELD APP Manuel utilisateur

VMware

VMware VCENTER CONFIGURATION MANAGER 5.3 - VCENTER DISCOVERED MACHINES IMPORT TOOL... Manuel utilisateur

VMware

VMware VSHIELD APP 1.0.0 UPDATE 1 - API Manuel

VMware

VMware VFABRIC GEMFIRE Manuel utilisateur

VMware

VMware VCENTER CONFIGURATION MANAGER 5.3 Manuel utilisateur

VMware

VMware VCLOUD SDK 1.0 - FOR NET DEVELOPERS GUIDE Manuel utilisateur

VMware

VMware CLOUD DIRECTOR 1.0 Manuel utilisateur

VMware

VMware VSHIELD Manuel utilisateur

VMware

VMware DISK MOUNT UTILITY WORKSTATION 5.5 Manuel utilisateur

VMware

VMware VFABRIC TC SERVER Manuel utilisateur

VMware

VMware THINAPP 4.6 - MIGRATING APPLICATIONS TECHNICAL... Guide de l'utilisateur

VMware

VMware VCENTER CHARGEBACK Manuel utilisateur

VMware

VMware VSPHERE 4.0 - SOFTWARE COMPATIBILITY MATRIX Manuel utilisateur

VMware

VMware VCENTER CHARGEBACK 1.5 - API Manuel utilisateur

VMware

VMware ESX 4.0 - GETTING STARTED UPDATE 1 Manuel utilisateur

VMware

VMware VCLOUD REQUEST MANAGER 1.0.0 Manuel utilisateur

VMware

VMware VCENTER APPLICATION DISCOVERY MANAGER 6.1.1 -... Manuel utilisateur

Manuels Logiciel populaires d'autres marques

COMPRO

COMPRO COMPROFM Manuel utilisateur

Muratec

Muratec OFFICEBRIDGE ONLINE Manuel utilisateur

Oracle

Oracle Contact Center Anywhere 8.1 Manuel utilisateur

Adobe

Adobe 65007312 - Photoshop Lightroom Manuel

Avaya

Avaya NULL One-X for RIM Blackberry Manuel utilisateur

HP ProLiant BL420c Manuel utilisateur

PS Audio

PS Audio PowerPlay Manuel du propriétaire

Brady

Brady LOCKOUT PRO 3.0 Manuel de service

Avaya

Avaya Interaction Center Manuel utilisateur

Texas Instruments

Texas Instruments TI-83 Plus Silver Edition Manuel

Novell

Novell GROUPWISE 8 - INTERNET AGENT Manuel utilisateur

Oracle

Oracle Application 9i Manuel utilisateur

Acer

Acer RDM Manuel utilisateur

Canon

Canon Vixia HF21 Manuel utilisateur

Canon

Canon ZR950 Manuel utilisateur

Samsung

Samsung Auto Backup Manuel utilisateur

Polycom

Polycom Vortex EF2201 Instructions d'installation et d'utilisation

Brocade Communications Systems

Brocade Communications Systems Brocade 8/12c Manuel utilisateur

VMware vSphere 4 Manuel d'installation et d'exploitation

Manuels Logiciel populaires d'autres marques