Last year, I found a curious bug in Windows regarding the handlingof certain invalid opcode sequences. At the time, I simply documentedit and then forgot about it. Recently, however, I was reminded of thebug, so I thought that other people might be interested in readingabout it.
Because of the way in which the Intel x86 architecture works, whenan invalid opcode exception occurs, there is no easy way to tell why itoccurred. By this, I mean that without actually looking at the faultingopcode sequence, it's not possible to tell the difference between anunsupported opcode and an invalid use of the LOCK prefix. For thisreason, Windows runs this code:
mov ecx, 4 ;maximum prefix count
mov al, byte ptr es:[esi] ;points to faulting opcode sequence
cmp al, f0h ;looks like LOCK?
je op_lock ;yes
add esi, 1 ;no, continue with next byte
loop look_op ;until no more bytes
mov eax, c000001d ;STATUS_ILLEGAL_INSTRUCTION
mov eax, c000001e ;STATUS_INVALID_LOCK_SEQUENCE
While there are only three classes of prefixes that can appear inaddition to an otherwise valid lock sequence (segment override,operand-size override, and address-size override), no current CPUinstruction allows REP to be combined with LOCK. This is the reason forthe value of 4 in ecx. The bug is that Windows checks for only
the LOCK prefix and no other. Thus, if the value "f0" happens to appearanywhere within the first four bytes of the faulting opcode sequence, even if it is not truly a LOCK (e.g. fe f0), then Windows will return the wrong exception value.
This is a particular problem for operating-system emulators, sincesuch a condition would occur only rarely, so it seems likely that noone has support for this behaviour. Surprise!