The way I understand ULP is that it is the gap between two consecutive floating point numbers. The book I'm reading says that ULP = machine epsilon times two to the exponent. This seems correct to me only if the two numbers have the same exponent.
For instance, say you have a 5 bit mantissa and a 3 bit exponent, and you have the floating point number 11111 100. In order to find the next largest floating point, you can't simply add 1/32 to the mantissa, because you would overflow. So, can you even say a number like 11111 100 has a ULP?
“ULP” stands for “Unit of Least Precision.” So, going by name, it is the position value of the lowest digit in the significand1. With this, the ULP of the largest representable finite number is the so-called epsilon scaled by the exponent.
As njuffa notes, there are multiple definitions of an ULP or similar concepts, so which definition you use depends on what purpose you want to serve. It is useful for discussing or reasoning about the spacing between floating-point numbers, but there are various features that have to be taken into account.
For example, it is incorrect to state an ULP is the distance to the next representable value. One reason is, as you have noted, that fails at the largest representable finite number. Another is it fails for negative numbers at each point where the exponent changes. The distance from −1 to the next greater representable number is, in a base-two format, half an ULP of −1, not a full ULP, due to the exponent decrease.
Rounding in floating-point arithmetic operations behaves as if there were another representable value one ULP greater than the largest representable finite value. If rounding would produce that next number, ∞ is produced instead. So, in a certain way, the ULP is the step size to the “next” number.
Another issue with ULP arises when you want to discuss the spacing between floating-point numbers at a certain number, but that number is not representable in the floating-point format; it is between two representable numbers. Since it is not representable in the format, it does not have a significand in the format, so it does not have a lowest digit in its significand. You need a measurement other than a nominal ULP.
Mostly I prefer to use ULP to mean the spacing of representable numbers in the exponent interval containing the absolute value of the number, [be, be+1), and then discuss/prove whatever error, precision, or rounding property I need in terms of that unit. But that definition still needs to be completed for an interval around zero and for beyond the largest representable finite value.
1 “Significand” is the preferred term for the fraction part of a floating-point number. “Mantissa” is an old term for the fraction part of a logarithm. Mantissas are logarithmic; adding to the mantissa multiplies the number. Significands are linear; adding to the significand adds to the number (as scaled by the exponent).