A while ago, I wrote
an article about NDS caching
, how it can interfere with DMA transfers and what you can do about them.
A little later I got a
pingback from ant512,
who had tried the “safe” DMA routines I made and found they
weren’t nearly as safe as I’d hoped. I’m still not sure what the actual
problem was, but this incident did make me think about one possible
reason, namely the one that will be discussed in this post: problematic
cache invalidation.

1 Test base

But first things first. Let’s start with some simple test code, see below.
We have a simple struct definition, two arrays using this struct, and
some default data for both arrays that we’ll use later.

And now we’re going to do some simple things with these arrays that
we always do: some reads, some writes, and a struct copy. And for the
copying, I’m going to use DMA, because DMA-transfers are fast,
amirite(1)? The specific actions I will do are the following:

Initialization

• Zero out g_src and g_dst.
• Initialize the arrays with some data, in this case from
  c_fooIn.
• Cache-Flush both arrays to ensure they’re uncached.

Testing

• Modify element in g_src, namely g_src[0].
• Modify an element in g_dst, namely g_dst[3].
• DMA-copy g_src[0] to g_dst[3].

In other words, this:

Note that there is nothing spectacularly interesting going on here.
There’s just your average struct definition, run of the mill array
definitions, and boring old accesses without even any pointer magic
that might hint at something tricky going on. Yes, alignment is forced,
but that just makes the test more reliable. Also, the fact that I’m
incrementing Foo.id rather than just reading from it is
only because ARM9 cache is read-allocate, and I need to have these
things end up in cache. The main point is that the actions in
test_dmaCopy() are very ordinary.

2 Results

It should be obvious what the outcome of the test should be. However,
when you run the test (on hardware! not emulator), the result seems to
be something different.

Notice that the changed values of g_src[0] never
end up in g_dst[0]. Not only that, not even the
original values g_src[0] have been copied.
It’s as if the transfer never happened at all.

The reason for this was covered in detail in the original article.
Basically, it’s because cache is invisible to DMA. Once a part of
memory is cached, the CPU only looks to the contents of the cache and
not the actual addresses, meaning that DMA not only reads out-of-date
(stale) source data, but also puts it where the CPU won’t look.
Two actions allow you to remedy this. The first is the
cache flush, which write the cache-lines back to RAM and
frees the cache-line. Then there’s cache invalidate, which
just frees the cache-line. Note that in both cases, the cache is
dissociated from memory.

With this information, it should be obvious what to do. When DMA-ing
from RAM, you need to flush the cache before the transfer to update
the source’s memory. When DMA-ing to RAM, you need to invalidate
after the transfer because now it’s actually the cache’s data that’s
stale.
When you think about it a little this makes perfect sense, and it’s easy
enough to implement:

Unfortunately, this doesn’t work right either. And if you think about
it a lot instead of merely a little, you’ll see why. Maybe showing the
results will make you see what I mean. The transfer seems to work now,
but the earlier changes to g_dst[3] have been erased. How
come?

The problem is that a cache-invalidate invalidates entire
cache-lines, not just the range you supply. If the start or
end of the data you want invalidate does not align to a cache-line,
the adjacent data contained in that line is also thrown away. I hope
you can see that this is bad.

This is exactly what’s happening here. The ARM9′s cache-lines are 32
bytes in size. Because of the alignment I gave the arrays, elements
0 through 3 lie on the same cache-line. The changes to
g_dst[3] occur in cache (but only because I read from it
through +=). The invalidate of g_dst[0]
also invalidates g_dst[3], which throws out the
perfectly legit data and you’re left in a bummed state. And again,
I’ve done nothing spectacularly interesting here; all I did was
modify something and then invalidated data that just happened to be
adjacent to it. But that was enough. Very, very bad.

Just to be sure, this is not due to a bad implementation of
DC_InvalidateRange(). The function does exactly what it’s
supposed to do. The problem is inherent in the hardware. If your
data does not align correctly to cache-lines, an invalidate will apply
to the adjacent data as well. If you do not want that to happen, do
not invalidate.

3 Solutions

So what to do? Well, there is one thing, but I’m not sure how foolproof
this is, but instead of invalidating the destination afterwards, you
can also flush it before the transfer. This frees up the cache-lines
without loss of data, and then it should be safe to DMA-copy to it.

Alternatively, you can also disable the underlying reason behind the
problem with invalidation: the write-buffer. The ARM9 cache allows
two modes for writing: write-through, which also updates
the memory related to the cache-line; and write-back, which
doesn’t. Obviously, the write-back is faster, so that’s how libnds
sets things up. I know that putting the cache in write-through mode
fixes this problem, because in libnds 1.4.0 the write-buffer had been
accidentally disabled and my test cases didn’t fail. This is probably
not the route you want to take, though.

4 Conclusions

So what have we learned?

• Cache – DMA interactions suck and can cause really subtle bugs.
  Ones that will only show up on hardware too.
• Cache-flushes and invalidates cover the cache-lines of the requested
  ranges, which exceed the range you actually wanted.
• To safely DMA from cachable memory, flush the source range first.
• Contrary to what I wrote earlier, to DMA to cachable memory,
  do not cache-invalidate – at least not when
  the range is not properly aligned to cache-lines. Instead, flush
  the destination range before the transfer (at which time
  invalidation should be unnecessary). That said, invalidate should
  still be safe if the write-buffer is disabled.

Link to test code.

The short story is that officially new should throw an
exception when it can’t allocate new memory. Exceptions come with about
60 kb worth of baggage. Yes, this is more or less the same stuff that
goes into vector and string.

The long story, including a detailed look at a minimal binary,
a binary that uses new and a solution to the exception overhead (in this particular case anyway) can be read below the fold.

• 1 Minimal project
• 2 Standard C++ new/delete
• 3 Custom new/delete
• 4 Other considerations and conclusions.

1 Minimal project

The following is essentially an empty project. It should represent
the smallest binary you can get with the current DKA (r26) and
libnds (1.3.7). This is the primary reference case.

This actually already leads to a binary of 53.5 kb. To analyze what
goes on in there, we can look at the map file. Not the mapfile
generated by the linker, mind you, but by the arm-eabi-nm tool,
whose generated files are considerably easier to read. To use this tool,
add the following line to $(BUILD) rule in the makefile,
so that it looks like below. If you want to know what the flags mean,
please RTFM.

And this is the resulting mapfile, in full.

Now, I expect you can’t really tell much from this, so here’s a summary.

The 0100:xxxx and 0B00:xxxx ranges belong to
ITCM and DTCM, so those are irrelevant when looking at main RAM size.
The libc, impure and misc bookkeeping sections are stuff related to the
C library and C overhead, accounting for about 1.5 kb. The boot code,
crt0.S also covers close to 1.0 kb. As expected, the code for
main.c –the actual project– is more or less
nothing.

The rest, about 7 kb, is libnds. Now, you may say that this is quite a bit
of overhead, but it really isn’t. Pretty much all of it relates to
interrupts and the fifo system, which takes care of ARM7-ARM9
communication. You need to have these parts. Okay, you could try
to roll your own to shrink this down to the bare essentials, but in all
likelihood that’s more trouble than it’s worth.

The observant of you should have noticed something: we’re only at 9.5 kb,
but the file size is 53.5 kb. So what the hell happened to the other 44 kb?
Well, I don’t know, to be honest. It doesn’t appear in MWRAM to be sure.
It’s probably the stuff ndstool adds. My guess it that that’s
where the ARM7 binary goes, along with the icon, titles and possibly
DLDI interfaces, but I really can’t say right now.

2 Standard C++ new/delete

And now, let’s look at what happens when you invoke new.

Just this small thing increases the file size to 117 kb! And remember,
that’s not merely a doubling of the size, as 44 kb of the binary is not
put in memory. The memory load has gone from about 10 kb to over 70 kb.
What causes this increase? Well, let’s see:

Well, I did say this was going to be the long story, didn’t I?
Everything that was in the base project is in here as well. The
additional parts can summarized as follows.

There are three main areas to discern:

• d_*() routines, presumably for debug printing.
  (size: 12k)
• Stdio formatting and related. This includes file handling, device
  handling and many forms of printf, which brings a
  whole lot of bagage (some allocation, format parsing and
  math/floating point routines). There’s also some abort and
  signalling routines. (size: 28k).
• Exception handling. Not just routines for handling them, but also
  the typeinfo stuff required, the output strings and the output
  string pointers. (size: 18k)

These roughly 60k of stuff is the overhead of exceptions –
any potential exception. In this case, it’s because
new requires a bad_alloc exception when it’s
unable to allocate more.

The problem is that exceptions have
many dependencies: to do exception handling, you keep track and unwind
the stack. You also need to be able to tell the type of exception
thrown, which requires RTTI. And then you say which exception was
thrown, so you need error messages, and a list of pointers to
those messages, and a way to format and write those messages,
hence the d_*() routines and all the stdio stuff.

3 Custom new/delete

There is a way around this, though: redefine new and
related functions. Technically speaking, this is a bad idea
if you don’t know what you’re doing, but it can be done. Note that
you would need overload four operators: new,
delete and their array counterparts.

This way, you just incur the cost of malloc() and
free(), which are only about 3k. But again, this is going
against the standard and you’ll really have to ask yourself if the
(at best) 2% of main RAM you save with this is really worth it.

More on this can be read at

http://brewforums.qualcomm.com/showthread.php?t=2033.

4 Other considerations and conclusions.

The binary size is not the same as the main RAM footprint.
About 44 kb other stuff.

The overhead of the standard new is 60 kb, which is all
due to exceptions. You cannot remove it by
using the compiler options -fno-exceptions and
-fno-rtti, because that only affects your own code, not the
standard libraries. You can remove this overhead by using overloading
new and related functions, but you have to be really
careful with this.

I’ve also done a little bit of testing with vector, and
it seems that vector‘s overhead also comes from
new and can be removed the same way. However, other parts
of vector (and STL) may use other exceptions, so it’s
quite possible it won’t work in all cases.

Note that roughly 28 kb of the exception overhead is actually
stdio related – specifically formatted printing:
*printf. If you’re using printf anyway, the
effective overhead of exceptions is reduced considerably.

Finally, remember that the exception overhead amounts to roughly 2% of
main RAM at most. In most homebrew cases it won’t matter that much.
When it does start to affect your app, you will likely have other parts
that are easier and safer to optimize out.

Test project + notes.

The integer datatypes in C can be either signed or unsigned. Sometimes,
it’s obvious which should be used; for negative values you clearly
should use signed types, for example. In many cases there is no obvious
choice – in that case it usually doesn’t matter which you use.
Usually, but not always. Sometimes, picking the wrong kind
can introduce subtle bugs in your program that, unless you know what
to look out for, can catch you off-guard and have you searching for
the problem for hours.

I’ve mentioned a few of these occasions in Tonc
here and
there, but I think
it’s worth going over them again in a little more detail. First, I’ll
explain how signed integers work and what the difference between signed
and unsigned and where potential problems can come from. Then I’ll
discuss some common pitfalls so you know what to expect.

• 1 Basics
• 2 Potential problems
• 3 Summary

1 Basics

The signedness of a variable refers to whether it can be
used to represent negative values or not. Unsigned variables can only
have positive values; signed values can be both positive or negative.

In the computer world, signedness is mostly a matter of interpretation.
Say you have a variable that is N bits long. This is enough
room for 2^N distinct numbers, but it says nothing about
which range of numbers you should be using them for. Interpreted as
unsigned integers, its range would be [0, 2^N−1].
Under a signed interpretation, you’d use some bit-patterns for negative
numbers. There are actually several ways of doing this, but the most
commonly used is known as two’s complement which leads to
a [−2^N−1, 2^N−1−1] range:
half positive and half negative.

1.1 Two’s complement theory

Two’s complement is sometimes seen as an awkward system, but it
actually follows quite naturally when you only have a fixed number
of digits to write down numbers with.
Consider the whole line of positive and negative integers. As you
move away from zero, the numbers will grow larger and larger.
Now suppose you have an
counting device
composed of a limited number of digits, each of which can only display
numbers 0 through 10−1. With N digits, you only have
room for 10^N different numbers, and once those are used up
(at 10^N−1), the counter returns to 0 and counting
effectively resets. In essence, the number on the counter works in
modulo 10^N.

The key is that this works in both positive and negative directions.
As far as the counter is concerned, 0 and 10^N are the
same thing. This being the case, you can argue that −1
(that is, the number before zero) is equivalent to 10^N−1;
and −2 ≡ 10^N−2, and so on.
Note that this works regardless of what 10 actually is; it can be
ten (decimal), two (binary) or sixteen (hexadecimal).

The 10^N possible numbers form a window over the number line,
but where the window starts is up to the user. For signed numbers,
you can move the window so that the upper half of the 10^N
range is interpreted as negative numbers.

Fig 1 shows how this works for 8-bit numbers
(written in hex for convenience). The black numbers represent the
entire number line, where numbers can have as many digits as you
need. With only two nybbles, the counter repeats every
100h = 256 values. FFh, 1FFh, but also −1 all reduce to the same
symbol, namely FFh. In Fig 2 you can see
how the available symbols are mapped to either signed or unsigned
values. In the unsigned case, numbers simply count from 0 to FFh;
for signed, the top half of the symbol range is put on the left side
of zero and are used for negative numbers.

Fig 1.

Fig 2.

The mathematical reason behind all this like this. Assume for
convenience that N = 1, so that 0 is equivalent to 10 and
in fact every multiple of 10. By definition, subtracting a value
from itself gives 0. Because subtraction is merely addition by its
negative value, you get the following:

(1)	title="\begin{eqnarray} x &-& x &=& 0& \\ x &+& (-x) &=& 0 & \\ x &+& (-x) &\equiv& 10 & \\ & & (-x) &=& 10 &- x \end{eqnarray}" alt="\begin{eqnarray} x &-& x &=& 0& \\ x &+& (-x) &=& 0 & \\ x &+& (-x) &\equiv& 10 & \\ & & (-x) &=& 10 &- x \end{eqnarray}" />

The term −x in the last step should be seen as a unit,
call it C. Numerically, C is the number that, when added
to x, gives 10. In decimal, if x = 1, then C = 9.
C is called the 10′s complement of x, because
it’s what’s needed to complete the 10. It’s called the two’s
complement in binary, because then 10 equals two.

In binary, there is an alternative to calculate the twos complement
of a number. Subtracting a number from 2^N is equivalent to
inverting all its bits, so you get:

(2)	title="\begin{eqnarray} (-x) &=& 2^N - x \\ &=& 2^N -1 - x + 1 \\ (-x) &=& \sim x + 1 \\ \end{eqnarray}" alt="\begin{eqnarray} (-x) &=& 2^N - x \\ &=& 2^N -1 - x + 1 \\ (-x) &=& \sim x + 1 \\ \end{eqnarray}" />

Using two’s complement(1)
for negative numbers has some interesting
properties. First, subtraction and addition are basically the same
thing. This is nice for arithmetic implementers for two reasons:
the same hardware can be used for both operations, and it can be used
for both positive and negative numbers.

Second, because the top half
is now used for negative numbers, the most significant bit can be
seen as a sign bit. Note: a sign bit, not the sign bit.
There is a subtle linguistic difference here. When talking about
the
sign bit, one may thing of it as a single bit that indicates the sign.
For example, 8-bit +1 and −1 could be `0000 0001‘
and `1000 0001‘, respectively. In two’s complement,
however, +1 and −1 are actually `0000 0001‘
and `11111111‘ (the sum of which is
`1,00000000‘ ≡ 0, as
it should be).

1.2 Declaring signed or unsigned

In the end, whether a particular group of bits is signed or unsigned is
a matter of interpretation. For example, the 8-bit group
`1111 1111‘ can be either 255 or −1, depending on
how you want to look at it. You can’t determine the signedness
from just the bits themselves.

Also, when you’ve decided you’re going to use a signed interpretation,
whether the group forms negative number or not depends on the size of
the group. for example, consider the two bytes `01 FF‘.
As separate bytes, these would form +1 and −1, respectively.
However, if you view them as a single 16-bit integer
(‘short”), it forms 0x01FF, which is a positive number.

In C, you specify signedness when you declare a variable. The general
rule is that an integer is signed unless the keyword
`unsigned‘ is used. The exception to the rule is
`char‘, whose default signedness is platform and
compiler-dependent! Be careful with this particular datatype.

Because they’re shorter and more descriptive, the following typedefs
are often used for variable declarations. Basically, it’s
‘s’ or ‘u’ for signed
or unsigned, respectively, followed by the size of the type in bits.
Unsigned variants are also sometimes indicated by
‘u”+typename.

In assembly, you can’t declare the signedness of variables, because
there’s no such thing as variables. There’s only labels and how you
use those labels determines what the related data are. Technically,
there is only one datatype: the 32-bit word, corresponding to C’s
int or long. The other datatypes are essentially emulated, or
defined by how which memory instructions you use:
LDRB/LDRSB/STRB for bytes and
LDRH/LDRSH/STRH for halfwords. For most data
operations, signedness is irrelevant and as such mostly ignored.
Only in a few cases does the sign actually matter and as these are
essentially the topic of the rest of the article, we’ll get
to those eventually.

2 Potential problems

The following sections are cases where signedness may become
problematic. I say “may”, because often it just works
out. But that’s just the thing: it can work most of the time and then
things can go horribly wrong all of a sudden. The root of the problem
comes down to one thing: negative numbers; usually, negative numbers
becoming large positive numbers when interpreted as unsigned values.

For example, 32-bit signed −1 = 0xFFFFFFFF = unsigned 4294967295
(= 2³²−1). If nothing else, remember that part.

2.1 Sign extension, casting and shifting

When you go from a small datatype to a larger one, you’re essentially
adding a new set of bits at the top, and these bits have to be
initialized in a meaningful way. The addition of these bits should have
no effect on the value itself. For example, +1 should remain +1 and
−1 should remain −1. What this boils down to for two’s
complement is that the new bits need to be filled with the sign-bit
of the old value. This is called sign extension, because
the top-bit (the sign-bit) is extended into all the higher bits. There
is also zero-extension, which is when the higher bits are
zeroed out. These two forms effectively correspond to signed
and unsigned casting. (2).

Fig 3.

Conversions of this kind actually happen all the time,
without any kind of direct intervention from the programmer. Data
operations are always done in CPU words and any time you use a smaller
datatype, there is the need to sign- or zero-extend.
This also brings forth the question of which type of extension will be
used: sign- or zero-extension. As the following bit of code shows, it
depends on the signedness of the variable you’re converting from.
8-bit variables sc and uc are both initialized
by 0xFF, which is either −1 or 255 (you can use either of those too,
by the way). After that, these are used to initialize signed or unsigned
words.

As you can see from the output, the value in the words correspond
to the signedness of the bytes, not the words. Also note that printing
sc (the signed byte) gives 0xFFFFFFFF and not the 0xFF you
initialized it with, and which are in fact its actual contents since
0xFFFFFFFF is too large to fit into a byte. However, when using it with
anything, it’s automatically extended to word-size. This becomes great
fun when you later compare it to 0xFF again.

Sign- and zero-extension also play a role in right-shifts. When using
shifts for arithmetic (shift-right is short-hand for a division by
power of two), you want the sign preserved. For example, when dividing
−16 = 0xFFFF:FFF0 by 16 (shift-right by 4), you want the result
to be −1 (=0xFFFF:FFFF), and not 268435455 (=0x0FFF:FFFF).
The right-shift that preserves the sign is the
arithmetic right-shift, and is used for signed numbers.
For unsigned numbers, or if the variable is considered a set of bits
instead of a single number, a logical right-shift is
appropriate, since that uses zero-extension.

In assembly, arithmetic and logical right-shift are called
ASR and LSR, respectively. In Java and other
languages where the keyword unsigned does not exist
the difference is indicated by >> (sign-extend)
and >>> (zero-extend). In C, however,
both types use the same symbol: >>. As such,
you cannot tell which type of extension is used from just the
expression; you’d have to look at the signedness of the operands
(including temporaries) to see if it’s a logical or arithmetic
right-shift.

Table 2: Right-shifts for different languages. Language Signed Unsigned ARM asm asr lsr C >> >> Java(script) >> >>>

Fig 4.

This ambivalence of shift symbols in C can be a major source of pain in
fixed-point calculations. Since unsigned has precedence over signed, if
you have an unsigned variable at any point of the calculation,
all subsequent calculations are unsigned too and you can kiss negative
numbers goodbye. If everything starts going wrong as soon as you move in
another direction or if rotations aren’t calculated properly, this will
be the cause.

The code below illustrates the problem in a very common situation. You
have a position p, and a directional vector for movement,
u. Since you want sub-pixel control of these, you use
fixed-point notation for both (I’m assuming non-FPU system
here). The u vector is a unit vector
(say, cos(α), sin(α)); to get to the full velocity vector,
we have to multiply u by some speed. The procedure comes down to
something like this:

pnew = pold + speed·u

In the example, I’m only considering the x-component for
convenience. Now, because position and direction can have negative
components, those would be signed. The speed, however, is a length
and therefore always positive, so it makes sense to make it unsigned,
right? Well, yes and no. As you can see from the result, mostly no.

With speed = +1 and u_x = −1, the
end result should be +1*−1 = −1, which would be 0xFFFFFF00
in Q8 fixed-point notation. However, it isn’t, thanks to the
unsignedness of speed, which makes subsequent arithmetic
unsigned so the right-shift does not sign-extend. So instead of the
small step you intended, you get a giant leap into no man’s land.

This mistake is depressingly easy to make, even for those who generally
think about which datatype to use. Especially those people, as
they’re prone to optimize prematurely and automatically pick unsigned
for a variable that will never be negative. The danger is that
unsigned arithmetic has precedence, which can screw up at later
right-shifts.

Bottom line: variables used in fixed-point calculations should be
signed. Always.

2.2 Division

This isn’t really a signed-vs-unsigned item per se, but integer
division behaves in a peculiar way for negative numbers. It becomes
one, however when you throw right-shift in the fray, which doesn’t
quite work like a division equivalent anymore for negative numbers.
To discriminate between integer and normal division, I will use
‘\ ’ for integer division in this section. Note also the
modulo operation is intimately tied to division, so this section applies
to that as well.

What integer division comes down to is taking a normal division and throwing
away the remaining fraction. For example, 7 / 4 = 1¾. The
integer division is just 1. This is also true for negative numbers:
−7 / 4 = −1¾, so 7 \ 4 = −1. In short,
integer division rounds towards zero. With bit-shifting, however, you get
something slightly different. Theoretically, x>>n is
equivalent to x \ 2ⁿ. For positive numbers, this is
true: 7>>2 in binary is
00000111>>2 = 00000001.
But with −7>>2 you get
11111001>>2 = 11111110 = −2.
Division-by-right-shift always rounds to negative infinity.

The upshot of this difference is that for negative numbers, the results
of x \ 2ⁿ and x>>n will be out
of sync, as Table 3 illustrates. They still give
identical results for positive numbers though.

There are some other consequences besides the obvious difference in
results. First, there’s how compilers deal with it. Compilers are very
well aware that a bit-shift is faster than division and one of the
optimizations they perform is replacing divisions by shifts
where appropriate(3). For
unsigned numerals the division will be replaced by a single shift.
However, for signed variables some extra instructions have to added to
correct the difference in rounding.

Second, note that the standard integer division does not give an equal
distribution of results: there are more results in the zero-bin.
Shift-division spreads the results around evenly. In some cases, you
will want to use the shift version for that reason. One clear example
of this would be tiling: using the ‘proper’ integer
division would give you odd-looking results.

2.3 Comparisons

The last area where signedness can be a factor is comparisons. The
next bit of code is from my implementation of a filled circle renderer
with boundary clipping. The circle is centered on
(x₀, y₀). Variables
x and y are local variables that
keep track of where we are on the circle, because these
can be negative, they must be signed. Variables dstW
and dstH are the destination image’s width and
height. Since width and height are unsigned by definition,
it’d make sense to make these unsigned, right? Right?

Well, apparently not. When I tested this, right and bottom edge
clipping went fine, but when the circle went over the top or
left edge, it disappeared completely.

The problem lies with the line in bold, which does the trivial rejection
test. Variables left and right are the left and
right-most edges of the scanline of the circle. If this is completely
to the left of the screen (right < 0) or to
the right of the screen (left ≥ dstW)
then there’s nothing to do.

Technically, the tests on that line are correct, so the code
should work.
The reason it doesn’t actually occurs a few lines earlier: the
definition of dstW as an unsigned variable. Because of
this, the second condition is an unsigned comparison. Now think of
what happens when left moves over the left of the
screen. left becomes becomes a (small) negative number,
which is converted to postive number for the comparison.
A large positive number for that matter – one that’s
quite a bit larger than the width of the image and as a result
the routine thinks the circle is out of bounds.

So again, a routine went all wonky because I assumed that, since
a width is always positive, using an unsigned variable would be
a good idea.

The worst part of this particular bit, however, is that I should have
known this. The compiler actually issues a warning for this type of
thing:

warning: comparison between signed and unsigned integer expressions

Or at least it would have if I hadn’t disabled the warning
because the message was cropping up everywhere in my normal and sign-safe
for-loops. Let this be a lesson: disable warnings at your own risk
and for Offler’s sake do not ignore them.

2.4 Well, duh

The problems covered above are the subtle ones, where you have to be
aware of some of the details that go into the C language itself. There
are also a few issues where the programmer really should have known
they were going to be a problem from the start.

The first example is, again, one that can occur when optimizing
prematurely. You may have heard that loops work better when you count
down instead of count up, because in machine code a subtraction is an
automatic comparison to zero. So, a clever programmer may turn this:

into this:

There are two problems with this code. First, the change probably will
not matter with modern compilers because they are aware of the
equivalence and can do this conversion themselves(4), so there’s nothing to gain from this.

The real problem, however, is the terminating condition:
`i>=0‘. Since i is unsigned, it can
never be negative, and therefore the condition is always true.

The second example involves bitfields. As it happens, bitfields can be
signed or unsigned as well. For the most part, handling this is like
handling normal signedness, but there is one situation where you have
to be careful.

In the code above I’ve created a bif-fielded struct with both
signed and unsigned members. There are two 7-bit fields and two
1-bit fields, and these are initialized to −1 and +1,
respectively. The values are then printed.

The 7-bit fields work as you might expect. f.s7 is
−1, as it’s signed, and f.u7 is 127, which is the
7-bit equivalent of −1. The interesting case is for
f.s1. This is initialized to 1, but comes out as
−1, because for a single signed bit the possibilities
are 0 and −1, and not 0 and +1! Without this knowledge,
a later test like `f.s1==1‘ might give unexpected results.

3 Summary

So, summarizing:

• Unsigned variables only represents positive numbers; signed ones
  can have positive or negative values.
  Negative numbers are usually represented via two’s complement,
  which is based on the cyclical nature of counters when you have
  a limited number of digits.
• In C, integers are signed unless specified otherwise, except
  for char, whose signedness is compiler dependent.
• Careless use of signed and unsigned types can result in subtle
  runtime bugs with not-so-subtle results. Usually, what happens
  is that a negative number is reinterpreted as a very large
  positive number and everything goes banana-shaped.
• Unsigned has a higher operator precedence than signed. If
  one of the operands is unsigned, the operation will use unsigned
  arithmetic. This can cause problems for divisions, modulos,
  right-shifts and comparisons.
• For negative numbers, division/modulo by 2ⁿ is not
  quite the same as right-shifts/ANDs. Analyse which is best for
  your situation, then act accordingly.
• Ignore compiler warnings at your own peril.
• The place where a bug manifests is not always the place where it
  originates. The declaration of variables matters! Do not forget
  this when debugging or when asking for assistance.

There isn’t really a hard rule on when to use which signedness, but
here are a few guidelines nonetheless.

• If a variable can, in principle, have negative values, make it
  signed. If it represents a physical quantity (position, velocity,
  mass, etc), make it signed.
• A variable that represents logical values (bools, pixels, colors,
  raw data) should probably be unsigned.
• And now the big one: just because a variable will always be
  positive doesn’t mean it should be unsigned. Yes, you may waste
  half the range, but using signed variables is usually safer. If
  you must have the larger range (for the smaller datatypes, for
  example), consider defining the storage variables unsigned, but
  convert them to local signed ints when you’re really going to
  use them.
• If mathematical symbols were gods, the minus sign would be
  Loki. Be extra careful when you encounter them.
  If there are minus signs anywhere in the algorithm, or even
  the potential for negative numbers, everything should be
  done with signed numbers.