Sigh. Gaddammit >_<

This is pretty much where I wanted to go with this game, so it’s now version 1.0. I may add some other things later, but for now I guess it’s done.

Changes

• new: FAT saving enabled :D. It’s intended for
  argv-capable
  cards, though. You can still save without one of those, but you’ll have to
  put in a little work.
• new: Reset/Back buttons for high-score screen.
• fix: Bug in shuffle code. Fixes the occasional hang at start of a new game.
• tweak: I’ve altered the decay constant slightly; it should be closer to
  the original tatset now. The upshot is that scores will be a little higher.
• tweak: Slightly bigger buttons.

Links

• setds project page

A while ago, I wrote
an article about NDS caching
, how it can interfere with DMA transfers and what you can do about them.
A little later I got a
pingback from ant512,
who had tried the “safe” DMA routines I made and found they
weren’t nearly as safe as I’d hoped. I’m still not sure what the actual
problem was, but this incident did make me think about one possible
reason, namely the one that will be discussed in this post: problematic
cache invalidation.

1 Test base

But first things first. Let’s start with some simple test code, see below.
We have a simple struct definition, two arrays using this struct, and
some default data for both arrays that we’ll use later.

And now we’re going to do some simple things with these arrays that
we always do: some reads, some writes, and a struct copy. And for the
copying, I’m going to use DMA, because DMA-transfers are fast,
amirite(1)? The specific actions I will do are the following:

Initialization

• Zero out g_src and g_dst.
• Initialize the arrays with some data, in this case from
  c_fooIn.
• Cache-Flush both arrays to ensure they’re uncached.

Testing

• Modify element in g_src, namely g_src[0].
• Modify an element in g_dst, namely g_dst[3].
• DMA-copy g_src[0] to g_dst[3].

In other words, this:

Note that there is nothing spectacularly interesting going on here.
There’s just your average struct definition, run of the mill array
definitions, and boring old accesses without even any pointer magic
that might hint at something tricky going on. Yes, alignment is forced,
but that just makes the test more reliable. Also, the fact that I’m
incrementing Foo.id rather than just reading from it is
only because ARM9 cache is read-allocate, and I need to have these
things end up in cache. The main point is that the actions in
test_dmaCopy() are very ordinary.

2 Results

It should be obvious what the outcome of the test should be. However,
when you run the test (on hardware! not emulator), the result seems to
be something different.

Notice that the changed values of g_src[0] never
end up in g_dst[0]. Not only that, not even the
original values g_src[0] have been copied.
It’s as if the transfer never happened at all.

The reason for this was covered in detail in the original article.
Basically, it’s because cache is invisible to DMA. Once a part of
memory is cached, the CPU only looks to the contents of the cache and
not the actual addresses, meaning that DMA not only reads out-of-date
(stale) source data, but also puts it where the CPU won’t look.
Two actions allow you to remedy this. The first is the
cache flush, which write the cache-lines back to RAM and
frees the cache-line. Then there’s cache invalidate, which
just frees the cache-line. Note that in both cases, the cache is
dissociated from memory.

With this information, it should be obvious what to do. When DMA-ing
from RAM, you need to flush the cache before the transfer to update
the source’s memory. When DMA-ing to RAM, you need to invalidate
after the transfer because now it’s actually the cache’s data that’s
stale.
When you think about it a little this makes perfect sense, and it’s easy
enough to implement:

Unfortunately, this doesn’t work right either. And if you think about
it a lot instead of merely a little, you’ll see why. Maybe showing the
results will make you see what I mean. The transfer seems to work now,
but the earlier changes to g_dst[3] have been erased. How
come?

The problem is that a cache-invalidate invalidates entire
cache-lines, not just the range you supply. If the start or
end of the data you want invalidate does not align to a cache-line,
the adjacent data contained in that line is also thrown away. I hope
you can see that this is bad.

This is exactly what’s happening here. The ARM9′s cache-lines are 32
bytes in size. Because of the alignment I gave the arrays, elements
0 through 3 lie on the same cache-line. The changes to
g_dst[3] occur in cache (but only because I read from it
through +=). The invalidate of g_dst[0]
also invalidates g_dst[3], which throws out the
perfectly legit data and you’re left in a bummed state. And again,
I’ve done nothing spectacularly interesting here; all I did was
modify something and then invalidated data that just happened to be
adjacent to it. But that was enough. Very, very bad.

Just to be sure, this is not due to a bad implementation of
DC_InvalidateRange(). The function does exactly what it’s
supposed to do. The problem is inherent in the hardware. If your
data does not align correctly to cache-lines, an invalidate will apply
to the adjacent data as well. If you do not want that to happen, do
not invalidate.

3 Solutions

So what to do? Well, there is one thing, but I’m not sure how foolproof
this is, but instead of invalidating the destination afterwards, you
can also flush it before the transfer. This frees up the cache-lines
without loss of data, and then it should be safe to DMA-copy to it.

Alternatively, you can also disable the underlying reason behind the
problem with invalidation: the write-buffer. The ARM9 cache allows
two modes for writing: write-through, which also updates
the memory related to the cache-line; and write-back, which
doesn’t. Obviously, the write-back is faster, so that’s how libnds
sets things up. I know that putting the cache in write-through mode
fixes this problem, because in libnds 1.4.0 the write-buffer had been
accidentally disabled and my test cases didn’t fail. This is probably
not the route you want to take, though.

4 Conclusions

So what have we learned?

• Cache – DMA interactions suck and can cause really subtle bugs.
  Ones that will only show up on hardware too.
• Cache-flushes and invalidates cover the cache-lines of the requested
  ranges, which exceed the range you actually wanted.
• To safely DMA from cachable memory, flush the source range first.
• Contrary to what I wrote earlier, to DMA to cachable memory,
  do not cache-invalidate – at least not when
  the range is not properly aligned to cache-lines. Instead, flush
  the destination range before the transfer (at which time
  invalidation should be unnecessary). That said, invalidate should
  still be safe if the write-buffer is disabled.

Link to test code.

Okay, so it’s only a card game; but a game nonetheless.

The game in question is an NDS implementation of
SET. Set is a card-matching game
with 81 cards (see below). The figures on the cards have four
properties and 3 possibilities for each property. The key is to find
three cards for which the values of each property are either all
equal or all different. Looking at the color property,
for example, a “Red Red Red” combination could (yes “could”; there
are still three other properties to consider) form a set.
“Red Green Blue” would also work, but “Red Green Green” would not.

Further details can be found in the readme and the game itself.

The game is mostly finished. There may be some tweaking to do
here and there, but right now I don’t want to get bogged down in a
massive fine-tuning-fest – especially since I’m not sure what
parts need fine-tuning … and because there’s
other stuff I really should get
back to.

That said, all important aspects work … with one
exception: hiscore saving. Yes, that. I’ve seen the
multitude of threads on the subject but sofar I’m unsure of what would
work on both hardware and emulator, so I’m leaving it as is for now.
If anyone has a tidy hw+emu solution, please do tell.

Links

• binary: setds-bin.zip (146k)
• source: setds-src.zip (354k)
• readme: setds-readme.txt

Oh, and merry Christmas everybody.

The short story is that officially new should throw an
exception when it can’t allocate new memory. Exceptions come with about
60 kb worth of baggage. Yes, this is more or less the same stuff that
goes into vector and string.

The long story, including a detailed look at a minimal binary,
a binary that uses new and a solution to the exception overhead (in this particular case anyway) can be read below the fold.

• 1 Minimal project
• 2 Standard C++ new/delete
• 3 Custom new/delete
• 4 Other considerations and conclusions.

1 Minimal project

The following is essentially an empty project. It should represent
the smallest binary you can get with the current DKA (r26) and
libnds (1.3.7). This is the primary reference case.

This actually already leads to a binary of 53.5 kb. To analyze what
goes on in there, we can look at the map file. Not the mapfile
generated by the linker, mind you, but by the arm-eabi-nm tool,
whose generated files are considerably easier to read. To use this tool,
add the following line to $(BUILD) rule in the makefile,
so that it looks like below. If you want to know what the flags mean,
please RTFM.

And this is the resulting mapfile, in full.

Now, I expect you can’t really tell much from this, so here’s a summary.

The 0100:xxxx and 0B00:xxxx ranges belong to
ITCM and DTCM, so those are irrelevant when looking at main RAM size.
The libc, impure and misc bookkeeping sections are stuff related to the
C library and C overhead, accounting for about 1.5 kb. The boot code,
crt0.S also covers close to 1.0 kb. As expected, the code for
main.c –the actual project– is more or less
nothing.

The rest, about 7 kb, is libnds. Now, you may say that this is quite a bit
of overhead, but it really isn’t. Pretty much all of it relates to
interrupts and the fifo system, which takes care of ARM7-ARM9
communication. You need to have these parts. Okay, you could try
to roll your own to shrink this down to the bare essentials, but in all
likelihood that’s more trouble than it’s worth.

The observant of you should have noticed something: we’re only at 9.5 kb,
but the file size is 53.5 kb. So what the hell happened to the other 44 kb?
Well, I don’t know, to be honest. It doesn’t appear in MWRAM to be sure.
It’s probably the stuff ndstool adds. My guess it that that’s
where the ARM7 binary goes, along with the icon, titles and possibly
DLDI interfaces, but I really can’t say right now.

2 Standard C++ new/delete

And now, let’s look at what happens when you invoke new.

Just this small thing increases the file size to 117 kb! And remember,
that’s not merely a doubling of the size, as 44 kb of the binary is not
put in memory. The memory load has gone from about 10 kb to over 70 kb.
What causes this increase? Well, let’s see:

Well, I did say this was going to be the long story, didn’t I?
Everything that was in the base project is in here as well. The
additional parts can summarized as follows.

There are three main areas to discern:

• d_*() routines, presumably for debug printing.
  (size: 12k)
• Stdio formatting and related. This includes file handling, device
  handling and many forms of printf, which brings a
  whole lot of bagage (some allocation, format parsing and
  math/floating point routines). There’s also some abort and
  signalling routines. (size: 28k).
• Exception handling. Not just routines for handling them, but also
  the typeinfo stuff required, the output strings and the output
  string pointers. (size: 18k)

These roughly 60k of stuff is the overhead of exceptions –
any potential exception. In this case, it’s because
new requires a bad_alloc exception when it’s
unable to allocate more.

The problem is that exceptions have
many dependencies: to do exception handling, you keep track and unwind
the stack. You also need to be able to tell the type of exception
thrown, which requires RTTI. And then you say which exception was
thrown, so you need error messages, and a list of pointers to
those messages, and a way to format and write those messages,
hence the d_*() routines and all the stdio stuff.

3 Custom new/delete

There is a way around this, though: redefine new and
related functions. Technically speaking, this is a bad idea
if you don’t know what you’re doing, but it can be done. Note that
you would need overload four operators: new,
delete and their array counterparts.

This way, you just incur the cost of malloc() and
free(), which are only about 3k. But again, this is going
against the standard and you’ll really have to ask yourself if the
(at best) 2% of main RAM you save with this is really worth it.

More on this can be read at

http://brewforums.qualcomm.com/showthread.php?t=2033.

4 Other considerations and conclusions.

The binary size is not the same as the main RAM footprint.
About 44 kb other stuff.

The overhead of the standard new is 60 kb, which is all
due to exceptions. You cannot remove it by
using the compiler options -fno-exceptions and
-fno-rtti, because that only affects your own code, not the
standard libraries. You can remove this overhead by using overloading
new and related functions, but you have to be really
careful with this.

I’ve also done a little bit of testing with vector, and
it seems that vector‘s overhead also comes from
new and can be removed the same way. However, other parts
of vector (and STL) may use other exceptions, so it’s
quite possible it won’t work in all cases.

Note that roughly 28 kb of the exception overhead is actually
stdio related – specifically formatted printing:
*printf. If you’re using printf anyway, the
effective overhead of exceptions is reduced considerably.

Finally, remember that the exception overhead amounts to roughly 2% of
main RAM at most. In most homebrew cases it won’t matter that much.
When it does start to affect your app, you will likely have other parts
that are easier and safer to optimize out.

Test project + notes.

• 1 The ARM 9 core
• 2 Cache example
• 3 Cache vs DMA solution
• 4 Test cases
• 5 Conclusions

DMA, or Direct Memory Access, is a hardware method for transferring
data. As it’s hardware-driven, it’s pretty damn fast(1). As such,
it’s pretty much the standard method for copying on the NDS.
Unfortunately, as many people have noticed, it doesn’t always work.

There are two principle reasons for this: cache and TCM. These are
two memory regions of the ARM9 that DMA is unaware of, which can lead
to incorrect transfers. In this post, I’ll discuss the cache, TCM and
their interactions (or lack thereof) with DMA.

The majority of the post is actually about cache. Cache basically
determines the speed of your app, so it’s worth looking into in more
detail. Why it and DMA don’t like each other much will become clear
along the way. I’ll also present a number of test cases that show
the conflicting areas, and some functions to deal with these problems.

1 The ARM 9 core

Fig 1.

The first thing to know is that the DMA trouble only relates to the
ARM9 processor of the NDS. Work with the ARM7 should be fine. The most
relevant items of the ARM9 are illustrated by Fig 1.
The processor consists of the actual logic unit, and caches and two
Tightly Coupled Memory (TCM) units. There are two caches
and TCMs, one for data and one for instructions. The point here is
that (as far as I know), these areas are on the chip, and
as such accessible only by the CPU itself. CPU-only, as in not the
DMA controller.

1.1 ITCM and DTCM

The Instruction and Data TCM areas (ITCM and DTCM) are basically
fast-RAM areas. Technically the addresses of these sections are
arbitrary, but set to the 0100:0000 and 0B00:0000 ranges,
respectively, in libnds. Exactly which addresses they use isn’t
important though, since that’s all taken care of by the linker anyway.
What is important is that the stack (where local variables
and function arguments go(2)) is also put in DTCM. This means that you can’t
use DMA with local arrays. It also means that you can’t use a local
variable as a source for a DMA-fill. This is why the NDS ARM9 has
special DMA registers for these, called
REG_DMAnFILL.

The DMA-fill routines in libnds correctly use REG_DMAnFILL
so fortunately you don’t have to worry about that. However, copying from
local arrays is still impossible, and there’s just no way around that
using just DMA.

1.2 Cache

The cache for the ARM9 is a little more complicated. Or perhaps
“obscured” is a better term here. The TCMs have
addresses, so you can toy with them yourself. The cache, however,
is completely hidden from the view of the user.
Before going into detail about how the cache works and why DMA
and cache hate each other, let’s look at why cache is useful,
especially in light of you not being able to use it directly.

Where there is memory, there are waitstates. Generally, a CPU can handle
data faster than the RAM can supply it, so the CPU will have to wait
until it can continue. The slowdown can easily be a factor 100 on
PCs, or even millions if you include disk memory. Fortunately, it’s
only about ten for the NDS, I think, but that’s still quite a bit.

Cache is one method of getting around memory waitstates. Instead of
having to go to RAM all the time for something, you store recently used
data in an area that the CPU can have faster access to. Then next
time it needs that data, it can retrieve it from there instead of
going to RAM again. For good measure, the area around the data is also
cached, because that might be accessed soon as well. Since
closely-related data is often stored closely together as well, the
caching process can significantly increase the overall speed of an
application.

1.3 The NDS ARM9 cache

GBATEK gives us the
following information about the cache that the NDS has:

Data Cache 4KB, Instruction Cache 8KB
4-way set associative method
Cache line 8 words (32 bytes)
Read-allocate method (ie. writes are not allocating cache lines)
Round-robin and Pseudo-random replacement algorithms selectable
Cache Lockdown, Instruction Prefetch, Data Preload
Data write-through and write-back modes selectable

Which to most people will probably mean absolutely nothing. Now, I’m
not exactly an expert in all things cache, but I’ll try to explain what
it all means.

First, as noted earlier, there are actually two caches: one for
data and one for instructions. Having a separate instruction cache is nice
because then you can be sure that a function that processes a lot of
data won’t push the that function out of the cache. Instruction cache
also means that loops in code will be in cache except for perhaps the
first iteration. Effectively, all the code that really matters (i.e.,
inner loops that do most of the work) will always be in fast memory
automatically.

It is common that cache works in groups of bytes instead of individual
bytes. These groups are the cache lines. A cache line maps
onto a RAM chunk of the same size, and if anything within a chunk
is to be put in cache, the whole line will be filled. The NDS cache
lines are 32 bytes long.

This is probably a good time to introduce two important terms:
cache hit and cache miss. A cache hit is when the data
you’re looking for is already in cache and so access is fast. A
cache miss is when it’s not in cache. This means two things.
First, the access will be slow thanks to the memory waitstates.
Second, if this triggers a cache-line fill, you’ll have to wait for
the entire cache line to be read. While this block read will
be faster than if you were reading the block without cache, it’ll
still take longer than getting just the byte or so you were looking
for. Moral of the story: cache hit good, cache miss bad.

Cache hits and misses add a consequence to how your data is stored.
If data is tightly packed and sequential (think structs/arrays), you’re
more likely to have cache hits and work will be fast. If the data is
all over the place (linked lists for example), the chance of cache
misses increases dramatically.

That is how cached or non-cached data operates. What’s also important
is when data is put in cache in the first place – when cache
allocation occurs. There are two types here: read-allocate
or write-allocate, These terms refer to whether a
cache-line will be tied to a memory block when it’s from, or when
it’s written to, respectively. As you can see from the GBATEK data,
the NDS cache is read-allocate. A memory-write will not require a new
cache line.

Now, Suppose a block is in cache and something is written to that block.
This write will update the data in cache, but what about the RAM it’s
tied to? The process dealing with this is called the write policy,
and two options exist. There’s write-through, which means that
both cache and RAM are written to. In write-back mode, only the
cached data is changed; RAM is not updated! This is the main
cause of trouble with DMA. Apparently, the write policy is selectable,
but it’ll usually be write-back.

Lastly, there’s the replacement policy, which stipulates
how cache lines relate to RAM, and when to kick data out ot cache.
Unfortunately I don’t know much about this part, but it’s of lesser
importance anyway. The rest of the terms do not affect the potential
cache-DMA conflict either. For more details, visit the wikipedia page
on CPU_cache.

2 Cache example

At this point I think it’s useful to give an example of how it works
in practice. For this, I will use a fictional CPU that uses a cache with
following properties.

• 2 cache lines, 4 bytes each.
• Read-allocate and write-back.
• Direct mapped cache: RAM-block n goes into cache-line
  n%2. In other words, even blocks go to line 0, odd blocks
  to line 1.

The next set of pictures illustrate what happens when you do a
number of reads and writes. Fig 2 shows the basic
system in the initial state. There is the CPU on the left with the
core and two cache lines. I’ve also included a register called x
here for convenience. On the right there is 16 bytes of RAM,
distributed over four 4-byte blocks (the equivalents of the cache
lines). RAM is already initialized; cache is still empty. In the
figures, green is used to indicate reads from RAM (loads) and purple
for writes (stores).

Fig 2.	Fig 3.
Fig 4.	Fig 5.
Fig 6.	Fig 7.
Fig 8.	Fig 9.

0. Initial state.
1. RAM[0]= 'R'. A write to RAM does not trigger cache
   allocation, so it goes straight to RAM. Slowly.
2. x= RAM[1]. The read from RAM[1] causes a cache
   allocation RAM[1] is part of block 0 (even), so that goes into
   cache line 0. Line 0 and block 0 are identical.
   Cache miss + allocation; very slow.
3. RAM[2]= RAM[3]= 'S'. Two writes; this is where it
   gets tricky. These addresses are in cache (block 0), and I
   said this cache was in write-back mode. This means that the writes
   go to the cache, but NOT the actual RAM. So now the data
   in cache is different that the equivalent block in RAM: the RAM’s
   gone stale. This is a cache hit; a fast write.
4. x= RAM[3]. Again, RAM[3] is cached, so data is taken
   from cache instead of from RAM. x is now 'S',
   as was expected from the last statement. The fact the real RAM[3]
   is different doesn’t matter, because the CPU doesn’t look there
   anyway. If something other than the CPU (like, say, DMA)
   reads from RAM[3], though, chaos ensues. Cache hit, fast read.
5. x= RAM[4]. RAM[4] is in block 1, which wasn’t cached
   yet, so a new line is allocated. Block 1 goes into line 1, because
   it’s an odd-numbered block. Very slow operation.
6. RAM[4]= RAM[5]= 'T'. Much like before, The writes go
   into cache rather than the real RAM. Another cache hit.
7. x= RAM[8]. This is also an interesting case. Two
   things happen here. RAM[8] belongs to block 2 (even), which hadn’t
   been cached yet. It’s supposed to go into line 0, but that’s
   already filled. The new data will replace the old data. Cache line is
   tied to block 0, so addresses 0 through 3 will be filled with the
   data from line 0; this block is now up to date again. After that,
   line 0 receives the data from block 2. Cache write-out + new
   allocation; this should be awful.

This should cover all important cases: reads/writes to non-cached
addresses, to cached addresses and a little bit about allocation
and replacements. At some points, cache and RAM start to disagree. This
wouldn’t be a problem if RAM was only accessed by the CPU, but
unfortunately it isn’t.

3 Cache vs DMA solution

Fig 5 and Fig 8 illustrate the main
problem. The data in RAM is out of date and when DMA tries to read
it, it actually uses the wrong data. the reverse is also possible.
DMA could write to RAM that had been cached; in this case it’s actually
the cache that’s out of date.

The solution to this is to align cache and RAM manually. The two
actions involved are called flushing and invalidating. A
cache flush dumps the contents of cache back into RAM. Now
that cache and RAM contain the same data again, it’s safe to DMA-read
from. An invalidate tells the CPU to simply delete cache
lines, because its assumptions regarding the contents of the original
RAM have become invalid. The next CPU-read would come from RAM again.
This is what you need after DMA writes to RAM.

Fig 10 and Fig 11 pick up from
case 6 (Fig 8). They show what happens when you
flush or invalidate a cache line. You actually supply a RAM block
number because the cache lines themselves are completely hidden from
view.

7. Flush block 0. In this case, you want to synchronize RAM
   block 0 to the cache. Since block 0 is indeed in cache and using
   line 0. Therefore, the contents of line 0 are written back to block 0.
8. Invalidate block 1. Suppose that previously, some contents of
   block 1 had been written to without cache’s knowledge, so that cache
   is out of date. The invalidate throws away the line related to
   block 1, making RAM the primary source for the block again.

Fig 10.

Fig 11.

3.1 libnds cache functions

libnds contains functions to
flush or invalidate.
They can either affect the whole cache, or just certain address ranges.
I am unsure of the timings of these functions, but I expect there will
be a cost. Invalidating itself could be fast, but it’d make all
subsequent reads cache misses. A flush would require a large amount of
of writes to memory. To keep these costs down, use the ranged versions
as much as possible.

3.2 Some safe DMA functions

To guard against potential DMA failures, it’s useful to have a few
functions that take care of those themselves. dmaCopySafe()
and dmaFillSafe() check if DMA can reach the source and
destination regions and will return false if not. They
also check what chunk-size is appropriate by looking at the source,
destination and size. Odd alignments fail completely; word-alignment
and sizes use 32-bit transfers and the rest uses 16-bit transfers.
They also flush and invalidate where appropriate.

Note that for a completely safe version, you’d need much more checking.
For example, each region has its own size that would have to be looked
at, and some sections are read-only like ROM. Checking for all
possibilities, however would just make the function too unwieldy and
have therefore been omitted.

4 Test cases

Fig 12.

All this talk about what to do and when is nice and all, but it’s
always best to run a few tests to see if everything happens like you
expected. Fig 12 illustrates how the tests operate.
There are two source bitmaps of the letters ‘A’ and
‘B’. In each test case, one of these is copied into
a secondary buffer by either a CPU- or DMA-based copy. This
second buffer is blitted to VRAM in two different places via
a CPU- or DMA-based blit. At various points, a flush or invalidate
may be inserted to see the effects.

In terms of code, ever case is split into two parts. First, there’s a
setup that initializes each case. This clears the console and prints
some description for the case and erases the RAM buffer and the VRAM
rectangles. The second part alternatively copies the letters into the
buffer, does cache operations, and blits. Since the first part is
boring, only the case-specific part will be given here.

4.1
Direct blit: ‘A’ → VRAM

Result: both correct.
Explanation: the data in the source buffer never changes,
so this should always be okay.

4.2
Indirect Blit I: ‘B’ → buffer → VRAM

Result: both correct.
Explanation: buffer used for the first time, so no cache
incoherency possible. Yet.

4.3
Indirect Blit II: ‘A’ → buffer → VRAM

Result: CPU okay, DMA blit corrupted.
Explanation: The buffered data is in cache, so the
memcpy() to the buffer goes to cache as well. Meanwhile,
the actual buffer (in RAM) still holds (parts of) ‘B’,
which is where DMA gets its data from. The result is a mix of
‘A’ and ‘B’ for dmaBlit(). It’s
a mix because apparently some cache lines have already been flushed
out by the replacement policy.

4.4
Indirect Blit + flush: ‘B’ → buffer, flush → VRAM

Result: both correct.
Explanation: again, the memcpy() will result in
corrupt data in RAM, but this time we force the up-to-date cache lines
back to RAM to get cache and RAM in synch again. At this point, both
the CPU and DMA-based blits will use the correct data.

4.5
Indirect Blit + invalidate: ‘A’ → buffer, invalidate → VRAM

Result: both screwed in the same way.
Explanation: as said in the note, you need to do the right
operation at the right time. This is an example of what happens if you
don’t. A cache invalidate simply erases cache lines. The data in RAM is
now considered valid. However, after the memcpy(), RAM
isn’t valid, as one can see from case 3. This is why now both
blits fail.
Nice
job breaking it, hero.

4.6
Indirect Blit III: ‘A’ (dma)→ buffer → VRAM

Result: CPU blit fail.
Explanation: in this case, the source→buffer transfer is done
via DMA rather than the CPU-based memcpy(). This time the
CPU doesn’t know the contents of RAM have been altered. At this point,
the cache will contain mostly the cleared data from the
memset() done in the case set-up; cpuBlit()
will pick up some straggling lines from RAM during blit, resulting in
the image you see here. Naturally, dmaBlit() works
properly.

4.7
Indirect Blit + invalidate II: ‘B’
(dma)→ buffer, invalidate → VRAM

Result: both correct.
Explanation: as case 6, but with an invalidate after the
transfer to the buffer. The invalidate removes the allocated cache
lines, so that the next CPU-reads come from RAM. Unlike case 5,
RAM is the most up-to-date area, so the invalidate works as
it’s supposed to.

4.8
Indirect Blit + invalidate/flush: ‘A’
→ buffer, invalidate/flush → VRAM

Result: both fail.
Explanation: The invalidate frees cache lines (see case 5).
The subsequent flush does nothing, because no lines are associated
with those addresses anymore.

This is an example of what I like to call LOL-type programming.
As in “What are you doing?!?” – “I dunno lol”.
This type of coding generally indicates a very confused mind that
hopes that if you throw enough shit against a wall maybe something
will hold up. He may have heard terms like flush and invalidate
used around DMA and decided to try them randomly. This never works.
If you find code like this, be afraid; very, very afraid.
This will likely not be the only instance of cargo-cult programming
in the code-base and it’d be best to consider the whole thing suspect.

4.9
dmaSafe I: ‘B’ → buffer (safe)→ VRAM

Result: both correct.
Explanation: this uses the dmaCopySafe() function given
previously in the DMA blitter. Since that function checks whether a
flush is appropriate, everything should be fine. And it is.

4.10
dmaSafe II: ‘A’ (safedma)→ buffer → VRAM

Result: both correct.
Explanation: dmaCopySafe() also performs an
invalidate if necessary so, again, it all works.

4.11
Buffer in stack: ‘B’ → local buffer → VRAM

Result: DMA fail.
Explanation: in this case, I’m using a local buffer for the
bitmap instead of a global one. The difference here is that a local
buffer goes onto the stack, which is in DTCM rather than RAM. As
discussed earlier, DTCM is invisible to DMA, so dmaBlit()
doesn’t work. Note that using dmaBlitSafe() wouldn’t
work either, but at least you’d get an return value indicating failure
back instead of nothing at all.

5 Conclusions

• The ARM9 has DTCM and ITCM sections that DMA can’t access. DMA
  transfers to and from there will fail. Because the stack is in
  DTCM, this includes transfers to/from (non-static) local variables.
• The ARM9 has cache that DMA can’t see either. If DMA tries to
  read/write from RAM block that have been cached, the wrong data
  may be transferred.
• The ARM9 uses 32-byte cache lines that are initiated when addresses
  are read from, but not when written to (read-allocate). It also uses a
  write-back policy: cached data is written back to RAM only when the
  cache line is replaced.
• Cache-miss bad; cache-hit good. Stale cache also bad, since it’s the cause
  of incorrect DMA transfers.
• DMA-reads from RAM should be preceded by a cache flush, which writes
  cache lines back to RAM.
• DMA-writes to RAM should be preceded or followed by a cache
  invalidate, which clears cache lines so that the next CPU-read will be
  from RAM again.
• As far as I know, most emulators do not handle DMA-DTCM correctly.
  None emulate cache. If you suddenly find corrupted data after
  copying on hardware but not emulators, look at your DMA calls.
• Making graphics with rounded corners and intricate wide lines take
  forever to get right.

Related test project: arm9vsdma.zip

The sizes in Table 1 are for a bare source file with just the VBlank initialization and swiWaitForVBlank() plus whatever’s necessary to use a particular component. For the IO parts this means a call to consoleDemoInit(); for vectors and strings, it means defining a variable.

Even an empty project is already 15k in size. Almost all of this is FIFO code, which is required for the ARM9 and ARM7 to communicate. Adding consoleDemoInit() and a printf() call adds roughly 71k. Printf has a lot of bagage: you have to have basic IO hooks, character type functions, allocations, decimal and floating point routines and more.

Because printf() uses the usually unnecessary floating point routines for float conversions, it is often suggested to use the integer-only variant iprintf(). In that case, it comes down to 55k. The difference is mostly due to two functions: _vfprintf_r() and _dtoa_r(), for 5.8k and 3.6k, respectively. The rest is made up of dozens of smaller functions. While the difference is relatively large, considering the footprint of the other components, the extra 16k is probably not that big of a deal. For the record, there is no difference in speed between the two. Well, almost: if the format string doesn’t contain formatting parts, printf() is actually considerably faster. Another point of note is that the 55k for iprintf() is actually already added just by using consoleDemoInit().

And now the big one. People have said that C++ iostream was heavy, and indeed it is. 266k! That’s a quite a lot, especially since the benefits of using iostream over stdio is rather slim if not actually negative(1). Don’t use iostream in NDS projects. Don’t even #include <iostream>, as that seems enough to link the whole thing in.

Related to iosteam is fstream. This also is about a quarter MB. I haven’t checked too carefully, but I think the brunt of these parts are shared, so it won’t combine to half a Meg if you use both. Something similar is true for the stdio file routines.

Why are the C++ streams so large? Well, lots of reasons, apparently. One of which is actually its potential for extensibility. Because it doesn’t work via formatting flags, none of those can be excluded like in iprintf()‘s case. Then there’s exceptions, which adds a good deal of code as well. There also seems to be tons of stuff for character traits, numerical traits, money traits (wtf?!?) and iosbase stuff. These items seem small, say 4 to 40 bytes, but when there are over a thousand it adds up. Then there’s all the stuff from STL strings and allocators, type info, more exception stuff, error messages for the exceptions, date/time routines, locale settings and more. I tell you, looking at the mapfile for this is enough to give me a headache. And worst of all, you’ll probably use next to none of it, but it’s all linked in anyway.

Finally, some STL. This is also said to be somewhat big-boned, and yes it isn’t light. Doing anything non-trivial with either a vector or string seems to add about 60k. Fortunately, though, this is mostly the same 60k, so there are not bad effects from using both. Unfortunately, I can’t really tell where it’s all going. About 10k is spent on several d_*() routines like d_print(), which is I assume debug code. Another 10k is exceptions, type info and error messages and 10 more for. But after that it gets a little murky. In any case, even though adding STL strings and vectors includes more that necessary, 60k is a fair price for what these components give you.

Conclusions

The smallest size for an NDS binary is about 14k. While printf() is larger than iprintf(), it’s probably not enough to worry about, so just use printf() until it becomes a pressing matter (and even then you could probably shrink down another part more easily anyway). There is no speed difference between the two. The C++ iostream and fstream components are not worth it. Their added value over printf and FILE routines are small when it comes to basic IO functionality. STL containers do cost a bit, but are probably worth the effort. If you need more than simple text handling or dynamic arrays and lists, I’d say go for it. But that’s just my opinion.

Please note, the tests I did for this were fairly roughly. Your mileage may vary.

Lastly. The nm tool (or arm-eabi-nm for DKA) is my new best friend for executable analysis. Unlike the linker’s mapfile, nm can sort addresses and show symbol sizes, and doesn’t include tons of crap used for glue.

The state of register names for NDS homebrew is a bit of a mess.
First, there are the GBATek names. Since GBATek is considered
the source of GBA/NDS information, it would make sense to
adhere to those names pretty closely. But, of course, that’s not
how actually is in the de facto library for NDS homebrew, libnds.

libnds has two sets of names. This probably is a result of serving
different masters in its early days. One set uses
Mappy‘s nomenclature.
That’s the one without the REG_ in front of it, and
uses things like _CR, and _SR. This is
one you’re most likely to see in the current NDS tutorials.
The second set uses GBATek’s names (mostly) plus a REG_
prefix. If you’ve done GBA programming, these should feel quite
familiar.

For the most part, I don’t really care much for the the Mappy names.
One reason for this is that they don’t match up with the GBATek names,
making it harder to look-up what they do. More importantly, though,
without any sort of prefix that indicates they’re registers, they all
look like the constants you put in the
registers. After all, that’s what all-caps usually means: a constant.
When I see the identifier WIN0_X0, I’m more likely to
think that it’s something to set a window coordinate to, rather
than that it is how to control the window. Having to wonder constantly
whether a particular identifier is a constant or a register is pretty
annoying(1).
But perhaps this is just me.

What’s especially grating about these names is that the elements of
the affine matrix
are transposed. HDY is actually VDX and vice
versa. Shame, actually, because the HDX-VDY naming
scheme does make more sense than old PA-PD names.

The elements XDY and YDX are also transposed
… I think. It’s hard to be sure since this nomenclature is highly
ambiguous anyway. I really can’t tell how the elements are supposed to
be read, much less determine if they are in the correct order.

The REG_ registers do not suffer from those problems, but
they have their own issues. The registers that had a GBA counterpart
are generally okay, but in the case of the NDS-specific ones, some
follow the GBATek names, some almost follow them
(MTX_LOAD_4x4 vs REG_MTX_LOAD4x4, for
example; note the underscores), and some are the Mappy-like names
plus prefix.

Finally, registers of both nomenclatures are grouped by topic and not
necessarily by address. This makes sense from an OOP point of view,
but it does mean they have been sprinkled across multiple files, making
it tricky to see where the things are acrually defined. Some named are
also clumped inside registers_alt.h, which is a sort of
transitional file. This file is not included by nds.h, which
can make it hard to know which names are included or not until you
either search though the whole include hierarchy or get an error
from the compiler. And then have to search the hierarchy to find out
the exact spelling.

To make it a little easier to deal with the fragmentation of the libnds
headers regarding registers, here is a list of all the registers as they
appear in GBATek, the primary libnds set (plus the file where they’re
defined), and the alternate libnds set (registers_alt.h).
I’ve also included links to where you can find the descriptions
in GBATek.

• The bold black names are those corresponding exactly to GBATek.
• The bold red items are problematic in some
  way; usually the datatype is not what is should be, or some registers
  have been swapped somehow. Follow the footnotes or hover over them to
  see my comments. Note that some of these are already fixed in CVS.
• Sometimes there are gaps. In this case, that particular set has
  no name for it.

1 Arm 9 registers

2 Arm 7 registers