For Windows CE users: on Windows CE, GDI does *not* garbage-collect objects. If an object is referenced when you try to delete it, DeleteObject returns FALSE. When the DC is deleted, the object is leaked.
I'm sure I recall reading that Windows NT-family reference-counts the objects. It's possible that this was changed some time after Windows XP to some form of garbage collection.
@Joshua: My reading of that statement is that 'sometimes' refers to it lying about having deleted the object, not about subsequently cleaning up. But I could be wrong.