The 04 doesnt have a spacer, other than the sleeve in the hub(dont see one in the parts diagram for '09 either). Is the mount itself loose, or is the caliper floating? When you first install, if the pads have been backed off to allow easy re-install, the caliper will float and feel like something is wrong. Pump up your rear brakes and see if it goes away. Otherwise, the only thing that could be missing is the sleeve that goes in the hub on the wheel.
Also the mount has a groove in it, and the swingarm has a Key so to speak that has to be lined up before stabbing the axle through.
'12 HD Wide Glide Metallic Blue Flame
'04 YZF R1 Velocity Stacks, race filter, PC3, SATO Rearsets GP Shift, Emission blockoff plates
'94 KTM 440 MXC: Big Bore 2 Stroke