You are right,
using support displace brim / skirt to the outer-site area.
But when using support you don't need brim to enforce sticking the smal object to the print-bed. So to demonstrate with a sphere. Using raft may help
Normally I use brim with a distance of 0.05 mm.
H.