My pcie device has a bug that it cant dma to addresses below 4G, What mask should i use with dma_set_mask to tell this to the kernel?
No. pci_set_dma_mask() does exactly the opposite of what you want: it tells the kernel not to use higher addresses.
The easiest thing to do will probably be to reserve the high memory at boot time and use only that memory. Have a look at linux/include/linux/dma-contiguous.h for inspiration. In particular dma_declare_contiguous().
If allocating memory early isn't possible, things will get more tricky. In that case, you'll have to read through what the 'normal' PCI allocation code does, and roll your own.
I found the slides of this presentation from last year's Embedded Linux Conference insightful. They certainly contain lots of pointers to useful headers and source code to read through. (==> EDIT : The link is dead and you can find the slide here )