iOS Different Font Sizes within Single Size Class for Different Devices

In iOS 8, we can design a different UI layout for each size class. The issue I'm facing is, I've designed a layout for Compact Width and Regular Height (size class for all iPhones in portrait) but i want to keep font size of labels smaller for 3.5 and 4 inch devices (iPhone 4 and 5), then relatively bigger for 4.7 inch (iPhone 6) and more bigger for 5.5 inch (iPhone 6 Plus) devices. I've searched but unable to find a solution to set different font size for different devices within same size class.

Edit: I don't recommend this anymore. This approach doesn't scale well when new devices come out. Use a combination of dynamic font sizes and size classes-specific fonts.

Say a new iPhone model comes out, if you are using Auto Layout and Size Classes you don't have to fix all the constraints manually to make your app compatible with this newer device. However, you can still set the font size of the UILabel using the following code:

if UIScreen.mainScreen().bounds.size.height == 480 {
    // iPhone 4
    label.font = label.font.fontWithSize(20)     
} else if UIScreen.mainScreen().bounds.size.height == 568 {
    // IPhone 5
    label.font = label.font.fontWithSize(20)
} else if UIScreen.mainScreen().bounds.size.width == 375 {
    // iPhone 6
    label.font = label.font.fontWithSize(20)
} else if UIScreen.mainScreen().bounds.size.width == 414 {
    // iPhone 6+
    label.font = label.font.fontWithSize(20)
} else if UIScreen.mainScreen().bounds.size.width == 768 {
    // iPad
    label.font = label.font.fontWithSize(20)
}