How should I set up a JPanel to display with white space?

I'm trying to create a class which represents a playing card as a JPanel, (which I add to a window in another class), but I don't know how to make the panel stay at a specific size when I add it to the display window.

I need to display a grid of 6 cards in the window, so I was using a GridBagLayout to arrange them. Here's what it looks like now though:

The "cards" on the left don't have a number, just a center icon. The ones on the right should have the number in the center, and the icon in the top right, smaller than normal. I can't figure out how to get the panel to not lose all its background space and size, kind of like this I made in paint:

There's missing context in my actual code so I've replicated it in a simpler class here (contained all in main):

JFrame frame = new JFrame("test code");
JPanel ex[] = new JPanel[6];
for(int i=0; i<6; i++) ex[i] = new JPanel();
    
for(JPanel p: ex) {
    JLabel imgLabel = new JLabel(new ImageIcon("file path"));
    JButton numDisplay = new JButton("10");

    //the button's resizing seems to do nothing
    numDisplay.setSize(50, 50);
    numDisplay.addActionListener(e -> { /*used for testing*/ });
        
    //I was hoping that this would make a border that doesn't get omitted
    p.setSize(200, 300);
    p.setOpaque(true);
    p.setBackground(Color.black);
    p.setPreferredSize(new Dimension(200, 300));

    //my current attempt at the card UI design
    p.setLayout(new BorderLayout());
    p.add(imgLabel, BorderLayout.NORTH);
    p.add(numDisplay, BorderLayout.CENTER);
}
        
frame.setSize(500, 800);
frame.setLayout(new GridBagLayout());
GridBagConstraints g = new GridBagConstraints();
for(int i=0; i<6; i++) {
    g.gridx = i%2;
    g.gridy = i%3;
    frame.add(ex[i], g);
}
        
frame.setVisible(true);

I tried to make the panel border appear by making it opaque, changing preferred size, etc. I don't know if its the layout manager of the frame, the pane, or something else because I've only ever set used a null manager before now, and I can't figure out what method I need to get this to work.

Set your JPanel's layout manager to FlowLayout() or BorderLayout() and set its margins using setBorder(BorderFactory.createEmptyBorder(top, left, bottom, right)).