How to remove the borders in usmap plot

I am trying to make a US election map based on the package 'usmap'. The thing is it seems like the package can only do a white background. If I changed the background colour it would show borders. I tried to remove the borders by setting panel.border = element_blank() but it did not seem to work. Is there any way to address this?

library(usmap)
poll <- read_csv('poll.csv')
poll$fips <-fips(poll$State)
poll$Attitude <- factor(poll$Attitude, levels = attitude) 
plot2 <- plot_usmap(data = poll, values = "Attitude",  color = 'white', labels=FALSE) + scale_fill_manual(values=c("#1055b6", "#67b5e2", "#cccccc","#ffaca3","#ed4748")) + guides(fill = FALSE) 
plot2+ theme(plot.background = element_rect(fill = "#fdf1e5"),
             panel.background = element_rect(fill = "#fdf1e5"), 
             panel.border = element_blank())

Here's my data.

   State                Abbrev Code  n_votes Attitude              fips 
 1 Alabama              Ala.   AL          9 Solid Republican      01   
 2 Alaska               Alaska AK          3 Toss-up               02   
 3 Arizona              Ariz.  AZ         11 Toss-up               04   
 4 Arkansas             Ark.   AR          6 Solid Republican      05   
 5 California           Calif. CA         55 Solid Democrat        06   
 6 Colorado             Colo.  CO          9 Leaning to Democrat   08   
 7 Connecticut          Conn.  CT          7 Solid Democrat        09   
 8 Delaware             Del.   DE          3 Solid Democrat        10   
 9 District of Columbia D.C.   DC          3 Solid Democrat        11   
10 Florida              Fla.   FL         29 Leaning to Democrat   12   
11 Georgia              Ga.    GA         16 Toss-up               13   
12 Hawaii               Hawaii HI          4 Solid Democrat        15   
13 Idaho                Idaho  ID          4 Solid Republican      16   
14 Illinois             Ill.   IL         20 Solid Democrat        17   
15 Indiana              Ind.   IN         11 Leaning to Republican 18   
16 Iowa                 Iowa   IA          6 Leaning to Republican 19   
17 Kansas               Kans.  KS          6 Leaning to Republican 20   
18 Kentucky             Ky.    KY          8 Solid Republican      21   
19 Louisiana            La.    LA          8 Solid Republican      22   
20 Maine                Maine  ME          2 Solid Democrat        23   
21 Maryland             Md.    MD         10 Solid Democrat        24   
22 Massachusetts        Mass.  MA         11 Solid Democrat        25   
23 Michigan             Mich.  MI         16 Leaning to Democrat   26   
24 Minnesota            Minn.  MN         10 Toss-up               27   
25 Mississippi          Miss.  MS          6 Solid Republican      28   
26 Missouri             Mo.    MO         10 Leaning to Republican 29   
27 Montana              Mont.  MT          3 Solid Republican      30   
28 Nebraska             Nebr.  NE          2 Solid Republican      31   
29 Nevada               Nev.   NV          6 Leaning to Democrat   32   
30 New Hampshire        N.H.   NH          4 Leaning to Democrat   33   
31 New Jersey           N.J.   NJ         14 Solid Democrat        34   
32 New Mexico           N.M.   NM          5 Solid Democrat        35   
33 New York             N.Y.   NY         29 Solid Democrat        36   
34 North Carolina       N.C.   NC         15 Toss-up               37   
35 North Dakota         N.D.   ND          3 Solid Republican      38   
36 Ohio                 Ohio   OH         18 Toss-up               39   
37 Oklahoma             Okla.  OK          7 Solid Republican      40   
38 Oregon               Ore.   OR          7 Solid Democrat        41   
39 Pennsylvania         Pa.    PA         20 Leaning to Democrat   42   
40 Rhode Island         R.I.   RI          4 Solid Democrat        44   
41 South Carolina       S.C.   SC          9 Toss-up               45   
42 South Dakota         S.D.   SD          3 Solid Republican      46   
43 Tennessee            Tenn.  TN         11 Solid Republican      47   
44 Texas                Tex.   TX         38 Toss-up               48   
45 Utah                 Utah   UT          6 Leaning to Republican 49   
46 Vermont              Vt.    VT          3 Solid Democrat        50   
47 Virginia             Va.    VA         13 Leaning to Democrat   51   
48 Washington           Wash.  WA         12 Solid Democrat        53   
49 West Virginia        W.Va.  WV          5 Solid Republican      54   
50 Wisconsin            Wis.   WI         10 Leaning to Democrat   55   
51 Wyoming              Wyo.   WY          3 Solid Republican      56 

Thanks

Dan

The theme element panel.border inherits from the panel.background rect object. In your code, it's being overwritten by your rect object for panel.background. Your panel.background element only specifies fill=, so color= (the color of the border around the panel) is by default set to "black". If you want to specify the color of the boarder for the panel.background, you should do that inside the element_rect for that object. For example, this will remove your panel border (but keep the plot border):

plot2+ theme(plot.background = element_rect(fill = "#fdf1e5"),
             panel.background = element_rect(color=NA, fill = "#fdf1e5"))

Also, the plot.background is the entire plot which includes plotting area and the stuff around like axis labels, text, etc. The panel.background is the drawing area of the plot itself: it's where all the points are drawn on the plot. You have no doubt understood already that the inside box is the panel.background and the "outside" box is the plot.background. If you only specify the plot.background, then there's no need to specify the panel.background.

As an example, the following gives you the same result (removing the panel.background object entirely):

plot2+ theme(plot.background = element_rect(fill = "#fdf1e5"))

I was unable to get your data frame to work exactly (without doing a lot of work to import), so you can observe the effect using the built in population data from the usmap package:

plot_usmap(data=statepop, values="pop_2015", color='white', labels=FALSE) + guides(fill=FALSE) +
  theme(plot.background = element_rect(fill = "#fdf1e5"))

Oh, and if you wanted to remove all borders (not just the inside one), you should specify that right in the plot.background element in the same way:

theme(plot.background = element_rect(color = NA, fill = "#fdf1e5"))