Why does `difftime` carry label attribute from other variable in R?

In my example below I create a label for a date variable using Hmisc (can be the labelled package doesn't matter). In the second dataset I use difftime to take the difference between two dates. When you run attributes on the new column the label from dates (first variable supplied to difftime is carried over. Why is this attribute being carried over?

library(Hmisc)


trial <- data.frame(dates = seq(as.Date("1970-01-01"), as.Date("1970-01-01")+199,1),
                dates2 = sample(seq(as.Date('1999/01/01'), as.Date('2000/01/01'), by="day"), 200))

Hmisc::label(trial$dates) <- "New Date"


trial2 <- transform(trial, difftimer = difftime(dates,dates2))
attributes(trial2$difftimer)

difftime calls .difftime and the source code is

.difftime
function (xx, units, cl = "difftime") 
{
    class(xx) <- cl
    attr(xx, "units") <- units
    xx
}

It is just adding/updating attributes i.e. class and units in addition to the already existing attribute. The change is in the class as it is getting assigned to new one

attributes(trial$dates2) # // starts with class only attributes
#$class
#[1] "Date"

attributes(.difftime(trial$dates2, units = 'secs')) # //updated class
#$class
#[1] "difftime"

#$units # // added new attribute
#[1] "secs"

With the 'dates' column, there are two classes and an attribute for 'label'

attributes(trial$dates)
#$class
#[1] "labelled" "Date"    

#$label
#[1] "New Date"

attributes(.difftime(trial$dates, units = 'secs')) 
#$class #// changed class
#[1] "difftime"

#$label #// this attribute is untouched
#[1] "New Date"

#$units
#[1] "secs"