Horizontal ScrollView scrolls outside of the visible area - SwiftUI

As can be seen in the gif loop, the scrollbar will occasionally scroll randomly out of view and the scrollbar must first be dragged to the right for the items to appear.

This only happens every now and then and is totally random. I don't think it's due to proxy(scrollTo:), because even without scrolling to the selected element, the elements within the scrollbar are sometimes not visible.

here is the code behind it:

    @ViewBuilder func buildScrollView() -> some View {
        ScrollViewReader { proxy in
            ScrollView(.horizontal, showsIndicators: false) {
                HStack {
                    ForEach(scaleEntries, id: \.self.title) { entry in
                        VStack {
                            Button(action: {
                                self.selectedScaleEntry = entry
                                saveRating()
                                // this looks absolutly stupid but it needs to be done to update the state of the parent views
                                self.viewModel.selectedComparisonElement = self.viewModel.selectedComparisonElement
                            }) {
                                Text(entry.scaleValueIdentifier ?? "")
                                    .font(.custom("MB Corpo S Text WEB", size: 24).weight(.bold))
                                    .frame(width: 40, height: 40)
                                    .padding()
                                    .foregroundColor(.black)
                                    .background(Color(UIColor(hex: entry.color!)))
                                    .border(selectedScaleEntry?.title != entry.title ? .clear : .gray, width: 4)
                                    .id(entry.color)
                            }
                            Text(entry.scaleValueTitle ?? " ")
                                .font(.custom("Helvetica", size: 12).weight(.light))
                                .fixedSize(horizontal: false, vertical: true)
                        }
                    }
                }
                .frame(maxWidth: .infinity, maxHeight: 110)
            }
            .padding()
            .onChange(of: self.selectedScaleEntry) { _ in
                if selectedScaleEntry != nil {
                        withAnimation {
                            proxy.scrollTo(selectedScaleEntry?.color, anchor: .trailing)
                        }
                }
            }
        }
    }

I thought it could be the ScrollReader, but I don't think that's the case. It could possibly also be due to the frame, but I'm not sure about that either.

I added the .id() call on the VStack instead of the text element, somehow everything seems to be working now.