What data point this graph is using? Is the graph representing TTFB (time to first byte) or TTLB (time to last byte)
Here are the settings:
Graph is representing TTLB (time to last byte) because it uses JMeter's Response time
If you want TTFB (time to first byte), you need to use Latency
JMeter measures the latency from just before sending the request to just after the first response has been received. Thus the time includes all the processing needed to assemble the request as well as assembling the first part of the response, which in general will be longer than one byte
If you want to display TTFB, a trick could be to replace latency with response time columns values and then display values in graph