Problem with BGT instruction in 68k assembly when V flag is set but N and Z flags are clear

I’m working on a 68k assembly program to determine if a quadratic equation $Ax^2 + Bx + C = 0$ has real and distinct solutions. The logic should set the value of a variable to $1$ if the discriminant $b^2 - 4ac$ is greater than $0$, indicating real and distinct solutions. Below is the core of the code:

ORG $8000          
a dc.w 4000        
b dc.w 9999        
c dc.w 1000        
d dc.b 0           

START:
      clr.b d3     
      move.w a,d0  
      move.w b,d1  
      move.w c,d2  
      
      ; Calculate b^2 - 4ac
      mulu.w d1,d1 ; d1 = d1 * d1 (calculate b^2)
      
      mulu.w d2,d0 ; d0 = d0 * d1 (calculate a * c)
      
      mulu.w #4,d0 ; d0 = d0 * 4 (calculate 4ac)
      
      sub.w d0,d1  ; d1 = d1 - d0 (b^2 - 4ac)
      
      ; Check if b^2 - 4ac > 0, branch when Z and N flags are 0
      bgt VALID   ; If the result of b^2 - 4ac is greater than 0, jump to VALIDS
      
      move.b d3,d   
      bra END       ; Jump to END (skip the valid calculation part)
      
VALID:
    addq.b #1,d    
    bra END        

END:
    SIMHALT        ; Halt the simulation
    END     START

The Issue:

After executing the subtraction sub.w d0,d1 (calculating $b^2 - 4ac$), I observe that the V flag is set, but the N and Z flags are clear. I understand that the bgt (branch if greater) instruction should behave as follows:

It branches when the flags: Z, N and V flags are all clear
Or It branches when the Z flag is clear, but the N and V flags are both set.

Otherwise, the instruction is ignored.

According to this, bgt shouldn't have branched, even though $b^2 - 4ac$ is greater than $0$.

My Question:

How can I ensure that the branch happens when $b^2 - 4ac$ is greater than $0$, even in this situation where Z is clear , V is set and N is clear?

Solution

How can I ensure that the branch happens when $b^2 - 4ac$ is greater than $0$, even in this situation where Z is clear , V is set and N is clear?

Simple enough, just replace bgt with bra. But all joking aside, read on...

Your bgt seemingly does not work because you didn't calculate the expression correctly!

mulu.w d1,d1 ; d1 = d1 * d1 (calculate b^2)
mulu.w d2,d0 ; d0 = d0 * d1 (calculate a * c)   <<<< Do you see your typo?

Both these mulu instructions result in a 32-bit product that fills the whole of D1 for b^2 and the whole of D0 for a*c. Further operations, like the multiplication by 4 and the final subtraction, must therefore use the full longword.
68k doesn't have a longword mulu, but you're in luck with that factor 4 that nicely allows you to replace that multiplication with a double shift to the left instead: asl.l #2, d0.

I see that you tried to apply some of the logic that I used in my previous answer, but you somehow weren't able to avoid the extra BRA in the end. The trick is to use the opposite condition so you just bypass the increment to the temporary register D2. For bgt the opposite would be ble.

ORG $8000        
a dc.w 4000
b dc.w 9999
c dc.w 1000
d dc.b 0

START:
  clr.b  d2
  move.w a, d0
  move.w b, d1
  mulu.w d1, d1  ; d1 = b^2
  mulu.w c, d0   ; d0 = ac
  asl.l  #2, d0  ; d0 = 4ac
  sub.l  d0, d1  ; d1 = b^2 - 4ac
  ble.s  DONE    ; If (b^2 - 4ac) is less than or equal to 0, bypass the VALID part
VALID:
  addq.b #1, d2  ; Set D2.b = 1
DONE:
  move.b d2, d   ; Copy D2.b to variable D (should you need it)
  SIMHALT

Tip: Don't use 2 instructions if 1 can do the job already. No need to load the c variable to a register first if you can use the variable directly in the multiplication mulu.w c, d0 ; d0 = ac.