Merge two multidimensional arrays and remove duplicate values within indexed subarrays

I need to merge those two arrays:

$ar1 = array("color" => array("red", "green"), "aa");
$ar2 = array("color" => array( "green", "blue"), "bb");
$result = array_merge_recursive($ar1, $ar2);

Expected output:

[
    'color' => [
        (int) 0 => 'red',
        (int) 1 => 'green',
        (int) 3 => 'blue'
    ],
    (int) 0 => 'aa',
    (int) 1 => 'bb'
]

But it outputs:

[
    'color' => [
        (int) 0 => 'red',
        (int) 1 => 'green',
        (int) 2 => 'green', (!)
        (int) 3 => 'blue'
    ],
    (int) 0 => 'aa',
    (int) 1 => 'bb'
]

I'm looking for the simplest way to do this, my array inputs won't be deeper than those examples.

Here it is.

function array_merge_recursive_ex(array $array1, array $array2)
{
    $merged = $array1;

    foreach ($array2 as $key => & $value) {
        if (is_array($value) && isset($merged[$key]) && is_array($merged[$key])) {
            $merged[$key] = array_merge_recursive_ex($merged[$key], $value);
        } else if (is_numeric($key)) {
             if (!in_array($value, $merged)) {
                $merged[] = $value;
             }
        } else {
            $merged[$key] = $value;
        }
    }

    return $merged;
}