Process blank field values from API response using dataframe

I am getting two records for the ID I have sent in the request. Both records can have different output fields. The response for two records is

ref_securities =

{
"name": "ABC",
"flag": "Y",
"region": "USA"
}
,
 
{
"sed": "",
"name": "ABC",
"flag": "N",
"region": "USA"

}

I have defined refence fields in a list

ref_ids = ['name','flag','region','sed']

I am putting the two records which I get from API using below code in datafrane

soi = [
        {market_id: ref_security.get(market_id, None) for market_id in ref_ids}
        for ref_security in ref_securities
    ]
    final_df = pd.DataFrame(soi)

problem is since both records have different fields and 'sed' in second record is blank the order of the records in final_df is not same. the sequence is distorted. Is there any way to fix it?

Its not perfect and you'd need to handle for more api calls but this does do what i think you want.

import pandas as pd
import json

ref_securities1={
    "name": "ABC",
    "flag": "Y",
    "region": "USA"
}

ref_securities2={
        "sed": "",
        "name": "ABC",
        "flag": "N",
        "region": "USA"
    }

ref_ids = ['name','flag','region','sed']

ref_securities = json.dumps(ref_securities1), json.dumps(ref_securities2)

i = 0
while i < len(ref_securities):
    if i == 0:
        ref_securitiesjs = json.loads(ref_securities[i])
        df1 = pd.json_normalize(ref_securitiesjs, meta = ref_ids)
    else:
        ref_securitiesjs = json.loads(ref_securities[i])
        df2 = pd.json_normalize(ref_securitiesjs, meta = ref_ids)
    i = i + 1
frames = [df1, df2]
result = pd.concat(frames)
print(result)