I want to understand what is the difference between using Budgets(cost budget) and Configure Billing Alerts in AWS?
Billing Alerts are very limited as compared to Budgets. Most notably, the billing alarm represents only the amount you have been already charged. In contrast, a budget can alarm you based on forecasted charges, which can give you a bit of head up to figure out what's happening before you get hit with excess bill usage.
The other key difference is that Budgets allow you to create filtered alarms, only for some regions and services of interest. Filtering by region is not possible with billing alerts. Another thing is that budget support linked accounts, which again is not possible with the billing alerts.
Finally, budgets are far easier to use, set up and manage, than billing alerts.