I'm new to nextjs and the new updates are a bit overwhelming for me. It may sound dumb, but my question is: when should I use Server Actions and when should I use Route Handlers. Could I use them both in the same project or I can only use one of them (more likely Server Actions).

I have a bit of experience with Server Actions and I worked on a project that is using Server Actions to query from database. However I never used Route Handlers in nextjs (but I understand how they work).

Short answer

Both Server Actions and Route Handlers allow to write server-side code that can be invoked from the client-side. Route Handlers are a low-level abstraction, they allow to write REST API Code:

export async function GET(request: Request) {
const { searchParams } = new URL(request.url)
const id = searchParams.get('id')
const res = await fetch(`http://www.example.com/product/${id}`, {
headers: {
  'Content-Type': 'application/json',
  'API-Key': process.env.DATA_API_KEY!,
},
 })
 const product = await res.json()

 return Response.json({ product })
}

While Server Actions are a high-level abstraction, and they are preferably used to mutate data on the server (such as creating or updating a specific resource on the server). Server Actions under the hood use the POST HTTP method, and they can be invoked only by this HTTP method. Route handlers are preferably used to fetch data. Here's an example of Server Actions, a function updateUser that is located in the file actions.ts:

 // use 'use server' directive to indicate it is a server action
'use server'

 export async function updateUser(userId, formData) {
// ...
 }

And the client component can invoke updateUser as following:

'use client'

import { updateUser } from './actions'

export function UserProfile({ userId }: { userId: string }) {

// using `bind` to send the arguments to the sever action `updateUser`

const updateUserWithId = updateUser.bind(null, userId)

 return (
<form action={updateUserWithId}>
  <input type="text" name="name" />
  <button type="submit">Update User Name</button>
</form>
)
}

You can check the official doc about Server Actions and about Route Handlers

This link that states shortly the difference between the two.

Long answer

You can check this Reddit thread