dagster

create multiple jobs from similar assets with different partitions and schedules


I have a assets.py with the following:

from dagster import asset, job, Output

from my_modules import some_func


from ..partitions import daily_partitions_def, t1_daily_partitions_def

@asset(partitions_def=t1_daily_partitions_def)
def get_rfr_final_asset(context):
    dt = context.partition_key
    some_func(dt, provisonal=True)
    return True

@asset(partitions_def=daily_partitions_def)
def get_rfr_provisional_asset(context):
    dt = context.partition_key
    some_func(dt, provisonal=False)
    return True

In my main dagster file, how can I create two jobs, one with the first asset and a second with the second asset both on different schedules. If I use load_assets_from_modules it groups both assets together, I'd like a cleaner way to do this instead of having multiple files for each asset.


Solution

  • Something like this should work:

    from assets import get_rfr_final_asset, get_rfr_provisional_asset
    
    rfr_final_job = define_asset_job(
        "rfr_final_job", selection=[get_rfr_final_asset]
    )
    rfr_provisional_job = define_asset_job(
        "rfr_provisional_job", selection=[get_rfr_provisional_asset]
    )