python - Combining Beam FlatMap pvalue.AsDict and pvalue.AsList - Stack Overflow

admin2025-04-17  2

I have a situation where I want to have call function in beam pipeline with side inputs which will be the lists with some key and it all should resolve to the return list. I started like this:

transformed_output = (
    parsed_input_data
    | "Merge Data" 
    >> beam.FlatMap(
        process_records,
        beam.pvalue.AsList(auxiliary_data),
        beam.pvalue.AsDict(
            {
                "lookup_table_1": beam.pvalue.AsList(lookup_table_data_1),
                "lookup_table_2": beam.pvalue.AsList(lookup_table_data_2),
            }
        ),
    )
)

And here lookup_table_data is the result of previous pipeline. I am getting such error:

self._window_mapping_fn = sideinputs.default_window_mapping_fn( pcoll.windowing.windowfn) E AttributeError: 'dict' object has no attribute 'windowing'

Is it even possible? I want to have as an input anything which will store both list and keys, because I need to use that key later. Also this function will be called mulstiple time with different side inputs parameter.

转载请注明原文地址:http://www.anycun.com/QandA/1744874867a88852.html