I have implemented the streaming pipeline using pipeline builder and everything works fine. It get updated every 5 secs. Now I have to implement the same logic using code repository. Wanted to check if anyone have already tried to implement the same and how does it work? Is there any specific decorator or just @increment should be used?
I have implemented the streaming pipeline using pipeline builder and everything works fine. It get updated every 5 secs. Now I have to implement the same logic using code repository. Wanted to check if anyone have already tried to implement the same and how does it work? Is there any specific decorator or just @increment should be used?
Pipeline Builder is the main way to build streaming pipelines.
To my knowledge, you can't use a Code Repository to define your Streaming pipelines the same way you do with Pipeline Builder.
However, you can use Code Repositories to define Streaming UDFs, which then can be called from Pipeline builder like any other transformation. You can essentially build your own "library" of function (which can output 1 or N rows, keyed or not, etc.) and use those bricks in Pipeline Builder. https://www.palantir.com/docs/foundry/transforms-java/user-defined-functions#publish-to-a-deployment-repository
