Trying to use pipelines as sources and sinks - it isn't going well

Data Prepper
,
1

Trigger warning, extreme newbie question ahead:

(using “latest” - 2.11.0? - version of data prepper from docker repo)

I’m trying to get pipelines to use other pipelines as sources and sinks. (My ultimate goal is to be able to use routes, but I haven’t got that far yet.)

Using what I thought would be the simplest pipeline config I could come up with to test this:

pipeline1:
  source:
    random:
  sink:
    - pipeline:
        name: pipeline2
pipeline2:
  source:
    - pipeline:
        name: "pipeline1"
  sink:
    - stdout:

When I start the data prepper container with this, all I get is an ugly java crash dump. But if I do the same thing without the pipeline hand off, like this:

pipeline1:
  source:
    random:
  sink:
    - stdout:

It works fine.

What am I doing wrong?

2

OK, so by a pretty-much random walk through all the possible configurations of pipeline file syntax, I found the answer. Or at least, I found an answer that seems to work.

It appears that, even though the official documentation (here) says that the pipeline property should be specified as a sequence element - i.e. “- pipeline:” - it only actually works for me when specified as a map element - i.e. just “pipeline:”, without the “-”. So for me to get it to work I needed to change the pipeline config to:

pipeline1:
  source:
    random:
  sink:
    - pipeline:
        name: pipeline2
pipeline2:
  source:
    pipeline:
      name: "pipeline1"
  sink:
    - stdout:

TBH, I still haven’t got my head around the logic of when YAML elements in config files should be maps and when they should be sequences. I’m sure there must be some logic to it, but it eludes me for now. If anyone feels the urge to enlighten me on this point, it would be much appreciated.

But for now, it appears that either the documentation is incorrect, or the implementation is buggy; no?