<ahref="#ParallelDataPipelineIncludeHeaderFile">Include the Header</a>
</li>
<liclass="level1">
<ahref="#CreateADataPipelineModuleTask">Create a Data Pipeline Module Task</a>
</li>
<liclass="level1">
<ahref="#UnderstandInternalDataStorage">Understand Internal Data Storage</a>
</li>
</ul>
</div>
<divclass="textblock"><p>Taskflow provides another variant, <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a>, on top of <aclass="el" href="classtf_1_1Pipeline.html" title="class to create a pipeline scheduling framework">tf::Pipeline</a> (see <aclass="el" href="TaskParallelPipeline.html">Task-parallel Pipeline</a>) to help you implement data-parallel pipeline algorithms while leaving data management to Taskflow. We recommend you finishing reading TaskParallelPipeline first before learning <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a>.</p>
</div><!-- fragment --><h1><aclass="anchor" id="CreateADataPipelineModuleTask"></a>
Create a Data Pipeline Module Task</h1>
<p>Similar to creating a task-parallel pipeline (<aclass="el" href="classtf_1_1Pipeline.html" title="class to create a pipeline scheduling framework">tf::Pipeline</a>), there are three steps to create a data-parallel pipeline application:</p>
<oltype="1">
<li>Define the pipeline structure (e.g., pipe type, pipe callable, stopping rule, line count)</li>
<li>Define the data storage and layout, if needed for the application</li>
<li>Define the pipeline taskflow graph using composition</li>
</ol>
<p>The following example creates a data-parallel pipeline that generates a total of five dataflow tokens from <code>void</code> to <code>int</code> at the first stage, from <code>int</code> to <code>std::string</code> at the second stage, and <code>std::string</code> to <code>void</code> at the final stage. Data storage between stages is automatically managed by <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a>.</p>
<divclass="ttc" id="aclasstf_1_1DataPipeline_html"><divclass="ttname"><ahref="classtf_1_1DataPipeline.html">tf::DataPipeline</a></div><divclass="ttdoc">class to create a data-parallel pipeline scheduling framework</div><divclass="ttdef"><b>Definition</b> data_pipeline.hpp:254</div></div>
<divclass="ttc" id="aclasstf_1_1Executor_html"><divclass="ttname"><ahref="classtf_1_1Executor.html">tf::Executor</a></div><divclass="ttdoc">class to create an executor</div><divclass="ttdef"><b>Definition</b> executor.hpp:62</div></div>
<divclass="ttc" id="aclasstf_1_1Executor_html_a519777f5783981d534e9e53b99712069"><divclass="ttname"><ahref="classtf_1_1Executor.html#a519777f5783981d534e9e53b99712069">tf::Executor::run</a></div><divclass="ttdeci">tf::Future< void > run(Taskflow &taskflow)</div><divclass="ttdoc">runs a taskflow once</div></div>
<divclass="ttc" id="aclasstf_1_1Pipeflow_html"><divclass="ttname"><ahref="classtf_1_1Pipeflow.html">tf::Pipeflow</a></div><divclass="ttdoc">class to create a pipeflow object used by the pipe callable</div><divclass="ttdef"><b>Definition</b> pipeline.hpp:43</div></div>
<divclass="ttc" id="aclasstf_1_1Pipeflow_html_a295e5d884665c076f4ef5d78139f7c51"><divclass="ttname"><ahref="classtf_1_1Pipeflow.html#a295e5d884665c076f4ef5d78139f7c51">tf::Pipeflow::token</a></div><divclass="ttdeci">size_t token() const</div><divclass="ttdoc">queries the token identifier</div><divclass="ttdef"><b>Definition</b> pipeline.hpp:78</div></div>
<divclass="ttc" id="aclasstf_1_1Pipeflow_html_a830b7f204cb87fff17e8d424918d9453"><divclass="ttname"><ahref="classtf_1_1Pipeflow.html#a830b7f204cb87fff17e8d424918d9453">tf::Pipeflow::stop</a></div><divclass="ttdeci">void stop()</div><divclass="ttdoc">stops the pipeline scheduling</div><divclass="ttdef"><b>Definition</b> pipeline.hpp:88</div></div>
<divclass="ttc" id="aclasstf_1_1Taskflow_html"><divclass="ttname"><ahref="classtf_1_1Taskflow.html">tf::Taskflow</a></div><divclass="ttdoc">class to create a taskflow object</div><divclass="ttdef"><b>Definition</b> taskflow.hpp:64</div></div>
<divclass="ttc" id="anamespacetf_html_a8975fa5762088789adb0b60f38208309"><divclass="ttname"><ahref="namespacetf.html#a8975fa5762088789adb0b60f38208309">tf::make_data_pipe</a></div><divclass="ttdeci">auto make_data_pipe(PipeType d, C &&callable)</div><divclass="ttdoc">function to construct a data pipe (tf::DataPipe)</div><divclass="ttdef"><b>Definition</b> data_pipeline.hpp:171</div></div>
</div><!-- fragment --><p>The interface of <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a> is very similar to <aclass="el" href="classtf_1_1Pipeline.html" title="class to create a pipeline scheduling framework">tf::Pipeline</a>, except that the library transparently manages the dataflow between pipes. To create a stage in a data-parallel pipeline, you should always use the helper function <aclass="el" href="namespacetf.html#a8975fa5762088789adb0b60f38208309" title="function to construct a data pipe (tf::DataPipe)">tf::make_data_pipe</a>:</p>
</div><!-- fragment --><p>The helper function starts with a pair of an input and an output types in its template arguments. Both types will always be decayed to their original form using std::decay (e.g., <code>const int&</code> becomes <code>int</code>) for storage purpose. In terms of function arguments, the first argument specifies the direction of this data pipe, which can be either <aclass="el" href="namespacetf.html#abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" title="serial type">tf::PipeType::SERIAL</a> or <aclass="el" href="namespacetf.html#abb7a11e41fd457f69e7ff45d4c769564adf13a99b035d6f0bce4f44ab18eec8eb" title="parallel type">tf::PipeType::PARALLEL</a>, and the second argument is a callable to invoke by the pipeline scheduler. The callable must take the input data type in its first argument and returns a value of the output data type. Additionally, the callable can take a <aclass="el" href="classtf_1_1Pipeflow.html" title="class to create a pipeflow object used by the pipe callable">tf::Pipeflow</a> reference in its second argument which allows you to query the runtime information of a stage task, such as its line number and token number.</p>
<divclass="ttc" id="aclasstf_1_1Pipeflow_html_afee054e6a99965d4b3e36ff903227e6c"><divclass="ttname"><ahref="classtf_1_1Pipeflow.html#afee054e6a99965d4b3e36ff903227e6c">tf::Pipeflow::line</a></div><divclass="ttdeci">size_t line() const</div><divclass="ttdoc">queries the line identifier of the present token</div><divclass="ttdef"><b>Definition</b> pipeline.hpp:64</div></div>
</div><!-- fragment --><dlclass="section note"><dt>Note</dt><dd>By default, <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a> passes the data in reference to your callable at which you can take it in copy or in reference depending on application needs.</dd></dl>
<p>For the first pipe, the input type should always be <code>void</code> and the callable must take a <aclass="el" href="classtf_1_1Pipeflow.html" title="class to create a pipeflow object used by the pipe callable">tf::Pipeflow</a> reference in its argument. In this example, we will stop the pipeline when processing five tokens.</p>
<iframescrolling="no" frameborder="0" src="dot_pipeline_basic_dependency_graph.svg" width="566" height="252"><p><b>This browser is not able to show SVG: try Firefox, Chrome, Safari, or Opera instead.</b></p></iframe></div>
<p>By default, <aclass="el" href="classtf_1_1DataPipeline.html" title="class to create a data-parallel pipeline scheduling framework">tf::DataPipeline</a> uses <ahref="https://en.cppreference.com/w/cpp/utility/variant">std::variant</a> to store a type-safe union of all input and output data types extracted from the given data pipes. To avoid false sharing, each line keeps a variant that is aligned with the cacheline size. When invoking a pipe callable, the input data is acquired in reference from the variant using <ahref="https://en.cppreference.com/w/cpp/utility/variant/get">std::get</a>. When returning from a pipe callable, the output data is stored back to the variant using assignment operator. </p>
</div></div><!-- contents -->
</div><!-- PageDoc -->
</div><!-- doc-content -->
<!-- HTML footer for doxygen 1.13.1-->
<!-- start footer part -->
<divid="nav-path" class="navpath"><!-- id is needed for treeview function! -->