The Basics of Pipes and Filters Pattern

A brief look at the pipes and filters pattern and the aspects related to usability.The Pipes and Filters pattern is a component-based architectural design pattern that allows the decomposition of a monolithic component into smaller independent elements…

A brief look at the pipes and filters pattern and the aspects related to usability.

The Pipes and Filters pattern is a component-based architectural design pattern that allows the decomposition of a monolithic component into smaller independent elements which can be reused.

Problem statement

Task processing in a monolithic module

In the above implementation, a monolithic module is responsible to maintain the task lifecycle with multiple constraints, a few of which can be listed as follows:

  1. Tight coupling: Based on the use case, a few of the tasks might require different programming languages for ease of implementation.
  2. Resource constraints: Each or a group of tasks might need different kinds of resources, for instance, some of them might be CPU or memory intensive while others might need specialized hardware.
  3. Low scalability: Each task might require a different number of instances for graceful processing.

The above problems shouldn’t be a bottleneck in the system in terms of performance and cost-effectiveness.

Solution

One solution to the above design problem is to decompose the monolithic system into a separate pattern that offers reusability and each component performing a single specialized task. Toolchains such as Bit provide the architectural framework to build such a system.

How to Create a Composable React App with Bit

Each smaller component can be assembled together to make a pipeline. Similarly, it can be used to build multiple pipelines, each resulting in a different end state.

Let’s implement the solution using pipes and filters pattern

Pipes and Filters pattern

Topological Constraints

  1. Due to the nature of the pipes and filters where multiple sources may exist with at most one sink to aggregate processed data.
  2. Filters are isolated from one another and they need to be provided with sufficient context and state information. They can only communicate through the pipes.
  3. Since there can be no cycles in between pipes, the flow can only be unidirectional in nature.

In the above diagram of pipes and filters, we still have issues like performance bottlenecks and single-point-of-failure due to only one available instance of the filters. The approach can be modified as:

Scaling up each individual task component

Based on the use case, the number of replicas can be scaled up or down for individual components to make the pipeline efficient and optimized.

Issues and considerations

  1. Complexity: With the increase in reusability and scalability, complexities might arise in the pipeline. The solutions should keep the pipeline reliable and idempotent. Any kind of duplicity may cause retrigger of the components.
  2. Context and State: Each filter inside a pipeline runs in isolation. Any use case related to dependency needs to deal with sufficient context related to the state.

Usability

The pipes and filters design pattern is useful only when:

  1. The bigger task can be decomposed into smaller decoupled units.
  2. The system demands flexibility and the reordering of steps is a part of the functional requirement.
  3. Distributing tasks across multiple servers should result in performance or/and cost benefits.

References

Summary

In this article, we looked into the basics of pipes and filters pattern and the aspects related to usability. We will deep dive into some of the specific case studies of the distributed systems making use of this pattern in future articles.

Thank you ❤️ for reading this article, feel free to Subscribe, clap👏🏻 and share it.

From monolithic to composable software with Bit

Bit’s open-source tool help 250,000+ devs to build apps with components.

Turn any UI, feature, or page into a reusable component — and share it across your applications. It’s easier to collaborate and build faster.

Learn more

Split apps into components to make app development easier, and enjoy the best experience for the workflows you want:

Micro-Frontends

Design System

Code-Sharing and reuse

Monorepo

Learn more


The Basics of Pipes and Filters Pattern was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.


Print Share Comment Cite Upload Translate
APA
Kamini Kamal | Sciencx (2024-03-28T10:02:22+00:00) » The Basics of Pipes and Filters Pattern. Retrieved from https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/.
MLA
" » The Basics of Pipes and Filters Pattern." Kamini Kamal | Sciencx - Friday February 3, 2023, https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/
HARVARD
Kamini Kamal | Sciencx Friday February 3, 2023 » The Basics of Pipes and Filters Pattern., viewed 2024-03-28T10:02:22+00:00,<https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/>
VANCOUVER
Kamini Kamal | Sciencx - » The Basics of Pipes and Filters Pattern. [Internet]. [Accessed 2024-03-28T10:02:22+00:00]. Available from: https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/
CHICAGO
" » The Basics of Pipes and Filters Pattern." Kamini Kamal | Sciencx - Accessed 2024-03-28T10:02:22+00:00. https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/
IEEE
" » The Basics of Pipes and Filters Pattern." Kamini Kamal | Sciencx [Online]. Available: https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/. [Accessed: 2024-03-28T10:02:22+00:00]
rf:citation
» The Basics of Pipes and Filters Pattern | Kamini Kamal | Sciencx | https://www.scien.cx/2023/02/03/the-basics-of-pipes-and-filters-pattern/ | 2024-03-28T10:02:22+00:00
https://github.com/addpipe/simple-recorderjs-demo