This content originally appeared on Bits and Pieces - Medium and was authored by Ashan Fernando
Understanding the dynamics of Monorepo CI/CD
Over the years, monorepos has become a popular strategy for managing large codebases. The reason behind this is straightforward: it's the simplest to start with.
This also comes with many challenges, but one challenge stands out from the rest. That's when using a build system that understands different project contexts.
This article looks into why classical build systems fail in the context of monorepos and explores modern solutions that streamline CI/CD processes.
What is a Monorepo?
A monorepo is a version-controlled code repository that houses multiple projects or modules. Unlike multi-repo setups where each project has its own repository, monorepos consolidate all related projects into one. This approach offers several benefits:
- Improved collaboration: Teams working on different projects can collaborate more effectively.
- Simplified dependency management: Dependencies between projects are easier to manage within a single repository.
- Ease of code sharing: Developers can easily share and reuse code across different projects.
Companies like Google, Facebook, and Microsoft have successfully implemented monorepos to streamline their development workflows and enhance collaboration across teams. They have heavily invested in developing build systems and operating them at scale.
However, this could become an overhead for many businesses. Therefore, most of them would look for third-party tools and complete build systems that can handle the complexities of monorepos.
Challenges in Monorepo Buid Systems
Implementing CI/CD pipelines in a monorepo setup presents unique challenges. The interconnected nature of monorepos complicates build and test pipelines, making it difficult to achieve efficient and reliable CI/CD processes. Some common challenges include:
- Complexity of build and test pipelines: Monorepos require sophisticated build and test pipelines that can handle dependencies and changes across multiple projects.
- Incremental builds: Implementing incremental builds, where only the changed parts of the codebase are rebuilt, is crucial for reducing build times but is difficult to achieve with classical build systems.
- Parallel execution: Efficiently running builds and tests in parallel is essential for speeding up the CI/CD process.
- Managing dependencies and versioning: Ensuring consistency and compatibility across multiple projects within a monorepo requires careful management of dependencies and versioning.
To understand these complexities, let’s look at an example of a monorepo of Lambda functions. The project structure follows.
project-root/
├── api-gateway/
│ │ └── ...
├── lambdas/
│ ├── count-lambda/
│ │ ├── count-lambda.docs.mdx
│ │ ├── count-lambda.ts
│ │ └── index.ts
│ ├── date-lambda/
│ │ └── ...
│ ├── flights-lambda/
│ │ └── ...
│ ├── hello-lambda/
│ │ └── ...
│ ├── random-lambda/
│ │ └── ...
│ └── success-lambda/
│ └── ...
└── utils/
│ ├── date-util/
│ │ └── ...
│ ├── random-util/
│ │ └── ...
Here, each Lambda function has its own lifecycle from development to runtime. Setting up CI/CD for these Lambda functions is time-consuming yet doable.
But, as you can see, some of them are dependent on utility functions like date-util where if it's being modified, we need to deploy both date-lambda and hello-lambda functions.
But, how does your CI know that date-util get modified and it needs to build and deploy both Lambda functions?
Unless your CI is configured to do so, it won’t be able to follow the action of deploying both Lambda functions. This is a simple example of the complexity that a monorepo could introduce to your build system. Just imagine when the number of shared codes grows. The CI/CD configurations could get extremely complex.
But what if we don’t won’t deploy both Lambda functions when date-util get modified?
This is another type of complexity it adds with dependent components. To address this, we may need to package the date-util and publish it into a package registry so that each Lambda can refer to a particular version. However, requiring a separate CI/CD pipeline to publish this package also adds another level of complexity.
But what if we modify the api-gateway configuration, hello-lambda function, and date-util altogether?
These modifications introduce another problem: the CI should be intelligent enough to determine the dependency order and build the dependencies in order. Besides, non-dependent components in the code base could run in parallel to reduce the overall build duration. Yet, this would also add more complexity to the overall build system.
Once you set up a complicated list of workflows, handling dependencies, versions, etc., whenever the application gets modified, your build system becomes a bottleneck and could become fragile. Therefore, you may need modern tools and tech to address these challenges.
Modern Solutions for Monorepo CI/CD
Modern tools have been developed to address the limitations of classical build systems and address these challenges. These tools can be generally categorized into two.
Architectural Tools for Monorepo
These types or tools and platforms allow to define the structure of the monorepo that fundamentally elements the challenges listed above. Unfortunately not many tools out there under this category. One such tool is Bit that helps to create the entire monorepo with fundamental units called Bit Component. A component can be a frontend or backend code module comprising a specific functionality, configuration, and documentation.
This, way the code in your monorepo is built with unique boundaries where you compose them together to build different projects, applications, or even packages. If you are new to composable architectures, you can refer to the following article to get a better understanding.
Composable Software Architectures are Trending: Here's Why
Some knowledge is available for your CI to build components (or collections of components) that only get modified, navigating through its dependency tree. If we take the Lambda example above, let's say each component in the diagram is a Bit component. Bit understands which component got modified and knows which Lambda function (component) must be built and published into AWS, along with its dependency structure.
Each component can also use different versions of other components, allowing flexibility for different projects to upgrade at their own pace.
Utility Tools for Monorepos
There are many tools out there that provide utility functionality to address different challenges when setting up CI/CD for monorepos. Few such tools are.
- Nx: Nx is a powerful toolkit for monorepos that extends classical build systems with advanced features like incremental builds, efficient dependency management, and parallel execution. Nx enables developers to scale their monorepos without compromising on performance.
- Bazel: Developed by Google, Bazel is a build system designed for large-scale codebases. Bazel supports incremental builds and parallel execution, making it a suitable choice for monorepos.
- Lerna: Lerna is a tool for managing JavaScript projects with multiple packages. It provides features like dependency hoisting and incremental builds, simplifying the management of monorepos.
- Turborepo: Turborepo is a high-performance build system for monorepos that focuses on speed and efficiency. It offers features like caching and parallel execution to accelerate CI/CD processes.
For example, in a monorepo, where there are codes of different packages, these utilities can ensure that only the modified code is built. The remaining code could take the build artifacts from its cache. A good example of this would be NX workspaces.
These tools identify the problem and devise a solution to address that one by one.
However, we can avoid these problems by redefining how we create monorepos.
Conclusion
Monorepos offer significant advantages in terms of code sharing, collaboration, and dependency management. However, classical build systems fall short in handling the complexities of monorepo setups. Platforms like Bit, Nx, Bazel, Lerna, and Turborepo provide solutions to streamline CI/CD processes and improve efficiency.
By adopting composable monorepos and leveraging modern CI/CD tools, development teams can achieve faster build times, more efficient resource utilization, and enhanced developer productivity.
Thanks for reading. Cheers!
Learn More
- JavaScript Monorepos: Exploring Decentralized Alternatives
- Building a Vue.js Monorepo in 2023
- Solving the Stress of Monorepo Management with Bit
- 5 Common Problems with MonoRepos and How to Fix Them
- Monorepo, Poly-repo, or No Repo at all?
Monorepo: Why Classical Build Systems Fail? was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Bits and Pieces - Medium and was authored by Ashan Fernando
Ashan Fernando | Sciencx (2024-08-01T11:55:51+00:00) Monorepo: Why Classical Build Systems Fail?. Retrieved from https://www.scien.cx/2024/08/01/monorepo-why-classical-build-systems-fail/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.