Discussion of App Structure for 2.1

[dcr8898]

Thank you!!

First, I would like to thank all of the contributors for the amazing work that went into 2.0! I am salivating at the thought of what 2.1 will bring in full-stack awesomeness!

Summary

My proposal here is really just an invitation to open a discussion around how applications will be organized in Hanami 2.1. I first raised this issue in another proposal here.

In that proposal, I asked where mailers and repositories will be placed in 2.0. @jodosha replied that he was leaning towards placing both mailers and repositories in app/. I would like to propose another approach. In doing so, I am hoping to spark a conversation with the core team and other interested users.

I think this is an important issue that is already drawing attention in the community (see e.g., this recent issue: where to put the lib directory of a slice).

Hanami 1.x Simple App

For context, a basic Hanami 1.x app is organized like this, with the UI in the apps directory (with a default web app), and the business logic in the lib directory (other objects and directories are omitted for clarity):

Hanami 1.x Advanced App

A more advanced 1.x app might look something like this, with two apps, web and admin:

Discussion of 1.x

According to @jodosha, one of the problems with the 1.x organizational strategy is that mailers are located in lib, where they have difficulty utilizing assets that are stored in apps.

Also, there does not seem to be a a prescribed way to “slice” or modularize the business logic in the same way as the UI can be sliced into “apps.”

Hanami 2.0

As @jodosha indicated in his response to my previous proposal, in Hanami 2.0 mailers have been moved to the app directory. This makes a lot of sense, since mailers are essentially analogous to views and should be part of the UI.

However, since 2.0 does not include a persistence layer, it is not yet clear where repositories will go. The current guide example uses a lib/bookshelf/persistence directory. Meanwhile, @timriley’s decafsucks example app places the repository in the app directory, in a repo.rb file.

A larger app that follows this example might look like this:

Problems with this Approach

Placing the repository directories in the app directory creates a number of issues that might make long-term maintenance of Hanami apps difficult.

• It commingles persistence objects with UI objects.

• It couples repositories to slices, which essentially dictates that business logic be organized along the same seams that divide app slices.

• It reduces (eliminates, really) the cohesion of business logic and persistence.

A Modest Proposal

If Hanami is going to reach it’s potential to become a framework that can truly scale from “startup to IPO,” then it must be able to scale from Majestic Monolith to Modular Monolith.

If Hanami apps are going to avoid the middle-age crises that have famously plagued Rails apps that reach a certain size, then there must be a way to “modularize” (in an architectural sense, not the Ruby module sense) the apps’ business logic. This means building into Hanami a way to create vertical boundaries in the business logic, just as we can with slices in the UI.

Such an app might be organized as follows:

Advantages

• The seams in the business logic are free to fall independently from those of the UI.

• By keeping related repositories with the business logic module that utilizes them, we improve cohesion and eliminate coupling to other business logic modules or the UI–including stealth coupling through the database layer.

• All slices are free to utilize whichever business logic module they need, or multiple ones. In this diagram, the admin UI might only need the auth module, while the web UI needs to work with both auth and lending.

Conclusion

I apologize if all of this comes across as overly pedantic, but I very keen to see Hanami fulfill all of the promise it has shown. If this promise is to include projects that prioritize scalability and long-term maintainability, then there needs to be a pathway to the Modular Monolith. This proposal may or may not provide the best path, but it is my hope that it will spark some conversation here–the more ideas we can get out into the open for discussion and refinement, the better Hanami 2.1 (and 2.x) will be!

Thank you!

[timriley]

Thank you for this amazingly thoughtful contribution, @dcr8898!

I’d like to be able to think about this carefully and reply in depth, but I’m away at a Ruby conference as of today, so I just wanted to let you know that I’ll come back to this in about a week’s time.

[dcr8898]

@timriley, thank you for your response–and enjoy the conference! If you are speaking, I hope it will be recorded!

Upon reflection, I realize that (along with some pontificating ) I am carelessly conflating two issues in my proposal:

1. The location of repositories, and

2. The ability to “modularize” business logic.

While the two are connected, they are not the same. Nevertheless, I think that both are important for the usability of Hanami and the long-tern growth and maintainability of Hanami apps.

More Discussion . . . (not a lecture, I just need to get some thoughts out of my head)

For years developers have struggled with the maintainability of Rails apps that reach a certain size and complexity. Since Rails itself does not offer an official solution to this problem, developers have come up with countless different countermeasures.

The most common method is to lean-in to organizational schemes, like service objects or namespaces. This can work for a while, but ultimately no code organization scheme alone can avoid the big ball of mud as the code base increases in complexity. The only way to tame this complexity is to divide the code base with clear boundaries and define rules for communication across those boundaries.

Absent any canonical approach for this in Rails, most large projects have chosen to pursue a microservices strategy–essentially the Sledge Hammer Approach to code modularization. However, microservices may not be worth the added complexity and should not be necessary to avoid the “big ball of mud.” (And yes, I’m already tired of that term too.)

While microservices may be overkill for many projects, alternatives to microservices in the Rails world also tend to be complex. See, for example, Component-Based Rails Applications. The author of that approach, Steve Hagemann, has now dropped it in favor of “gradual modularization” based on Shopify’s packwerk gem. Gusto is pushing a similar approach they call ruby-at-scale.

I believe that all of these tools pale in comparison to the simplicity that Hanami “modules” could offer (okay, I recognize that we need a better term for this concept).

Back to the Current Proposal

1. Placing repositories in lib, at the same level as other business objects increases cohesion and decreases unnecessary coupling to the UI layer.

2. Providing a pathway for business logic to be “modularized” in Hanami projects will allow these projects to scale far beyond the point at which similar Rails apps become problematic to modify, maintain, and deploy.

A speculative gaze into the future . . .

While the 2.0 guides don’t yet have a section on architecture, the 1.3 guides discuss Hanami’s architectural philosophy in some detail. They describe the architectural goals of Hanami as “clean architecture and monolith first” .

In this design, the “Application Core” resides in lib/, and “The idea is to develop our application like a Ruby gem.” However, this goal was not really attainable because the necessary database configuration (and migrations) that the application core relied upon was located in the project root, in the db directory.

Imagine that we take the current proposal a step further and move all database configuration into the lib/<app_name> directory. In other words, in a new Hanami 2.1 app, we would move db/ from the project root to lib/<app_name>/db/. In more complex projects, each module under lib/<app_name>/ would have its own db/ directory with configuration and migrations for the tables used in that module.

In other words, each module could provide its own configuration and/or migrations, or it could choose to inherit any or all of these from its parent directory.

This arrangement would have two notable benefits:

1. Avoid stealth coupling between modules at the database level.

2. Allow for a lightweight framework for “*-hanami” gems (lighter than, say, Rails Engines).

This approach brings us much closer to the idea of “developing our application like a gem.” Someone could, for example, take Rodauth, wrap it in some code (complete with migrations), and provide a module that could be dropped into any Hanami app. It wouldn’t take much work to convert such a module into a Roduath-Hanami gem. Such a gem could provide all needed migrations, an interface for Rodauth, and perhaps supply scripts to generate actions and views utilizing this interface. Sounds nice, right?

Conclusion

I don’t know if what I propose is the best solution, but I would love to see a Hanami 2.1 that

• encourages cohesion,

• discourages coupling, and

• provides a clear pathway to a modular monolith.

What do you think?

Thanks,

Damian

[swilgosz]

I have added a rough idea of different approach, that we could maybe use for some inspiration to further discussion.

We could leverage slices, to achieve both: Input-based modules (admin, API, web), and business domains (ordering, booking, registering). Slice does not need to have actions and we could use DDD terminology to organize modules there if we wish.

Then the UI slices could import necessary components from business domains via the slice importing mechanism.

Hanami 2.0 App Structure

I thought about using slices in different ways, and because of how slices can export some components, and because we can load different slices conditionally, I came up with this idea (don’t know yet how it’ll work out). Use slices based on the interface type $ tree slices ├── slices │ ├── api │ ├── admin │ ├── web │ └── client Use slices based on the domain model $ tree slices ├── slices │ ├── ordering │ ├── publishing │ ├── booking │ └── registering I’m not sure, if allow…

Also, some questions raised by one of our team members related to this would be:

1. Business logic components don’t really need assets, while UI-oriented could need. How we would organize our assets pipeline to work with SPA
2. How to manage middlewares?
3. how to handle complex params validation with rules - where is the border, of using params, and when contract become a business-logic validaition? How do you distinguish between business model, and APP-level validations?

I’ll experiment with a sample OSS application very soon, but these questions are some that make me thiking a lot about the app structures in hanami.

I personally avoid using lib too much - I tend to place them things like: utils, and general, agnostic pieces of code, that will most likely be used by my whole app, and can be also extracted to external gems.

[dcr8898]

@swilgosz, thank you for your insight! I added a reply to your comment in the other thread.

I definitely agree that mailers belong in app or slices, since they are UI-centric like all of the other objects in that layer. This should give them free access to the assets in the asset pipeline. If the app is using a JS front end, like React, then I would expect to follow a pattern similar to Rails, where JS code is stored within asset directories in app, and then compiled to a public directory or something similar. In this case, JS-based business logic would be stored in app or its sub-directories, and not in lib.

Middlewares, if I understand you correctly, are part of the Rack app stack, but could invoke business logic in lib in the same way that the Hanami router/actions do (since Hanami is just another Rack app).

I don’t think that I am competent to respond to the third question about validations, but my instinct is to lean towards “boring controllers” (that is, super-thin), with validations in the business logic. This may not be workable in all cases.

In the other thread you made a comment that I thought would be best answered here. You said:

At the moment, I’m just fiddling around with ideas, however, . . . I am a huge fan of event-sourced systems, CQRS and DDD and will happily attend all the discussions related to these architectural topics :).

I believe that the app structure I proposed would lend itself well to DDD and the gradual adoption of the more advanced concepts you mentioned. I could see the fairy tale lifecycle of a Hanami app going something like this:

• hanami new gives us a simple, straightforward app framework with sane defaults, clean and clear conventions, and security best practices built in. The UI of our app lives in app and the business logic is organized however we see fit in lib/awesome_app/– we could go all Sandi Metz or Uncle Bob crazy in here, but personally I would organize around interactors, entities, and repositories (based on ROM), similar to Hanami 1.x.

• As the app grows, we start to split the UI into slices: web for the web user app, admin for user group admin functions, an API, and maybe a BI dashboard. Business logic remains in lib/awesome_app/, just as before.

• As more features are added, complexity and coupling in the business logic begin to take a toll. We decide to break up business logic into bounded contexts (DDD, ftw!): within lib/awesome_app/ we create directories for gatekeeping for identity and authorization functions; billing; appointments; etc. Interactors, entities, and repositories are grouped into their respective bounded contexts. Calls between bounded contexts are routed through a job queue for asynchronous processing.

• When our dev teams grow large enough that we need to reduce coupling even further, we can implement event sourcing, switch from the job queue to event listeners, and maybe convert selected transactions to event stores. At this point implementing a strategy like CQRS would be straightforward. Since this code is isolated in the repositories, the change is invisible to the entities.

• When Elon Musk finally makes us an offer–and tells us how if we were real developers, we would use microservices–then his Tesla engineers will have an easy time converting our shiny Modular Monolith to his master plan (while we sip mai tais in Hawaii).

That sounds like a pretty compelling story to me! Especially compared to the massive investments companies like Gusto and Shopify have made to twist Rails internals and conventions just to get some of these same benefits. I think with Hanami 2.1, this vision could be built right in from the start!

Thank you for adding your thoughts to the conversation!

[hieuk09]

Hi @dcr8898, thanks for the discussion, I agree with a lot of things here. @swilgosz introduces me to this thread, so I would love to discuss some ideas directly here.

I definitely agree that mailers belong in app or slices, since they are UI-centric like all of the other objects in that layer. This should give them free access to the assets in the asset pipeline. If the app is using a JS front end, like React, then I would expect to follow a pattern similar to Rails, where JS code is stored within asset directories in app , and then compiled to a public directory or something similar. In this case, JS-based business logic would be stored in app or its sub-directories, and not in lib .

In Hanami 1, one of the problem we have is sharing FE components between apps, which forces us to choose one of the below 3 options:

• Use a single app for FE-related works: this sometimes feels like we packing too many responsibility into this app
• Extract FE components to separate libraries: this requires overhead in managing libraries
• Move FE to completely separate repos

We currently use a mix-and-match of those 3 options, and feel like they all have their advantages & drawbacks. I would love to know if there are any standard or recommendation from the community on this.

Middlewares, if I understand you correctly, are part of the Rack app stack, but could invoke business logic in lib in the same way that the Hanami router/actions do (since Hanami is just another Rack app).

In Hanami 1, we currently put it in /app for middlewares that are only used for that app, and /lib if they are shared. I would love to know what other people do, too.

I don’t think that I am competent to respond to the third question about validations, but my instinct is to lean towards “boring controllers” (that is, super-thin), with validations in the business logic. This may not be workable in all cases.

We’re currently doing validation at controllers layer, mostly because it’s the edge of our system, and it’s much simpler to not worry about data validation when working with business logic. This seems to be what Hanami 1 encourages, too. However, with the change of Hanami 2 that rules is not available in params, I would love to see what is the direction that the community suggests.

[dcr8898]

I just watched @timriley’s great talk at RubyConfTH 2022 announcing the debut of Hanami 2.0 and describing his current explorations with the new framework. Very exciting stuff!

One of the strengths that Tim touted for Hanami is the ability to create boundaries in your code. I think this is a huge point that deserves more exploration in the context of this discussion on the structure of projects under Hanami 2.1.

Tim used his DecafSucks project as an example to illustrate one approach to code organization in Hanami 2.0: that is, to keep all logic, including business logic, in the app directory. As far as I can see, the only code Tim placed in lib are ROM-rb relations in a persistence directory. Although DecafSucks is not yet complex enough to justify creating multiple slices, it seems likely that Tim will continue to keep all code under app as the project grows. This approach is similar to what @swilgosz proposed in his post above.

I would like to continue to advocate for a path that provides for a complete separation of business logic from UI concerns. To be clear, I am not hung up on any particular directory, just on the idea that business logic is separate and can grow/evolve along different lines than UI logic. That can be achieved in lib or some other directory (app/lib is sometimes used in Rails projects, but I think it invites some confusion).

Summary of the Slices Approach

The idea of the slices approach is that any slice may contain one or more of the following (DecafSucks does not use entities or mailers, but it seems safe to assume that they would live here too):

• Actions

• Views/Templates/Partials

• Mailers

• Entities

• Repos

• etcetera?

As Seb points out, you could organize your slices in a number of different ways, and even nest them. Slices are also composable, since you can access a slice from another slice via dependency injection. The idea, as I understand it, is that you could have slices dedicated to UI (HTTP, mailers, etc.) that call business logic located in other slices dedicated solely to that purpose.

I have never constructed an app in this way in Hanami, but this seems similar to another strategy that I have explored: component-based Rails applications (linked above). The idea in CBRA is to create a collection of Rails engines, each of which might hold any of the objects commonly found in a normal Rails app. These engines are also composable through inclusion in other engines.

Concerns

Some of my concerns with the everything-in-slices approach:

• Slices will become a muddle of misdirection, without clear indication of the purpose of each slice. This is one of the drawbacks I found with CBRA. I don’t think that nesting/namespacing will be enough to remedy this issue.

• There is no clear separation of application layers (ui, business logic, persistence, etc.). Seb proposes different nesting strategies above to address this, but I think experience will show that nesting in this way will promote indirection, resist easy composition, and prove to be inflexible in practice.

• Slices do not promote modularization of the app. I would argue (gently) that slices do not do enough to promote boundaries and provide a pathway to modularize an app–that is, to create healthy boundaries.

Other Approaches

There is a good talk from Rails Conf 2022 by Alex Evanczuk describing how Gusto uses Shopify’s Packwerk gem to separate out business logic code into discrete packages (or “packs”), which continue to live in the monolithic codebase. The gem does this through a scheme of code organization, YAML files to define interfaces, and static analysis tools to assess correct use of the interfaces.

Looking outside of Ruby World, there are other good examples of separation of business logic. For example, in the Phoenix framework for Elixir, a new Hello app has the following directory structure (showing folders only, comments added):

├── _build
├── assets
├── config
├── deps
├── lib
│   ├── hello           # Business Domain
│   └── hello_web      # UI
├── priv
└── test

The contents of the lib/hello_web/ directory make it clear that the contents are concerned only with UI:

lib/hello_web
├── controllers
├── templates
│   ├── layout
│   └── page
├── views
├── endpoint.ex
├── gettext.ex
├── router.ex
└── telemetry.ex

The strong boundaries that Packwerk creates allow teams to assume ownership over units of code, and provide a public interface for other teams to utilize those units. Well defined boundaries and interfaces also simplify testing of the system. I am no Elixir developer, but I imagine that the separation of business logic from domain logic in Phoenix provides similar benefits.

Back to Hanami

What originally drew me to this project was the fact that separation of business logic was a central design principle. Business logic lived in lib and the “app” (meaning UI functions) lived in app. As the app grew, it could be divided into multiple apps (equivalent to slices). While there was no explicit way to divide the business logic into “slices,” this felt like a great start. I was (and am) very eager to see what improvements Hanami 2 will bring.

Could you do something similar to what I propose in other frameworks? Of course (even in Rails), but it was hugely important and impactful to me that the Hanami team placed good design principles at the center of their project. Convention over configuration, right down to a well marked pathway to evolve project design from inception to maturity.

Conclusion

Circling back to my original premise, I believe that placing business logic in a separate location creates well demarcated layers and provides a rational design path for the growth of the app. New apps could start with a simplified design using just actions, entities, and views–not that different, conceptually, from Rails. As complexity grows, business logic could be more explicitly structured. I loved that Hanami 1.x provided interactors as a pathway for managing transactional business logic, and I hope that Hanami 2.1 brings them back, but simple apps could handle the same responsibilities right in the actions. Again, a simple default with a clearly marked pathway for growth and improvement.

Finally, I would also point to Shopify’s use of static analysis tools in Sorbet and Packwerk as examples. I think such tools are going to become more common in Ruby, and could become a core feature of Hanami 2.x. For example, in a Hanami app created in 1.x form and using interactors, a hanami command-line task could return all of the interactors in an app and their expected parameters–effectively documenting all of the use cases addressed by the app. As in Packwerk, another task could document all calls to interactors in the entire app–and highlight any unintended coupling by, say, identifying unwanted calls between interactors (or between domains, if Hanami 2.x provided a means of organizing business logic into domains). Really powerful and useful stuff! However, I think it would be harder to accomplish this with the everything-in-slices approach.

I would love to discuss all of this more, but I feel like I am belaboring the point. What do the core team members (and others!) think?

[yuszuv]

Hi there

first, thank you for the time and effort you all spent in developing Hanami! A watched it growing all my time as a professional developer and I’m very happy to see 2.0 released (as well as dry-rb awesomeness reaching 1.0).

And although I’m not a professional developer anymore, I’m still doing some side projects with Hanami. And it feels very powerful in it’s architectural structure (and of course make a lot of fun to code and test). And as I (as part of a team) have struggeled myself with the way you have to organize a Rails codebase, I find it super exciting that these pitfalls are discussed here. And although I’m not an expert on the design topic mentioned above, I would heavily agree with @dcr8898 that a “everything as a slice” approach could lead the path to badly designed boundaries.

From my experience in (small to medium) rails projects I was part of, IMHO it’s very crucial to separate business logic from the UI (o boy, how long it take me to distinguish the one from the other in my first Rails apps ). I think a “convention over configuration” where you are to put UI stuff to /app and business log to /lib leads a better separation just by not having to think about where to put stuff. The drawing of the initial posts and @dcr8898 arguments seem very persuasive to me.

Having said, that I’m no expert on this topic, this are only my 2 cents and I know, that the answer to how to do things right in programming is very often “it depends” But however I would really like to further follow your discussions on this topic and I’m pretty curious how the final app structure for 2.1 will look like.

Thanks, Jan

[swilgosz]

Tha knyou all for the feedback! I do agree with you, that having everything in slices could not be the best, and lead to blurred boundaries.

The naming conventions mentioned above, appending the name with _web or _api could help a bit, but still, I personally think that a clear separation of the app and business layers would be better.

I have recently published a video presenting the discussed approach, with examples, but it just serves as a purpose to show off what can be done, if for convieniency one would like to leverage the power of preconfigured slice folders, with all the autoloading and so on.

Scale your app with slices!

I have already started creating own hanami 2 showcase application for everyone to share their thoughts on, but I already think I probably won’t follow the everything-in-slices approach.

Do you think that Having a guides section with instructions to configure lib subfolder correctly for easy work with business domains (similar to slices), could be helpful?

[dcr8898]

Hi, @yuszuv. Thanks for taking the time to respond. It seems that this discussion has moved to another post. You might want to add your voice there.