Hanami persistence proposal for 2.x

Proposal
1

APP structure

app/

app/
  relations/
    coffees.rb       # TestApp::Relations::Coffees < TestApp::Relation
  repositories/
    coffee_repo.rb   # TestApp::Repositories::CoffeeRepo < TestApp::Repository
  structs/
    coffee.rb        # TestApp::Structs::Coffee < TestApp::Struct
  relation.rb        # TestApp::Relation < Hanami::Relation
  repository.rb      # TestApp::Repository < Hanami::Repository
  struct.rb          # TestApp::Struct < Hanami::Struct
db/
  migrations/
  schema.rb

slices/

slices/
  admin/
    relations/
      users.rb         # Admin::Relations::Coffees < Admin::Relation
    repositories/
      coffee_repo.rb   # Admin::Repositories::CoffeeRepo < Admin::Repository
    structs/
      coffee.rb        # Admin::Structs::Coffee < TestApp::Struct
    relation.rb        # Admin::Relation < TestApp::Relation
    repository.rb      # Admin::Repository < TestApp::Repository
    struct.rb          # Admin::Struct < TestApp::Struct
    db/
      migrations/
      schema.rb

ENV Vars & Settings

  • DATABASE_URL: App database
  • ADMIN_DATABASE_URL: Admin database
# config/settings.rb

module TestApp
  class Settings < Hanami::Settings
    setting :database do
      setting :url, constructor: Types::String
    end
  end
end

TestApp.settings.database.url

Note: My suggestion is to use database.url instead of database_url to capture the following use case.

setting :database do
  setting :host, constructor: Types::String
  setting :port, constructor: Types::String
  setting :username, constructor: Types::String
  setting :password, constructor: Types::String
end

TestApp.settings.database      # => { ... } A Hash
TestApp.settings.database.host # => "..." A String

# slices/admin/config/settings.rb

module Admin
  class Settings < TestApp::Settings
    setting :database do
      setting :url, constructor: Types::String
    end
  end
end

Admin.settings.database.url

CLI

Generators

Model

$ bundle exec hanami generate model book
      create  app/structs/book.rb
      create  app/repositories/book_repository.rb
      create  app/relations/books.rb
      create  db/migrations/20230213123250_create_books.rb

$ bundle exec hanami generate model book --slice=admin
      create  slices/admin/structs/book.rb
      create  slices/admin/repositories/book_repository.rb
      create  slices/admin/relations/books.rb
      create  slices/admin/db/migrations/20230213123250_create_books.rb

Migration

$ bundle exec hanami generate migration create_books
      create  db/migrations/20230213123250_create_books.rb

$ bundle exec hanami generate migration create_books --slice=admin
      create  slices/admin/db/migrations/20230213123250_create_books.rb

Database

This is a 2.1 proposal

$ bundle exec hanami generate database --slice=admin --url="postgresql://localhost:5432/admin"
      add     .env                            # ADMIN_DATABASE_URL="..."
      add     .env.development                # ADMIN_DATABASE_URL="..."
      add     .env.test                       # ADMIN_DATABASE_URL="..."
      add     slices/admin/config/settings.rb # Settings block to match ENV var
      create  slices/admin/db/schema.rb
      create  slices/admin/db/migrations/.keep

The option --slice is mandatory

If --url is provided, fill in all .env* files.
If --url is NOT provided, only add ADMIN_DATABASE_URL with an empty string

Database

See 1.x commands for reference

Create

$ bundle exec hanami db create # CREATES ALL DATABASES

$ bundle exec hanami db create --database=app # CREATES App (app/) DATABASE

$ bundle exec hanami db create --database=admin # CREATES Admin (slices/admin) DATABASE

Drop

$ bundle exec hanami db drop # DROPS ALL DATABASES

$ bundle exec hanami db drop --database=app # DROPS App (app/) DATABASE

$ bundle exec hanami db drop --database=admin # DROPS Admin (slices/admin) DATABASE

Migrate

$ bundle exec hanami db migrate # MIGRATES ALL DATABASES

$ bundle exec hanami db migrate --database=app # MIGRATES App (app/) DATABASE

$ bundle exec hanami db migrate --database=admin # MIGRATES Admin (slices/admin) DATABASE

Rollback

$ bundle exec hanami db rollback # ROLLS BACK App (app/) DATABASE

$ bundle exec hanami db rollback 3 # ROLLS BACK App (app/) DATABASE

$ bundle exec hanami db rollback --database=app # ROLLS BACK App (app/) DATABASE

$ bundle exec hanami db rollback 3 --database=app # ROLLS BACK App (app/) DATABASE

$ bundle exec hanami db rollback --database=admin # ROLLS BACK Admin (slices/admin) DATABASE

$ bundle exec hanami db rollback 2 --database=admin # ROLLS BACK Admin (slices/admin) DATABASE

Prepare

$ bundle exec hanami db prepare # PREPARES ALL DATABASES

$ bundle exec hanami db prepare --database=app # PREPARES App (app/) DATABASE

$ bundle exec hanami db prepare --database=admin # PREPARES Admin (slices/admin) DATABASE

Apply

$ bundle exec hanami db apply # RUNS FOR ALL DATABASES

$ bundle exec hanami db apply --database=app # RUNS FOR App (app/) DATABASE

It generates db/schema.sql file

$ bundle exec hanami db apply --database=admin # RUNS FOR Admin (slices/admin) DATABASE

It generates slices/admin/db/schema.sql file

Version

$ bundle exec hanami db version # PRINTS ALL DATABASES VERSIONS

$ bundle exec hanami db version --database=app # PRINTS App (app/) DATABASE VERSION

$ bundle exec hanami db version --database=admin # PRINTS Admin (slices/admin) DATABASE VERSION

Rake

For compatibility with Ruby hosting vendors, we need to expose a db:migrate Rake task

$ bundle exec rake db:migrate # MIGRATES ALL THE DATABASES

$ HANAMI_DATABASE=app bundle exec rake db:migrate # MIGRATES App (app/) DATABASE

$ HANAMI_DATABASE=admin bundle exec rake db:migrate # MIGRATES Admin (slices/admin) DATABASE
2

@jodosha What is your reasoning behind placing relations in the same place where you have app logic? - Relations are part of persistence layer, and repositories belong to the application layer. Relations are adapter-specific, and map to to the underlaying structure of database(s) - slices structure could be completely different than your db.

I am afraid that with this configuration we could fail into the trap of building applications based on the database schema which is one of the caveats with other frameworks. I feel like keeping db structure independent of slices is important for apps to grow without technical debt increasing too much.

1 Like
3

@swilgosz Thanks for the feedback.

Let’s split your concerns.

Relations. Where do you believe they should go?
My thinking was to keep them in app/, alongside with repos and structs. The three of them go together (IMO) and so they have the same location.

Slices. In my proposal, they will share the same App database.

But they can have their own database, so when you build your Modular Monolith, you’re already partitioning the data on a domain basis (slice). In this way you avoid data coupling.

Can you please share with me a structure that you would like to see? Including the reasons. Thanks! :pray:

1 Like
4

How do you anticipate handling DATABASE_URL in test suites?

I greatly prefer this approach (as opposed to Rails’ database.yml) but it has lead to accidental test runs against the wrong database connection, because changing your env variables for a test run is easy to forget.

My solution to this problem has been twofold: using this helper method in my persistence provider:

def database_url
  uri = target[:settings].database_url

  if Hanami.env?(:test)
    uri = URI(uri)
    uri.path = "/app_test"
    uri = uri.to_s
  end

  uri
end

clearly that won’t do for a framework feature.

The second thing I ended up doing is implementing something similar to Rails’ database protection via a system_metadata table. (This was necessary following a production incident where I accidentally ran DatabaseCleaner on our production database :flushed:)

The database protection feature is good to have as a fail-safe, perhaps something I could upstream.

5

The same way we did in 1.x: via different .env.* files.

Read more at: :point_down:

Regarding the protection of production database, we already had this in 1.x.

Quoting from the guides:

In order to preserve production data, this command can’t be run in the production environment.

Read more at: :point_down:

6

Slices:

I’m not sure, unfortunately. I got a feeling that in Hanami 2, /app and /slices are the application layer, where business logic could be put. However, persistence (relations) should be split in case we have different databases - not per slice but allowing to pull data from multiple DBs.

AFAIK one of ROM’s powers is allowing such cross-DB, and cross-adapter even data fetching.

Relations

Having the above in mind, this is why I’m more into keeping persistence (Relations) in the /lib folder. I’m not too strict about it, just thinking loudly.

When you have relations (that correspond to your DB structures), then you can implement repositories, and app-level structs even in the /app or /slices.

To summarize, I’m fine with having repositories and App-specific structs in the app and slices, just relations seem to be sth different to me.

Proposal:

lib
  |-- persistence
    |-- DB1
      |-- relations
    |-- DB 2
      |-- relations 
    |-- File-based DB
    |-- CSV DB
    |-- HTTP based DB... 
  |-- ...
app
  |-- repositories # (common classes)
  |-- structs
  |-- ...
slices
  |-- admin
    |-- repositories # figures out which relations to use?
    |-- queries # objects with advanced queries, across multiple dbs?
    |-- structs

Please forgive me if in my reasoning there is a missing concept of ROM possibilities and assumptions, I’m still ActiveRecord-ish in thinking, don’t feel fluent in understanding all these concepts just yet, for very large or complex apps/domains.

7

So I’ll comment on this in full soon enough, but in the meantime, I did want to point out one thing.

Over in my example app I’ve just moved my relation classes into app/, as a way to start getting a feel for the structure @jodosha has outlined above: Move relations to app/ by timriley · Pull Request #17 · decafsucks/decafsucks · GitHub

This change required I change my rom auto-registration setup to the following:

    rom_config.auto_registration(
      target.root.join("app"),
      namespace: Hanami.app.namespace.to_s
    )

With target.root.join("app") being given as the auto_registration directory, it means that rom will also look there for mappers/ and commands/ subdirectories, as well as the expected relations/.

Now defining custom mappers and commands is fairly advanced rom usage, but either way, I don’t think we’d want to see these defined in top-level directories under app. Those directories feel much too prominent for what are otherwise fairly niche concerns.

In addition, there’s a fair chance from naming collisions here too: “mappers” and “commands” are fairly common terms and there’s a non-zero chance that a user will try and create directories like those as the namespace for unrelated objects.

I tried passing this option to rom’s auto_registration method:

component_dirs: {
  relations: :relations
}

As well as this:

component_dirs: {
  relations: relations,
  mappers: nil,
  commands: nil,
}

But both of these failed because rom’s auto-registration setup expects these to be defined at all times.

I think this points to the need for rom’s auto-registration setup to be a little more flexible. Ideally I think we’d want the following:

  1. Ability to auto-register relations only in with one given root dir (app/) and namespace (e.g. "AppNamespace::Relations")
  2. And then to separately auto-register mappers and commands with different given root dir and namespace

Implicit in the above is that any for any single rom auto_registration call, it should be possible to disable registration for particular component types, such as mappers and commends.

Then after this, as a separate matter, we’d want to think about how whether we want to support default auto-registration for rom mappers and commands within Hanami apps, and if so, what directories they should use.

I suppose an alternative for Hanami’s rom setup could be to avoid rom’s auto_register entirely, and instead find and manually register all the rom components. But it’d probably be nice if we could make rom’s auto_register more flexible anyway :slight_smile:

cc @solnic since I think these could be useful things to solve with the upcoming version of rom.

8

I need to revisit this because better auto-registration was one of the reasons of the setup overhaul in rom 6.0. I’ll get back to you once I confirm how this could be arranged in rom 6.0.

9

@timriley I am wondering, wouldn’t it be way easier if you would have:

app/persistence/relations
app/persistence/mappers
app/persistence/commands
app/persistence/changesets

With the Peristence namespace in place?

I agree, that naming conflicts may be a thing. I often used Commands namespace in my programming style.

10

Like @swilgosz, I don’t feel fluent enough in ROM concepts and usage to comment in-depth, but I would like to argue strongly for a pathway that allows business/domain logic to be segregated from from the UI–and even from the framework itself.

I applaud the decision not to have a default slice in new apps. This simplifies the early going, while providing a path to more complex apps that may be composed of many slices. However, in my view, these slices are all part of the framework/UI layer: actions, mailers, views (and related objects), templates, etc. These all feel like they belong together, organized neatly with slices. It feels forced to me to say that domain logic could just be placed in its own slice or slices (presumably without the UI components mentioned above).

To me, the greatest beauty of Hanami 1.0 was the respect it gave to developers to organize their domain logic as they saw fit, and the courtesy to provide a place to put that code. It also provided a sane default for domain logic in Interactors and Entities (a pattern that I still favor), but did not force this on anyone.

I don’t want to get too hung up on the specific folder name or location, but I am really hoping that Hanami 2.1 will provide a safe, separate area for developers to grow their domain code as independently from the framework as they may wish. This could be in /lib/TestApp/ or /app/lib/ or /app/TestApp/ or wherever, but it’s presence alone would be the single greatest differentiator from existing approaches in Ruby.

As far as persistence goes, I also agree with @swilgosz:

I am afraid that with this configuration we could fail into the trap of building applications based on the database schema which is one of the caveats with other frameworks. I feel like keeping db structure independent of slices is important for apps to grow without technical debt increasing too much.

While the persistence code artifacts (repositories, relations, etc.) are obviously part of the framework, they are consumed by the domain logic and therefore should be organized alongside the domain logic as closely as possible–and ideally in a way that would allow modularization of the persistence artifacts along with the domain objects they serve. I don’t think it is correct to assume that domain logic will tend to be modularized along the same lines as slices will be. Rather, I would argue that domain logic and UI exist at two different levels and should be free to evolve independently.

Going back to the absence of slices in starter Hanami 2.0 apps, I wonder if it would be possible to keep persistence logic in the app folder for simple apps (as @timriley is doing in decaf sucks). This would allow for fast and easy development new apps, but at the same time would probably not be a strong enough differentiator to win Hanami followers away from Rails. There would have to be some pathway for the persistence logic to move elsewhere alongside the domain logic (once that is fleshed out). Hanami 1 offered something truly novel by providing a pathway to organize, segregate, and modularize domain logic, and I am really hoping that Hanami 2 does too.

I feel that this is what is really missing in the Ruby ecosystem: a framework that encourages modularity of domain logic. Roda and Sinatra are agnostic, and Rails orthodoxy discourages abstractions not included in the framework, leading to the proliferation of poorly defined strategies like service objects. I believe Hanami 2.1 has the opportunity to stake out completely new ground in the Ruby world.

For example, I am no expert in Elixir or Phoenix either, but I appreciate their mantra that “Phoenix is not your app.” See this blog post, for example (citing a conference talk from a Phoenix core team-member):

Where to put business logic?
Phoenix is not your application, so the “application” gets moved outside of the web folder. It rests in the lib/appname of your app folder, and it is what would be called a “model” in other frameworks.
Here are two things that you most likely will include there:

  • Contexts. Basically, Elixir modules that host business logic. This is where you will put stuff that interacts with the database, for example.
  • Schemas. They map data from the database into Elixir structs and vice versa.

By providing a similar, well thought-out solution to this problem in Ruby, I believe that Hanami 2.1 could stake out a strong claim as the framework for professionals concerned with growing maintainable apps “from start-up to IPO” (to steal a phrase)–a story that would have immediately appeal to product developers/consultants and all of those currently struggling with large Rails apps.