NewRelic APM on Hanami 2

I spent some time this week implementing New Relic for Hanami 2. I would not recommend it, unless New Relic decides to officially support the framework, for reasons I will get into. I do not have the luxury of choosing an APM vendor presently.

The Ruby agent has seen better days. It is a labyrinthine codebase full of legacy baggage. The documentation website is years out-of-date and full of dead links. The APIs are highly dependent on magic strings that are only ever gestured at in documentation, but never fully explained.

It took me a week to make this work when it should have been an afternoon.

Controller Instrumentation

This part doesn’t stray far from the docs so I’ll be brief

module Actions::Instrumentation
	def self.included(klass)
		klass.include NewRelic::Agent::Instrumentation::ControllerInstrumentation
		klass.extend HandlerTracing
	end

	module HanderTracing
		def method_added(method_name)
			super
			
			return unless method_name == :handler
			return if public_instance_methods.include?(:handle_with_newrelic_transaction_tracer)
			
			add_transaction_tracer :handle, params: 'args[0].params.to_h'
		end
	end
end

Since we know the public interface of every Action, I see no sense in manually tracing them. However, you can’t just trace it from the base class because redefining the method in a child class will break the method alias chain it’s doing. This still has the weakness that if you subclass a traced class you’ll break the chain.

It would be preferable to use prepend here. There are parts of New Relic’s agent that use it, but not here. Their metaprogramming doesn’t even use kwargs yet.

I tried constructing my own method tracing instead, but the process encapsulated in add_transaction_trigger is so convoluted I was unable to get Actions to show up correctly.

Dependency Instrumentation

This is the interesting part. My first inclination was to use the dry-monitor plugin and feed that into New Relic like they do with ActiveSupport::Notifications.

There are two problems with the implementation of dry-monitor in dry-system that make this not viable:

  1. monitors must declare the key when registering a handler
  2. there is no way to enumerate keys before the container is frozen

So either I had to manually manage the list of keys to trace (yuck) or I had to find a way to trace a frozen container.

Perhaps I’ll come back to it, but I decided to drop dry-monitor but take its implementation as inspiration.

I wrote an alternative Proxy object based on Dry::System::Plugins::Monitoring::Proxy

class Instrumentation::Proxy
	extend Dry::Core::Cache
	include NewRelic::Agent::MethodTracer

	def self.for(target, name:, methods: [])
		traced_methods =
			if methods.empty?
				target.public_methods - Object.public_instance_methods
			else
				methods
			end

		fetch_or_store name, methods do
			Class.new(self) do
				traced_methods.each do |method_name|
					class_eval <<~RUBY, __FILE__, __LINE__ + 1
						def #{method_name}(...)
							trace_execution_scoped(["#{name}/#{method_name}"]) do
								__getobj__.public_send(#{method_name.inspect}, ...)
							end
						end
					RUBY
				end
				
				define_singleton_method(:__traced_methods__) { traced_methods }
				define_singleton_method(:inspect) { "#<Proxy transaction=#{name.inspect}" }
				
				define_method(:class) { target.class }
			end
		end
	end
end

A class_eval was necessary because I am taking a decorator approach, add_method_tracer does not work because it can’t detect the method object from within a SimpleDelegator.

Configuration

class Hanami::Config
	class Instrumentation
		include Dry::Configurable

		setting :tracing_enabled, default: false
		setting :do_not_trace, default: []
	  
		def ignore?(key) = config.do_not_trace.any? { key.to_s.start_with?(_1) }

		private
		
		def method_missing(name, ...)
			if config.respond_to?(name)
				config.public_send(name, ...)
			else
				super
			end
		end
		
		def respond_to_missing?(name, _include_all = false)
			config.respond_to?(name) || super
		end
	end
	
	setting :instrumentation, default: Instrumentation.new, mutable: true
end

More on this later

Decoration

Next, I need a way to decorate all my container keys. I really don’t want to manage all keys by hand, so Container#decorate is out. It would be really nice to have a hook that fires after finalization but before freeze.

I decided on a Key resolver. Perhaps hooking into the component loader would have made more sense, I don’t know. I’ll see how this goes.

class Instrumentation::KeyResolver < Dry::Core::Container::Resolver
	attr_reader :slice_name
	attr_reader :config
	
	def initialize(slice, ...)
		super(...)
		@slice_name = slice.slice_name.namespace_name
		@config = slice.config.instrumentation
		@proxy = nil
	end
	
	def call(container, key)
		return super unless config.tracing_enabled
		return super if config.ignore?(key)
		
		target = super
		
		if target.singleton_class < Proxy
			target
		else
			proxy(target)
		end
	end
	
	def without_trace(container, key, &)
		method(:call).super_method.(container, key, &)
	end
	
	private
	
	def proxy(key, target)
		Proxy.for(target, name: "Custom/#{slice_name}/#{key}", methods: traceable_methods(target)).new(target)
	end
	
	def traceable_methods(target)
		if target.respond_to?(:__trace_methods__)
			target.__trace_methods__
		elsif target.class.respond_to?(:__trace_methods__)
			target.class.__trace__methods
		else
			[]
		end
	end
end

That Custom/ prefix on the name is one of those magic strings I mentioned.

I’m defining a __trace_methods__ convention because I kind of hate how New Relic pollutes all your objects with DSL.

That’s all the pieces, now to link them together.

module Instrumentation
	module ResolveWithoutTrace
		module Container
			def resolve_without_trace(key, &)
				config.resolver.without_trace(_container, key, &)
			end
		end
		
		module Slice
			def resolve_without_trace(...) = container.resolve_without_trace(...)
		end
	end
	
	def self.included(slice)
		slice.container.after(:configure) do |container|
			container.config.resolver = KeyResolver.new(slice)
			container.extend ResolveWithoutTrace::Container
			slice.extend ResolveWithoutTrace::Slice
		end
	end
end

class App < Hanami::App
	include Instrumentation

	config.instrumentation.tracing_enabled = NewRelic::Agent.config[:agent_enabled]

	config.instrumentation.do_not_trace = %w[
		actions
		logger
		inflector
		persistence
		settings
	]
end

class SCIM::App < Hanami::Slice
	include Instrumentation
	
	config.instrumentation.do_not_trace += %w[crypto]
end

Conclusion

This felt a lot harder than it should have been. Perhaps my choice of decorator pattern contributed to this, but honestly I had the most trouble getting the controller instrumentation to work. I like that the instrumentation is arms-length from the actual business objects.

This is more of a post-mortem than a recommendation. We’ll see if I stick by this in the future, but for now it appears to serve my needs.

1 Like

@alassek Thanks for sharing your experience. I totally get your frustration, which is a direct consequence of a Ruby monopoly.

Anything that we should’ve done or that we can do to make this integration less painful?

I think the structure of Hanami made this significantly easier to do, because I was able to decorate my entire application thanks to the dependency system. If New Relic had anything approaching a reasonable API this would have been easy.

I think the only thing that I would have really wanted is the ability to enumerate the keys before the container freeze. That would have made decorating them significantly easier.

That is more a conversation regarding dry-system than Hanami per se.

1 Like

@timriley This is useful input for when we’ll design the App boot lifecycle API.

1 Like

Why not hooking into Container.finalize and do it there? Production monitoring should be configured only there anyway. Knowing keys upfront would require scanning the filesystem first, which would slow down booting process and even if we did that, it would not guarantee that all the keys would be present because you can register things manually via provider files and use whatever keys you want.

before_finalize fires before keys are populated, after_finalize fires after freeze has been called.

I agree that monitoring should ideally be done here. There is no need to know the keys upfront, I just want to enumerate and decorate them before the container is frozen. That should be done as late in the process as possible.

Is there an actual reason why after_finalize needs to take place after freeze? Is it possible to call that before freeze? Or perhaps we could introduce before_freeze for this if not.

That’s too bad :confused: I think we should add a way to hook into it viabefore_freeze like you suggested and make it public API :slight_smile:

Here is a potential surgical fix for this problem: Defer Dry::System::Container freeze.

The freeze call is gated by an argument, so Hanami can defer this until after the hooks run.

This is great discussion, folks!

I’ve responded in the PR with a suggestion that I think dry-container is probably the better suited place for this. Keen for your thoughts, @alassek.

While I’m here:

From my perspective I think that running a prepared-only Hanami app (i.e. not running Hanami.boot) is a completely valid way of operating Hanami for certain circumstances (imagine e.g. a serverless function that needs to run just one narrow slice of a fuller app).

In addition, we never fully boot Hanami in dev mode, either.

This does have an impact on what hooks make the most sense for extending framework behaviour.