Add Validation extension for input validation with Dry Validation

[aaronmallen]

Summary

Adds a new Validation extension that brings automatic input validation to operations using Dry Validation. Following the Hanami controller pattern, operations can define validation rules via contract, schema or params methods, all of which end up creating an internal contract class, which is later initialised and used to validate/coerce inputs before any operation logic executes. This provides fail-fast validation with detailed error messages.

Motivation

When building operations, input validation is a common requirement. Currently, users need to manually validate inputs within their operation methods, leading to boilerplate and inconsistent error handling. This extension provides a declarative way to define input contracts that integrate seamlessly with the operation's existing failure handling, inspired by Hanami's controller actions.

Usage

Basic params validation

require "dry/operation/extensions/validation"

class CreateUser < Dry::Operation
  include Dry::Operation::Extensions::Validation

  params do
    required(:name).filled(:string)
    required(:email).filled(:string)
    optional(:age).maybe(:integer)
  end

  def call(input)
    # input is already validated and coerced
    user = step create_user(input)
    step notify(user)
    user
  end
end

# Success case - input is validated and coerced
CreateUser.new.call(name: "Alice", email: "alice@example.com", age: "25")
# => Success(user) with age coerced to integer 25

# Failure case - validation errors returned before operation executes
CreateUser.new.call(name: "", email: "invalid")
# => Failure[:invalid_params, {name: ["must be filled"]}]

Reusable Params classes

class UserContract < Dry::Validation::Contract
  params do
    required(:name).filled(:string)
    required(:email).filled(:string)
  end
end

class CreateUser < Dry::Operation
  include Dry::Operation::Extensions::Params

  contract UserContract

  def call(input)
    # ...
  end
end

class UpdateUser < Dry::Operation
  include Dry::Operation::Extensions::Params

  params UserContract  # Reuse the same validation rules

  def call(input)
    # ...
  end
end

Custom validation rules with contract

class CreateUser < Dry::Operation
  include Dry::Operation::Extensions::Params

  contract do
    params do
      required(:name).filled(:string)
      required(:age).filled(:integer)
    end

    rule(:age) do
      key.failure("must be 18 or older") if value < 18
    end
  end

  def call(input)
    # ...
  end
end

Other details

• Uses a user-supplied instance-level @contract, if available. This allows for contracts to be injected as regular dependencies, and works nicely with Hanami's Deps mixing.
• Will validate either keyword args or the first positional arg. Keyword args are used if no positional args exist. Otherwise the first positional arg is used.
• When validation is applied to keyword args, pass through any named keyword args to the original method, even if they do not appear within the validated input. This ensures we still meet user intentions about method args that may be used separately to the validated payload.

[alassek]

I think this is a cool idea, but considering how similar it appears to the Action method, don't you think it should work the same?

Specifically, there are two things Action can do that this does not:

Re-use an existing Schema:

class CreateUser < Operation
  include Dry::Operation::Extensions::Params

  params UserSchema
end

Declare a Validation contract:

class CreateUser < Operation
  include Dry::Operation::Extensions::Params

  contract do
    params do
      # ...etc
    end
  end
end

[aaronmallen]

@alassek accepting a class as an argument is straightforward enough I like that idea. But maybe I'm misunderstanding the difference between a schema and a contract? Aren't we explicitly using the contract functionality here?

[alassek]

@aaronmallen well no, you're instantiating a Dry::Schema so it's not a Contract. Different things. See Hanami::Action::Validatable.

Dry::Schema is stateless, and its validation rules apply to single keys only.

Dry::Validation is stateful (supports injected deps), includes a Dry::Schema inside itself, and layers Rule processing on top. Validation rules can consider multiple keys at once when defining errors.

[aaronmallen]

@alassek I guess the question becomes Do we wrap the entire functionality behind the dependency on dry validation or do we support contracts in a follow up PR and extend this extensions functionality that is wrapped in the dependency of dry validation?

[alassek]

@aaronmallen well that is a good question that we should probably discuss with the others. My preference would be to always return a consistent result type, so that it's a Dry::Validation::Result regardless. Although off the top of my head I don't recall if there's an interface difference between them.

[paul]

This is cool! I did something similar as an addon for Dry::Transaction.

Instead of just params, I wrapped it in a validate block (with a nested params), so that I could have access to the full Dry::Validation rule DSL, not just a Schema.

An example of it in use:

class VerifyUserToken
  include ApplicationTransaction

  validate do
    params do
      required(:user_token).filled(type?: UserToken)
      required(:session).filled(type?: ActionDispatch::Session)
    end

    rule(:user_token) do
      key.failure("expired token") unless value.issued_at.after?(1.hour.ago)
    end

    rule(:user_token, :session) do
      session_key = values[:session][session_key_name]
      token_key = values[:user_token].session_key

      if session_key.blank? || !ActiveSupport::SecurityUtils.secure_compare(session_key, token_key)
        key.failure("session mismatch")
      end
    end
  end
end

Just something to consider.

[aaronmallen]

@alassek @timriley this is ready for re-review at your earliest convenience. I'm unsure what the build failures are about they seem totally unrelated to my changes.

@paul I didn't follow your example to the letter but I think you'll find the implementation very similar.

[waiting-for-dev]

Hey folks, unfortunately I haven’t been closely following Hanami’s direction lately. The extension looks great, and I definitely think that returning a Result::Failure type is the right approach here as we're making sanitization a step in the operation class.
That said, I do wonder if this might break separation of concerns. I understand it can be very convenient in simple cases, but as I see it, params should already be sanitized before reaching the domain layer; that’s one of the few responsibilities of the controller layer. This also improves reusability if you need to call another business transaction with the same params, and it makes testing in isolation much easier.
If we decide to go with this approach, I think it would also be good to allow mapping a schema to a specific transaction method, rather than applying it to all methods defined in the class.

[aaronmallen]

@waiting-for-dev IMO this doesn't really break separation of concerns for two reasons:

1. It's an opt in extension
2. It doesn't assume dry-operation is only used in a hanami application

[wilsonsilva]

I use dry-operation and dry-validation extensively with Rails, and I considered creating an extension like this.

[waiting-for-dev]

@waiting-for-dev IMO this doesn't really break separation of concerns for two reasons:

1. It's an opt in extension
2. It doesn't assume dry-operation is only used in a hanami application

Agree about 1. About 2, I think it's really the same case wherever it's used (e.g. Rails) 🙂 Eager to hear @timriley 's thoughts!

[paul]

While I recognize it is likely a gross violation of separation of concerns, in practice I found the implementation of it on Transactions to be wildly successful.

First of all, it provided almost a "type signature" that described what kwargs a transaction expected, the types of values, and the state it expected them in. Second is that we exposed the validation defined by the DSL as a class method on the Transaction. This allowed external callers decide if they should invoke the Transaction, and if a validation failure should be handled or ignored.

To provide an example, let me first describe part of our domain. The application is a messaging app, like the gmail UI, which had "Conversations" that could be "open" or "closed". They could be opened or closed manually by the user, but also as a side effect from a dozen other operations, like a new incoming message would reopen a closed conversation but not try to open an already open one.

To perform the OpenConversation task, we had a Transaction that would validate that the conversation was currently closed:

class OpenConversation
  include ApplicationTransaction

  validate do
    params do
      required(:conversation).filled
      required(:user).filled
    end

    rule(:conversation) do
      key.failure("is already open") if value.open?
    end
  end

  step :create_open_event
  step :transition_the_state_machine
  step :run_any_callbacks
  # ...

One of the step DSL methods we added was maybe, which would first "pre-flight check" the transaction before calling it, and not fail if that validation failed. So if we want an incoming message can ensure a closed conversation is opened, but not fail if its already open, then the code reads really nice:

class ReceiveMessage
  include ApplicationTransaction

  validate do ... end

  # Steps to parse the input, create a Message object, locate/create the Conversation, etc...
  maybe OpenConversation
  # More steps
end

The maybe step is conceptually equivalent to:

result = OpenConversation.validator.call(input)
result.failure? ? Success(input) : OpenConversation.new.call(input)

It was also beneficial for background jobs:

• Before enqueuing, run the validation and don't enqueue the job if it fails. This prevents enqueuing jobs that would just immediately fail anyways.
• When performing the job, run the validation. This was helpful like if that OpenConversation got enqueued several times, the first run would open it, then all the others would fail that validation and not try to keep opening it.

Overall, while yes it was probably a violation of separation of concerns, the pragmatic utility and convenience we got more than made up for it.

[alassek]

@waiting-for-dev I do wonder if this might break separation of concerns. I understand it can be very convenient in simple cases, but as I see it, params should already be sanitized before reaching the domain layer; that’s one of the few responsibilities of the controller layer. This also improves reusability if you need to call another business transaction with the same params, and it makes testing in isolation much easier.

I agree with you 100% that your user-input should always be sanitized at the outer boundary, but data validation needs to happen at all layers of an application, and that validation should become more constrained the deeper into the system it flows.

I've give you an illustrative example: consider an OAuth token endpoint. The different combinations of parameters it must accept is very broad; but when dispatching these requests to different Operations, you'll need much more constrained validation of params for each use-case. This kind of situation is when I have resorted to using Schema and Contracts to validate the public interface of Operations.

class Token < Action
  include Deps["authn.client_secret", "authn.client_assertion"]

  params do
    required(:grant_type).filled(:string)
    required(:client_id).filled(:string)
    required(:resource).value(T::Coercible::URI)
    optional(:scope).filled(:string)
    optional(:client_secret).filled(:string)
    optional(:client_assertion).filled(:string)
    optional(:client_assertion_type).filled(:string)
  end

  def handle(req, res)
    authn_result = Failure(:unauthorized)

    if req.params in { client_assertion: }
      authn_result = client_assertion.(req.params)
    elsif req.params in { client_secret: }
      authn_resuilt = client_secret.(req.params)
    end

    halt :unauthorized unless authn_result.success?

    # etc
  end
end

@waiting-for-dev If we decide to go with this approach, I think it would also be good to allow mapping a schema to a specific transaction method, rather than applying it to all methods defined in the class.

I would suggest an alternative approach: it should only apply to the public interface #call as a pre-validation step and then the rest of the logic should work normally.

But in addition to this, dry-operation should follow the do-notation convention that dry-monds uses of calling #to_monad on step input (if it exists). This would allow you to use a Schema/Contract as a step itself at any point of the Operation.

[timriley]

Thanks putting this together, @aaronmallen, and thanks to everyone for all the insightful discussion above!

I'm in favour of a feature like this. I think it'll be really useful and I expect a lot of users will want to adopt this. I've used validation contracts in many of my operation objects and I think there's good reason to have them there.

I haven't had the time to look closely at the code yet; I've been focused on getting Hanami 2.3 shipped next week and this hasn't left me with any spare time. I'd like to have a closer look at the design and the code before we figure out how to merge this. I agree with @alassek that there may be alternative implementations we want to consider.

Thanks again for your input and your patience, folks. Once we do get this in, it'll be a really nice enhancement for our users :)

[waiting-for-dev]

Thanks for the feedback, folks. I agree this is a great extension 🙏

[aaronmallen]

@waiting-for-dev any idears as to why I'm having build failures for the active_record extension here?

[waiting-for-dev]

@waiting-for-dev any idears as to why I'm having build failures for the active_record extension here?

@aaronmallen, I think that if you rebase from master everything will be green now 🙂 @timriley fixed it in #35

[wilsonsilva]

The tests are green. What is missing to complete this PR?

[timriley]

Final updates:

• Removed the ability to pass a contract class to .schema and .params. We already accept this via .contract, it would be confusing to do the same on these other methods, especially since they are intended to operate within a contract, rather than defining a contract as a whole.
• Tidied API docs
• Remove lots of redundant unit tests
• Added unit tests for our named ValidationStep ancestors
• Updated this PR description so it accurately describes the feature in its current state

With that, we can merge! Thank you so much for doing this, @aaronmallen, this is going to be a great feature and I also had a lot of fun working on this code; thanks for being cool with me doing that too :)

[timriley]

Whoops, I didn't update the user docs. I'll do that now.

[timriley]

Done in a19d553

[timriley]

FYI, I've just pushed #47, where I'm proposing we simplify our handling of the method arguments around validation input.