Patterns with Odoo: Humble Object

• What's the problem?
  • Bloated models
  • Tightly coupled tests
• What exactly is the Humble Object pattern?
• When does it make sense in Odoo?
• How to I apply it?
• A more complicated example with stock
  • Setup
  • The first test
  • The original code
  • Applying the pattern
    • Step 1: Figure out what it does
    • Step 2: Psuedocode a better version
    • Step 3: Extract functions
    • Step 4: Refactor
    • Step 5: Write and run tests
• A note about tools
• Takeaways

Take a look at the corresponding code for this article in the holdenrehg-samples/sample_odoo_humble_object repo.

Frameworks I've worked with typically fall into two camps with architecture.

1. You have "free for all" frameworks telling you to do mostly whatever you'd like, however you'd like. They chuck all the tools on the ground with descriptions of how each work, but they don't care if you want to build a house, a canoe, or pave a driveway. Developers usually refer to these as "unopinionated".

2. Then you have the "prefab" style frameworks where they put up the frame of your application for you. They often show you where to put certain things and often lock you in to certain tools or patterns by default. Want to use a steel frame instead of timber? Might not be a realistic option to swap that out. Developers call these ones "opinionated".

That's enough with the construction analogy. It doesn't line up perfectly, but the gist is that you have frameworks that are more like a collection of modules or libraries with sets of generalized utilities for common tasks. For example, Flask has a routing module that you explicitly use. There is no routing config file that is processed and handles the mapping for you. You import the module in python, write the endpoint function, call the function to load and parse the html, pass the data into the html, and return a response.

I don't want to get into the debate of which one of these strategies is best (surprise: the answer is "it depends" like most methodology conflicts in software).

I'm writing this all out because I wanted to talk about where Odoo falls on this spectrum. In my opinion, I think that Odoo falls pretty much as far to the "opinionated" side as possible. You have very little control or input when it comes to how views are rendered, requests are routed, how the database it interacted with, etc.

So there's obviously pros and cons there.

With this article I'm going to work through an example of implementing the humble object pattern within Odoo to solve some potential issues.

What's the problem?

There's a couple primary issues this pattern helps with:

1. Models end up huge, bloated, and difficult to work with.
2. Tests are tightly coupled with the environment and orm.

I have a habit of looking to the code around me as a baseline. This can be incredibly helpful in team environments to keep things consistent. At the same time, if there are bad choices made around you, it's easier to not think about them and just carry them into your code as well.

That's why I'm pointing out some examples of how the Odoo design choices have led to some not-ideal code. As Odoo developers adding even more modules into this environment, or potentially even into the open source project, we can work together to identify some of these issues and implement better solutions.

Bloated models

Taking a look at the core code today (currently v14) I can quickly find a few examples of massive and complicated models or sets of models.

The class Partner is 856 lines long with over 65 methods.

The class SaleOrder is 1125 lines long also about 65 methods and the longest method being 124 lines long.

Strictly using metrics like line counts is not perfect, but it's generally clear that risk goes up as certain pieces get larger, numbers of dependencies goes up, code is reaching into many other parts of the system, and responsibilities for data storage, business logic, and presentation are all intermingled together. It creates a spiderweb of complicated systems that developers struggle to change and work with.

Tightly coupled tests

We could dig into best practices of coupling and dependencies, which are very important, but honestly the biggest problem with tests tightly coupled into the framework and the database specifically, is that I cannot run my tests without running an entire instance and database.

This seems crazy to me.

If I make a my_test.py file with one standalone unit test and run python -m unittest my_test.py it takes 0.000s:

import unittest

class Test(unittest.TestCase):
    def test_sample(self):
        assert 1 + 1 == 2
                

Putting the same test into an Odoo module and running it (best case) on my laptop takes about 8 seconds. This of course is assuming you already have a demo database configured with the module already installed. I've worked on projects where about 100 tests takes nearly 5 minutes.

For medium and large sized projects this is huge hindrance to teams. It slows developers down and makes the continuous integration process more difficult.

What exactly is the Humble Object pattern?

I'm not sure the exact origins of the term "humble object" term but it was referenced in the xUnit Patterns book by Gerard Mezzaros. Also, Martin Fowler has a short write up on it.

The gist of the idea is simple.

Extract business logic code from hard to test classes and put it in easy to test classes.

Let's break it down a couple pieces of this.

By "business logic", I mean code that does some sort of non-trvial operation or calculation specific to your application. This is code that is critical for you to test. This might be a function that calculates the total revenue for a specific customer, or determines the lead time for a product, or checks the stock availability of your inventory items.

By "hard to test" I mean classes that hard to initialize. models.Model is a good example. What do I need to create an instance of a model? I need a PostgreSQL application running, an Odoo application running, a demo database, and demo data for my new object. Code that is hard to test is wrapped up in dependencies pointing to complex systems that we also need to initialize to run our code.

By "easy to test" I mean pure python functions and pure python data structures. Doesn't really get much simpler than that.

When does it make sense in Odoo?

I think about this pattern while writing methods that I end up then sticking onto the most relevant model. As soon as that method lands on the model, I again am writing tests that depend on a more-complex-than-I-would-like™ environment.

I also think about this pattern when there's some "system" that could be broken up into smaller components. Usually as Odoo developers we think about groupings of code as modules/addons. Generally that works alright as a place to draw a boundary line between things that fit together. But there are scenarios where it makes sense to draw more lines and break things up within a single module. The stock module is a great example because it has a decent amount of logic, but probably would not be worth separating out into dozens of small addons. Instead of packing everything onto a few models though, we could separate out some "sub systems" within the stock module. I'll go through an example of this.

Getting to the point, how do I apply it?

In short, use more pure python or python that doesn't depend on anything that you can't easily initialize.

A trivial example may look like:

import scanner
from odoo import fields
from odoo.models import Model

class Receipt(Model):
    _name = "receipt.receipt"

    vendor = fields.Char()
    amount = fields.Char()

    def scan(self, picture):
        receipt = scanner.scan(picture)
        self.vendor = receipt.find("Vendor")
        self.amount = receipt.find("Amount")
                

We have our small Receipt class with one method scan. Scanning will take in the picture of the receipt, pass it over to a scanner library (details of how this library would work not important) and then we extract information out of the picture to store on our object.

from odoo.tests import TransactionCase

class ReceiptTests(TransactionCase):
    def test_scan(self):
        receipt = self.env["receipt.receipt"].create({})

        assert not receipt.vendor
        assert not receipt.amount

        with open("./sample_receipt.png") as file:
            receipt.scan(file)

        assert receipt.vendor == "Home Depot"
        assert receipt.amount == 102.56
                

Now for a test, we need all the usual suspects in Odoo. The database, running instance, etc. etc. etc. Here's our problem where developers are a little frustrated with the process of running tests, there's a local database with an issue blocking them from testing their code, they need 5 different Odoo environments to test the same piece of code in each version, it's all slow, etc.

Moving to a humble object (or in this case more a humble function), we can create a little sub system for handling receipt scanning:

import scanner

def scan(picture):
    receipt = scanner.scan(picture)
    return {
        "vendor": receipt.find("Vendor"),
        "amount": receipt.find("Amount")
    }
                

from odoo import fields
from odoo.models import Model
from . import receipt_scanner

class Receipt(Model):

    ...

    def scan(self, picture):
        receipt = receipt_scanner.scan(picture)
        self.vendor = receipt["vendor"]
        self.amount = receipt["amount"]
                

It's a subtle difference, but now we have a pure function that we can test easily. There's no need to initialize the entire Odoo environment:

import unittest
import receipt_scanner

class ReceiptScannerTests(unittest.TransactionCase):
    def test_scan(self):
        with open("./sample_receipt.png") as file:
            receipt = receipt_scanner.scan()
            assert receipt["vendor"] == "Home Depot"
            assert receipt["amount"] == 102.56
                

With a setup like this we can just run the module from the command line:

.
----------------------------------------------------------------------
Ran 1 test

OK
                

A more complicated example with stock

Hopefully you've got the general idea of how this works now. I'll dig into a concrete example using the stock module.

Setup

There's not one solution to structure your modules and tests when you use this pattern, but here's what I like:

└── my_module
    ├── core
    │   ├── __init__.py
    │   └── tests
    │       └── __init__.py
    ├── controllers
    ├── helpers
    ├── models
    ├── views
    └── tests
                

You can have all your standard module scaffolding with controllers, helpers, models, views, etc. except with an added core folder. The core folder is the extracted out business logic for your application. You can think about it as these "sub systems" or domain logic. Essentially anything that you want to standalone and be testable in isolation. This means that we also need a separate test suite. Trying to merge the Odoo tests cases with these will lead to all sorts of import issues.

The first test

Let's add one test and get it running:

└── my_module
    └── core
        ├── __init__.py
        └── tests
            ├──   test_sample.py
            └── __init__.py
                

import unittest

class SampleTest(unittest.TestCase):
    def test_one_plus_plus(self):
        assert 1 + 1 == 2
                

Now run it. From the root directory of my_module you are going to execute the core package:

$ python3 -m unittest discover -s core -t .

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
                

The original code

Here's the code we're going to try to refactor away from:

Warning: In my opinion, this method is complicated at first glance and you probably don't want to take the time to figure it out. So I'm only putting it here so you understand what we're working with and then I'll break it down.

from odoo import api
from odoo.models import Model
from odoo.tools import float_compare


class Quant(Model):
    _name = "stock.quant"

    def _get_available_quantity(self, product_id, location_id, lot_id=None, package_id=None, owner_id=None, strict=False, allow_negative=False):
        """ Return the available quantity, i.e. the sum of 'quantity' minus the sum of
        'reserved_quantity', for the set of quants sharing the combination of 'product_id,
        location_id' if 'strict' is set to False or sharing the *exact same characteristics*
        otherwise.
        This method is called in the following usecases:
            - when a stock move checks its availability
            - when a stock move actually assign
            - when editing a move line, to check if the new value is forced or not
            - when validating a move line with some forced values and have to potentially unlink an
                equivalent move line in another picking
        In the two first usecases, 'strict' should be set to 'False', as we don't know what exact
        quants we'll reserve, and the characteristics are meaningless in this context.
        In the last ones, 'strict' should be set to 'True', as we work on a specific set of
        characteristics.

        :return: available quantity as a float
        """
        self = self.sudo()
        quants = self._gather(product_id, location_id, lot_id=lot_id, package_id=package_id, owner_id=owner_id, strict=strict)
        rounding = product_id.uom_id.rounding
        if product_id.tracking == 'none':
            available_quantity = sum(quants.mapped('quantity')) - sum(quants.mapped('reserved_quantity'))
            if allow_negative:
                return available_quantity
            else:
                return available_quantity if float_compare(available_quantity, 0.0, precision_rounding=rounding) >= 0.0 else 0.0
        else:
            availaible_quantities = {lot_id: 0.0 for lot_id in list(set(quants.mapped('lot_id'))) + ['untracked']}
            for quant in quants:
                if not quant.lot_id:
                    availaible_quantities['untracked'] += quant.quantity - quant.reserved_quantity
                else:
                    availaible_quantities[quant.lot_id] += quant.quantity - quant.reserved_quantity
            if allow_negative:
                return sum(availaible_quantities.values())
            else:
                return sum([available_quantity for available_quantity in availaible_quantities.values() if float_compare(available_quantity, 0, precision_rounding=rounding) > 0])
                

Applying the pattern

Step 1: Figure out what it does

After sorting through the conditionals here, this method is essentially:

1. a. Gathering a set of stock.quant objects based on the params product, location, lot, and owner.
2. b. Determining the availability quantity of those quants, which depends on if the products are tracked and if negative quantities are allowed.

Step 2: Psuedocode an ideal version

In the original code you have 8+ difference branches of if statements to follow. I rearranged those first to try to create some more readability.

if product.id.tracking == "none":
    available_quantities = [ ... ]
else:
    available_quantities = [ ... ]


if allow_negative:
    # filter_negative(available_quantities)
    ...

return sum(available_quantities)
                

So if we can get available_quantities into the same data structure despite if it's trackable or not, it simplifies. The same sort of allow_negative filter is applied either way and we can use the same summation function at the end too.

Step 3: Extract functions

Now we can start pulling out some methods into our core code. I'm going to make a core/stock_availability.py module to start.

First I want to break down exactly what's happening with the "lot trackable" scenario. The original code is creating a dictionary where the key is the lot_id and the value is the available quantity for that lot. Then all of the keys are essentially ignored and the values are pulled out into a list. I'm starting with creating a function that generates the map:

from typing import List, Dict
from functools import reduce


def filter_tracked(quant) -> bool:
    """Filter function that flags tracked quants to be filtered."""
    return quant.lot_id is not None


def filter_untracked(quant) -> bool:
    """Filter function that flags untracked quants to be filtered."""
    return quant.lot_id is None


def quantities_per_lot(quants) -> Dict[str, float]:
    """
    Generates a quantity per lot map. Example return value looks like:

        {
            "untracked": 12.5,
            "lot_1": 10.0,
            "lot_2": 11.0,
            "lot_3": 12.0,

            ...
        }
    """

    tracked_quants = list(filter(filter_tracked, quants))
    untracked_quants = list(filter(filter_untracked, quants))

    res = {
        "untracked": sum([quant.quantity - quant.reserved_quantity for quant in untracked_quants]),
    }
    for quant in tracked_quants:
        res[quant.lot_id] = res.get(quant.lot_id, 0.0) + qty_available(quant)
    return res

                

A little easier to understand:

• Takes in a list of quants.
• Splits quants into an untracked list and a tracked list. Tracked means they have a a lot_id and untracked means there is no lot_id assigned.
• A dictionary is returned containing a mapping of lot_id -> quantity available.

Now we can handle that if/else for generating available_quantities:

def availability_by_tracking(tracking, quants) -> List[float]:
    """
    Returns a list of available quantities based on a tracking status. Without
    tracking all of the quantities are summed together. With tracking the
    quantities are summed per lot.

    This does not provide mapping between the quantities and the lot numbers.
    Use the quantities_per_lot() function for that.

        availability_by_tracking("none", (stock.quant(1), stock.quant(2)))
            [56.0]

        availability_by_tracking("lot", (stock.quant(1), stock.quant(2), stock.quant(3)))
            [45.2, 34.9]
    """
    if tracking == "none":
        return [sum([quant.quantitiy - quant.reserved_quantity for quant in quants])]
    return list(quantities_per_lot(quants).values())
                

from odoo import models.Model
from odoo.addons.stock.core import stock_availability

class Quant(Model):
    def _get_available_quantity(
        self,
        product_id,
        location_id,
        lot_id=None,
        package_id=None,
        owner_id=None,
        strict=False,
        allow_negative=False,
    ):
        self = self.sudo()
        rounding = product_id.uom_id.rounding
        quants = self._gather(
            product_id,
            location_id,
            lot_id=lot_id,
            package_id=package_id,
            owner_id=owner_id,
            strict=strict,
        )

        available_quantities = stock_availability.availability_by_tracking(
            product_id.tracking, quants
        )

        if not allow_negative:
            available_quantities = filter(
                lambda qty: float_compare(qty, 0, precision_rounding=rounding) > 0,
                available_quantities,
            )

        return sum(available_quantities)

                

Step 4: Refactor

At this point, we have everything that we need to write tests and finish up, but it's a good point to refactor one more time. While writing the new code I noticed that I was repeating certain lines. For example, the calculation:

quant.quantity - quant.reserved_quantity
                

This is a pretty important line of code. It determines how the system decides what an "available" stock quantity is. This is a rule that ideally is stored in one place. Right now there's a risk that it gets changed in one calculation but not another causing a very difficult to find bug. Trust me I've seen too many of those types of bugs. So let's pull out a couple more functions for that:

from typing import List, Dict
from functools import reduce


def qty_available(quant) -> float:
    """Rule for the definition of "available quantity" based on 1 quant object."""
    return quant.quantity - quant.reserved_quantity


def sum_availability(val, quant) -> float:
    """Reducer function to sum a collection of quants."""
    return val + qty_available(quant)
                

Then update our original code:

from typing import List, Dict
from functools import reduce


def availability_by_tracking(tracking, quants) -> List[float]:
    if tracking == "none":
        return [reduce(sum_availability, quants, 0)]  # use the sum_availability reduce fn here
    ...


def quantities_per_lot(quants) -> Dict[str, float]:
    ...

    res = {
        "untracked": reduce(sum_availability, untracked_quants, 0)
    }
    for quant in tracked_quants:
        res[quant.lot_id] = res.get(quant.lot_id, 0.0) + qty_available(quant)  # use qty_available here
    return res
                

Step 5: Write and run tests

Last step, we can write tests. Now that we have a standalone, testable set of functions we can use unittest to test these directly. I was assuming the quant records are being passed around, but one way to handle that in tests is to mock the quant model with just the fields that we need as a pure python class. You'll see that below as QuantMock.

I'm not going to go through all of the tests and assertions that I did since it's really more about the workflow and setup than the actual test methods here.

import unittest
from dataclasses import dataclass
from functools import reduce
from . import stock_availability as stock


@dataclass
class QuantMock:
    quantity: float
    reserved_quantity: float
    lot_id: int = None


class TestStock(unittest.TestCase):
    def test_qty_available(self):
        quant = QuantMock(quantity=12.5, reserved_quantity=5.2)
        assert stock.qty_available(quant) == 7.3

    def test_sum_availability(self):
        quants = [
            QuantMock(quantity=11.0, reserved_quantity=2.0),
            QuantMock(quantity=11.0, reserved_quantity=3.0),
            QuantMock(quantity=11.0, reserved_quantity=4.0),
        ]
        assert reduce(stock.sum_availability, quants, 0) == 24.0

    def test_filter_tracked_quants(self):
        quants = [
            QuantMock(quantity=11.0, reserved_quantity=2.0, lot_id=2),
            QuantMock(quantity=11.0, reserved_quantity=2.0, lot_id=7),
            QuantMock(quantity=11.0, reserved_quantity=3.0, lot_id=None),
        ]

        filtered = list(filter(stock.filter_tracked, quants))
        assert len(filtered) == 2
        assert filtered == [quants[0], quants[1]]

    def test_filter_untracked_quants(self):
        quants = [
            QuantMock(quantity=11.0, reserved_quantity=2.0, lot_id=2),
            QuantMock(quantity=11.0, reserved_quantity=2.0, lot_id=7),
            QuantMock(quantity=11.0, reserved_quantity=3.0, lot_id=None),
        ]

        filtered = list(filter(stock.filter_untracked, quants))
        assert len(filtered) == 1
        assert filtered == [quants[2]]

    def test_availability_by_tracking_without_lots(self):
        quants = [
            QuantMock(quantity=11.0, reserved_quantity=2.0),
            QuantMock(quantity=11.0, reserved_quantity=3.0),
            QuantMock(quantity=11.0, reserved_quantity=4.0),
        ]

        assert stock.availability_by_tracking("none", quants) == [24.0]

    def test_availability_by_tracking_with_lots(self):
        quants = [
            QuantMock(quantity=11.0, reserved_quantity=2.0),
            QuantMock(quantity=11.0, reserved_quantity=3.0, lot_id=1),
            QuantMock(quantity=11.0, reserved_quantity=4.0, lot_id=2),
            QuantMock(quantity=11.0, reserved_quantity=5.0, lot_id=3),
            QuantMock(quantity=11.0, reserved_quantity=5.0, lot_id=3),
        ]

        quantities = stock.availability_by_tracking("lots", quants)
        assert quantities == [
            9.0,  # untracked
            8.0,  # lot_id=1
            7.0,  # lot_id=2
            12.0,  # lot_id=3
        ]

                

If you've got it all setup the way I've described above you can go into your module and run your tests directly from command line. On my machine, I'm getting tests to run in 0.027s. Imagine dealing with code as complex as stock.quant for standard Odoo tests. It would be quite a bit of setup to get unit tests like this done.

holden:$ python3 -m unittest discover -s core -t . -v

test_availability_by_tracking_with_lots (tests.test_stock.TestStock) ... ok
test_availability_by_tracking_without_lots (tests.test_stock.TestStock) ... ok
test_filter_tracked_quants (tests.test_stock.TestStock) ... ok
test_filter_untracked_quants (tests.test_stock.TestStock) ... ok
test_qty_available (tests.test_stock.TestStock) ... ok
test_sum_availability (tests.test_stock.TestStock) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.027s

OK

                

A note about tools.

I'm using unittest here with the standard test runner, but you have the freedom to use what you want. If you wanted to use pytest instead, or swap out the test runner for something like green, go for it! There's no lock in since it's just pure, simple python code.

Takeaways.

There's a ton I didn't discuss here that I've been thinking about for a while with Odoo. Some problems and some additional benefits with this type of setup, but I think overall the benefits outweigh the cons.

If enough of your code was written into more pure python, the cost saving of migrating from one major version of Odoo to another might be a big enough benefit on its own, without even thinking about the benefits to day to day developers working on these systems.

There's some work to do with integrating this into your CI pipeline or pre-commit checks, but relatively easy compared to the standard Odoo tests.

I hope this is a pattern that the Odoo community can start utilizing going forward. I'd love to talk to other developers about architecting their code so if you have any question or ideas, please reach out to me.

Best of luck coding.

- Holden