A rule engine should be used when a domain model has many rules that may change over time and need to be closely managed by domain experts.
By using a rule engine, rules become much more tangible. They can be browsed, searched, reviewed, and altered in a way that facilitates their management without the complications and restrictions of a full-blown IDE.
Rule engines are often avoided by DDD-ers (Domain Driven Designers) and OOP-ers (Object Oriented Programmers) because they pull domain logic out of the model and into a rule structure, disregarding many of the core features of OOP. (Separating the rules prevents classes from encapsulating their own behaviour, and doesn't allow inheritance or any form of overriding super class behaviour.)
Drools, a popular Java rule engine, uses an OOP language (Java-like) to write the RHS (rule execution) and to some degree, the LHS (rule condition). However, the rules themselves are still very much detached from the model. The rule is more like an /if/ condition with a /then/ procedure. If the code does not belong to an extend-able class hierarchy, it is anything but OO.
Tools could be written to allow a OO model, written in a language like Java, to be indexed and allow it to be browsed and searched - many IDEs already do this to a degree. However, the challenge is that the Java language (like many other languages) does not distinguish between the concepts (the classes and properties) and the behaviours (the rules). In Java, it is particularly difficult to distinguish between a property that retrieves and stores values and a method that applies logic to the state of the model.
To combine the features of OOP with the advantages of a rule engine, the domain model needs to be stored in a more formal structure to facilitate its management. It needs to be in a structure that clearly distinguishes the concepts and their properties from the behaviour rules, while providing OOP features to them. Such a structure would need to be index-able to allow it to be browsed and searched and straight-forward enough to allow alteration of individual rules without the risk of data loss.
Does such a rule engine exist? Check back next week.