From Rules as Executed to Citizen Explanation

Flora Magdalena Johanna Bos, Dutch Digital Service, Ministry of Economic Affairs and Climate; Vrije Universiteit Amsterdam
Anne Schuth, Dutch Digital Service, Ministry of Economic Affairs and Climate
Jieying Chen, Vrije Universiteit Amsterdam
Conference paper, JURIX 2026, 16 October 2026
Camera-ready version, to appear in the proceedings of JURIX 2026

Flora Magdalena Johanna Bos: Work done while being an intern at the Ministry of the Interior and Kingdom Relations. Anne Schuth: Work done while at the Ministry of the Interior and Kingdom Relations.

Abstract

Governments increasingly use automated decision-making systems to apply legislation in domains such as social benefits, taxation, and licensing. While Rules as Code (RaC) approaches enable legally consistent and transparent decision-making through executable legislation, they do not automatically generate explanations that are understandable for citizens. Large language models (LLMs) can support natural-language explanation generation, but may hallucinate facts or exclude legally relevant information. This paper investigates how Graph-based LLM architectures can generate citizen-focused explanations from deterministic rule-engine traces. A rule engine evaluates synthetic citizen profiles against formalised Dutch legislation and produces execution traces, which are transformed into structured decision representations for explanation generation. The approach is evaluated using automated metrics and human annotation. Results show that explicitly exposing decisive legal conditions improves faithfulness, and that the full Graph configuration, which combines a structured decision graph with a constrained prompt, further improves alignment with the rule-execution trace. Human evaluation shows that explanations perceived as understandable by citizens are not always considered legally correct by experts. The findings suggest that structured representations of deterministic legal reasoning can support legally faithful and citizen-focused explanations of automated government decisions. The source code is published.

1 Introduction

Governments increasingly use automated decision-making systems to apply legislation in domains such as taxes, social benefits, and licensing. Such systems support consistent, at-scale decision-making, but they also increase the need for explanations that citizens can understand. Rules as Code (RaC) approaches translate legislation into machine-executable representations, enabling reproducible decision processes [1,2]. We use the term Rules as Executed for the executable rule representations that actually produce government decisions [3]. While RaC systems can support legally consistent outcomes, they do not automatically produce explanations that are accessible to non-expert citizens.

Large language models (LLMs) offer strong natural-language generation capabilities and therefore appear promising for producing citizen-friendly explanations. However, LLMs may hallucinate facts, exclude legally relevant conditions, or generate explanations that appear plausible without accurately reflecting the underlying decision process [4,5]. In legal and governmental contexts, these limitations are particularly problematic, since explanations must remain faithful to the actual reasoning behind a decision.

Retrieval-Augmented Generation (RAG) aims to improve factual reliability by incorporating external information during generation [6], and recent Graph-based and knowledge-augmented approaches extend this principle [7,8]. Within the AI&Law community, Nguyen and Satoh’s ConsRAG similarly uses aspect-based constraints in retrieval and generation, with iterative backtracking, to reduce legal hallucination [9], and Wehnert et al. combine legal knowledge graphs with textual retrieval for legal norm retrieval [10]. However, such architectures primarily address outputs grounded in textual documents or knowledge graphs, rather than in deterministic rule-execution traces produced by government decision systems, where the complete set of legally relevant facts is already available, and the challenge is one of structuring and communicating executable reasoning rather than retrieving evidence.

At the same time, explainable AI (XAI) research emphasises that explanations must be understandable to affected stakeholders, not only technically correct [11]. Within AI&Law, Zuurmond et al. demonstrate a graph-database, question-driven approach to explaining rule-based legal decision systems [12], Nouwens et al. apply case-based reasoning to decision support in Dutch administrative law [13], and Bouche-Pillon et al. bridge rule formalisation with justification-oriented reasoning for law enforcement data processing [14]. Citizen-focused legal explanations therefore involve a trade-off: they must remain aligned with the underlying legal decision process while being accessible to non-expert users.

This paper investigates how graph-structured representations of rule-engine execution traces can support LLM-generated, citizen-focused explanations of automated government decisions. The rule engine remains solely responsible for deterministic legal decision-making, the language model generates explanations from a structured representation of the decision trace rather than from open-ended legal text. In this paper, we ask the following research question: how can graph-structured representations of rule-based government decision processes be integrated with LLMs to generate citizen-focused explanations that are legally grounded and supported by correct evidence?

We develop a graph-based explanation pipeline for three Dutch legal domains: the Health Care Benefit Law [15], the Social Assistance Law [16], and the Alcohol Law [17]. To answer this research question, generated explanations are evaluated along two dimensions: legal faithfulness to the execution trace and citizen-focused comprehensibility, combining automated metrics with human validation by legal experts and citizens across five LLMs and three regulations.

This paper contributes (i) a graph-based architecture that integrates deterministic RaC execution traces with LLM-based explanation generation, (ii) an evaluation framework that verifies explanations against executable legal reasoning traces rather than only textual reference sources, and (iii) empirical evidence, across five LLMs and three regulations, of a faithfulness and accessibility trade-off relevant to the deployment of AI-generated explanations in public administration. The source code for experiments in this paper is published.¹

2 Related Work

2.1 Rules as Code and executable legislation

RaC approaches translate legislation into machine-readable formats, enabling deterministic rule execution [1,2]. Work on formal representation of norm interpretation [18] and on deriving executable services from legal sources [19] underlies the Dutch executable-legislation tradition our rule engine builds on. A recent systematic assessment of RaC languages provides a framework for situating such systems [20]. Normware approaches conceptualise computational artefacts that explicitly encode norms and governance constraints, treating explainability as an architectural property [21]. Machine-readable legislation must remain transparent and contestable, since public trust depends on citizens understanding the rules applied to them [2]. Rule-based systems provide deterministic, verifiable decision-making [22,2], but do not inherently support natural-language explanation generation, creating a gap between decision logic and citizen communication [1,23].

2.2 From rule execution to explanation generation

Computational law research shows that legal texts can be transformed into executable logic while preserving traceability [22]. Structured NLP combined with formal logic supports rule extraction and explainability [23,24]. Explanation faithfulness, however, is not guaranteed by self-consistency between predictions and explanations [25], and remains one of the central, insufficiently defined properties in NLP explainability research [26]. Within AI&Law, Prakken and Ratsma’s case-based argumentation model for explaining classifier decisions [27] and Zuurmond et al.’s human-centred, question-driven explanation method [12] are the closest published to the architecture proposed here. Górski and Ramakrishna evaluate XAI methods (e.g., LIME, SHAP) from the perspective of practising lawyers, underscoring that technical explainability does not equal usability for legal professionals [28]. Human-focused XAI research more broadly emphasises that explanations should support stakeholder understanding rather than only technical correctness [11,29].

2.3 LLM-generated legal explanations and faithfulness evaluation

Recent AI&Law work directly investigates LLMs for legal explanation and judgment. Jiang and Yang propose legal syllogism prompting for interpretable legal judgment prediction [30], and Blair-Stanek et al. investigate the limits of LLM statutory reasoning [31]. Most relevant to our contribution, Lindsay et al. propose argumentative LLMs that generate legal-entailment decisions together with explanations explicitly designed to be faithful and contestable [32]. On the evaluation side, Gultekin et al. propose a two-dimensional (truthfulness and reasoning-soundness) LLM-as-judge framework for legal question answering [33], and Sanchi and Novotná combine automatic metrics with human-grounded evaluation for LLMs in the legal domain [34]. Blair-Stanek and Van Durme further document output instability in LLM legal question answering [35]. More broadly, NLI-based methods outperform QA-, embedding-, and n-gram-based factual consistency metrics [36,37], atomic claim decomposition improves fine-grained factual verification [38], and efficient models such as MiniCheck approach larger-model hallucination-detection performance [39]. Existing metrics, however, assume textual reference sources rather than deterministic computational traces [36,40], motivating the adapted, trace-based evaluation framework used in this paper.

2.4 Knowledge representation and public-sector deployment

Structured knowledge representations improve traceability and factual grounding when combined with LLMs [41,8], building on foundational legal ontology and rule representation work such as LegalRuleML [42,43] and LKIF-Core [44]. Beyond technical faithfulness, deploying such systems in government raises questions of algorithmic transparency and citizen trust. Grimmelikhuijsen shows experimentally that transparency and explainability of automated decision-making increase perceived trustworthiness [45], a finding directly relevant to citizen-focused government explanations, while the Dutch toeslagenaffaire childcare-benefits scandal illustrates the societal cost of opaque algorithmic risk profiling [46]. These developments motivate evaluating not only whether LLM explanations are legally faithful, but whether structured, trace-grounded generation can reconcile faithfulness with citizen comprehensibility, the gap this paper addresses.

3 Dataset

Existing factual-consistency and legal NLP datasets do not contain executable decision traces [38,24], and administrative datasets (e.g., CBS microdata, EU-SILC) lack law-specific eligibility variables. We therefore construct a dedicated synthetic dataset of Dutch citizen profiles, using aggregated CBS statistics [47] to guide realistic distributions of age, income, and household composition while avoiding identifiable personal data. Profiles are generated through deterministic balancing across demographic and socio-economic categories (gender, age group, cultural background, work status, income level, household composition), combined with controlled randomisation within categories to avoid repetitive profiles. Each profile is evaluated by the rule engine to produce a legal outcome and execution trace against the formalised Health Care Benefit Law [15], Social Assistance Law [16], and Alcohol Law [17], which together form the input/output structures used throughout the evaluation. The dataset contains 200 profiles, each evaluated under all three laws; with five models and three prompting conditions this yields 9,000 explanations (3,000 per condition), plus the additional Llama 3.1:70B runs.

4 Methodology

4.1 System overview

The pipeline (Figure 1) has three stages: (1) decision generation, a deterministic rule engine evaluates each citizen profile against YAML-formalised legislation, (2) trace extraction, a structured trace records the outcome, decisive conditions, calculations, and legal references, (3) explanation generation, the trace is transformed into a structured decision representation passed to an LLM.

Three prompting conditions are compared, differing only in how much of the trace is exposed to the LLM and whether it is structured. In the Open baseline, the LLM receives the complete, unfiltered rule-engine output together with the raw citizen profile in YAML. No fact or condition is flagged as decisive, so the model itself has to identify what is legally relevant within the trace and the profile. In the Flat condition, the LLM instead receives only the pre-extracted decisive conditions, the citizen facts each condition is evaluated against, and the calculated amount, presented as an unordered list of plain-text statements. This is a subset of the information given to the Graph condition, presented without explicit relations between individual facts, conditions, and the outcome. In the Graph condition, this same information is instead encoded as a directed graph with five typed node categories: Decision (the final outcome or awarded amount), Rule (an individual legal condition evaluated by the engine), Fact (a citizen-specific data point), Calculation (an intermediate computed value, if applicable), and Law (the statutory source), connected by typed edges: has_fact (the citizen fact it is evaluated against), satisfied / failed (whether that condition was met by the given facts), and used_in (the calculation or decision it feeds into). This graph is passed to the LLM with an explicit pass/fail status for every evaluated condition, so that the legally relevant facts and their relation to the decision are made explicit, rather than left for the model to infer as in Open, or left unstructured as in Flat. These node and edge types are derived directly from the execution trace itself rather than from an external legal ontology such as LKIF-Core or LegalRuleML [44,42]. The Graph condition differs from Flat in more than structure: it lists all evaluated conditions and their thresholds rather than only the decisive condition, uses a constrained system prompt that forbids facts outside the graph, and normalises rounded monetary amounts in the output to the exact engine value. Comparing Open versus Flat therefore isolates the effect of explicitly surfacing decisive conditions, while comparing Flat versus Graph measures the effect of the full Graph configuration, not of graph structure alone.

4.2 Rule-based decision engine and graph construction

Legal decisions are produced by a deterministic rule engine executing YAML-formalised regulations, treated as the authoritative source of legal reasoning for this proof-of-concept, formalisation errors would propagate into both explanations and evaluation (see Section 7). For each profile, a minimal decision subgraph is extracted containing only the decisive conditions, relevant facts, calculation results, and legal references needed to justify the outcome, reducing irrelevant context and limiting opportunities for hallucination.

5 Evaluation

5.1 Evaluation setup

Five LLMs are evaluated across all three regulations and prompting conditions under the same settings: GPT-4o [48], Claude Haiku 4.5 [49], Llama 3.1:8B [50], Mistral:7B [51], and DeepSeek-R1:8B [52]. Llama 3.1:70B [53] is added to assess the effect of model scale independently of prompting condition. Low-temperature generation is used throughout to reduce variance. In the Graph condition, the prompt explicitly prohibits introducing facts, rules, or conclusions that are not in the graph.

5.2 Counterfactual robustness

For a subset of profiles per law and model, one decisive variable (income, age, insurance status, household composition) is modified across a legally relevant threshold, and the model is asked, as a follow-up in the same conversation, to explain the updated trace, to assess whether explanations update consistently with it.

5.3 Human annotation

A subset is annotated by three legal experts, each with more than ten years of experience in interpreting and applying Dutch administrative law, and three citizens with varied backgrounds. For each law, two fixed citizen cases are evaluated across all five models and both the Open and Graph conditions, so each annotator rates 60 explanations (360 ratings in total), independently and in randomised order. Legal experts additionally receive the execution trace and legal references and assess legal correctness, completeness, and factual consistency. Citizens assess readability, trust, and actionability. Inter-annotator agreement uses Cohen’s $\kappa$ (categorical) and Kendall’s $\tau$ [54] (ordinal). Furthermore, alignment between automated metrics and human judgement is assessed via Pearson’s $r$, Spearman’s $\rho$, and Kendall’s $\tau$ [54].

5.4 Metrics

Legal grounding and faithfulness. Following evidence that NLI-based evaluation and claim-level verification outperform document-level scoring [36,38,55], each explanation is decomposed into up to three verifiable claims derived from the execution trace: the decision outcome, the awarded amount (where applicable), and the decisive condition. Outcome and amount are checked by exact matching, the decisive condition by keyword matching. Each claim $i$ is scored as $x_i = 1$ if it is correctly represented in the explanation and $x_i = 0$ otherwise, and faithfulness is the proportion of correctly represented claims: $$\text{Faithfulness} = \frac{1}{N}\sum_{i=1}^{N} x_i, \qquad x_i \in \{0,1\},$$ with $N \in \{2,3\}$ depending on whether an awarded amount applies to the case.

Because string matching is fragile to paraphrasing, it is complemented by a model-based estimate: an NLI entailment probability per claim is computed with mDeBERTa [56,57], treating the generated explanation as the premise and the trace-derived claim statement as the hypothesis, so that a claim is credited whenever it is semantically entailed by the explanation, independently of exact wording. This entailment probability is compared against the binary string-matching faithfulness labels via ROC AUC [58], following the TRUE framework for evaluating factual-consistency metrics [36]. The faithfulness scores reported in Tables 1 and 2 use these NLI judgements, counting a claim as correctly represented when its entailment probability is at least 0.5.

Readability and contestability. Readability uses the Dutch Flesch Reading Ease (FRE) score [59,60]. Contestability scores the presence of the decisive condition, a counterfactual, and a next step ($\tfrac{1}{3}\sum e_i$, $e_i \in \{0,1\}$), reflecting calls for explanations to support contestability rather than only technical correctness [11,12].

6 Results

6.1 Performance across models and laws

Across all evaluated models, the Graph condition achieves the highest faithfulness (Table 1). Claude Haiku, Llama 3.1, and Mistral all show substantial improvements from Open to Flat and again from Flat to Graph, indicating that much of the improvement comes from explicitly exposing decisive conditions, with the Graph configuration adding a further increase. The largest gains occur for Llama 3.1 and Mistral (Open ≈0.50 → Graph ≈0.95), suggesting smaller open-source models benefit most from explicit structure. Open prompting is generally most readable and richest in counterfactual statements, while Graph explanations are less readable but more focused on the executed decision path. Scaling Llama 3.1 from 8B to 70B improves readability and slightly improves faithfulness for Open/Flat, but does not change the ranking of prompting conditions, indicating the proposed architecture contributes more than model size alone.

The effect of prompting differs by legal domain (Table 2). For the Alcohol Law, all conditions score high faithfulness, likely reflecting a relatively simple decision structure and a dataset skew toward straightforward rejections (many synthetic profiles lack a Chamber-of-Commerce registration, since profiles are drawn from general rather than horeca-specific CBS statistics). For the Health Care Benefit Law, the most complex regulation, involving multiple thresholds and a numerical calculation, faithfulness rises from 0.17 (Open) to 0.52 (Flat) to 0.89 (Graph), the largest gain observed, indicating that structured representations become more valuable as legal complexity increases. For the Social Assistance Law, most of the improvement already occurs at the Flat layer. ROC AUC values below 0.5 (Social Assistance, Open: 0.07) indicate disagreement between string matching and NLI entailment, not a separate faithfulness failure.

6.2 Example

Table 3 shows one case. Open gets the outcome and amount right but replaces the decisive conditions with plausible, unchecked reasons; Flat adds an unsupported claim about age; Graph names each evaluated condition with its threshold.

6.3 Counterfactual robustness

Across evaluated counterfactual cases (GPT-4o, Claude Haiku, Llama 3.1, Graph condition), 55.4% of explanations explicitly incorporated the modified input variable and 62.2% clearly signalled the resulting outcome change, indicating that the pipeline is sensitive to changing legal conditions but does not yet consistently communicate counterfactual reasoning to citizens.

6.4 Human annotation

Citizens generally rated explanations more positively than legal experts, especially when explanations were readable but not legally decisive. This pattern was strongest for the Health Care Benefit Law, where explanations were often perceived as understandable despite legal inaccuracies, while Social Assistance showed closer agreement between citizens and experts. Experts rated Graph explanations as more legally correct than Open explanations for the Alcohol Law (3.90 vs. 2.93 on a 0–5 scale) and the Social Assistance Law (3.60 vs. 2.70), but not for the Health Care Benefit Law (2.03 vs. 2.38), where both conditions scored low. Inter-annotator agreement was stronger for ordinal dimensions (legal correctness: Kendall’s $\tau=0.55$, $p<0.01$, comprehensibility: $\tau=0.37$, $p<0.001$) than for categorical dimensions (missing information: Cohen’s $\kappa=0.44$, actionability: $\kappa=0.01$), suggesting citizen-focused quality remains partly subjective even under structured guidelines. Automated faithfulness correlates positively with expert-rated legal correctness ($\rho=0.59$, $p<0.001$), and contestability correlates positively with citizen scores and clarity ($\rho=0.38$–$0.44$, $p<0.05$). Flesch readability correlates negatively with both legal correctness ($\rho=-0.68$, $p<0.001$) and citizen clarity ($\rho=-0.50$, $p<0.01$), indicating that easier-to-read explanations are not necessarily more correct or more useful: legally precise explanations often require more explicit detail, which can reduce readability while improving faithfulness.

7 Discussion

The results show that structuring rule-engine traces improves the faithfulness of LLM-generated explanations, with the largest improvements for regulations involving multiple conditions and calculations. Several limitations qualify this conclusion. First, synthetic profiles generated from aggregated CBS statistics [47] cannot capture the ambiguity and edge cases of real administrative data, most visibly for the Alcohol Law, where the absence of a horeca-specific population inflates the share of trivial rejection cases. Second, the framework assumes the YAML formalisation correctly implements the underlying legislation, any formalisation error propagates into both explanations and evaluation, a limitation shared with other RaC systems [2]. Third, ROC AUC scores reflect agreement between string-matching and NLI-based verification rather than an externally validated ground truth, unlike the human-annotated TRUE benchmark [36]. Fourth, the Graph condition combines graph structure with a constrained prompt and amount normalisation, so its gain over Flat cannot be attributed to structure alone; normalisation in particular favours the amount claim. Fifth, faithfulness checks only the outcome, amount, and decisive condition, and does not penalise additional unsupported statements; completeness is assessed only through the expert judgement of missing information. This is most visible for the Health Care Benefit Law, where the large gain in automated faithfulness did not carry over to expert-rated legal correctness, and experts frequently reported missing information. Sixth, we report means and standard deviations without significance tests or confidence intervals, so small differences, such as between models within the Graph condition, should not be over-interpreted. Finally, scaling to hundreds of regulations would require a law-selection layer before rule execution, for example along the lines of HippoRAG [61].

Beyond these technical limitations, the human annotation results reveal a trade-off between legal faithfulness and citizen perception: Graph explanations were more faithful and, for two of the three laws, rated more legally correct, yet citizens sometimes preferred the more elaborate Open explanations. Improving legal faithfulness alone is therefore unlikely to be sufficient. Future work should investigate explanation strategies that preserve alignment with deterministic legal reasoning while meeting citizens’ informational needs. The results should not be interpreted as evidence that such systems are ready for operational deployment without human oversight, legal validation, and governance procedures for evolving legislation [29,12].

8 Conclusion

Structuring deterministic rule-engine traces improves faithfulness relative to less constrained generation, and explicitly exposing decisive legal conditions accounts for much of this improvement, with the Graph configuration providing a further gain across five LLMs and three Dutch regulations. At the same time, explanations most closely aligned with the execution trace were not always the most readable or citizen-preferred, and explanations rated as understandable by citizens were not always judged legally correct by experts. Explanation quality in this domain therefore cannot be reduced to faithfulness or readability alone, but requires balancing legal precision, transparency, and citizen understanding.

References