starMC: an automata based CTL* model checker

Petri nets

A place-transition (P/T) Petri net N is defined (Murata, 1989) as a tuple $N=⟨P,T,A,W⟩$, where P is the set of places, T is the set of transitions, $A\subseteq (P\times T)\cup (T\times P)$ is the set of arcs, $W:A\to {\mathbb{N}}_{\ge 1}$ is the arc weight function. A Petri net system is the pair $NS=⟨N,m_{init}⟩$ where $m_{init}:P\to \mathbb{N}$ is the initial marking. Markings represent states of the system, i.e. assignments of tokens to places. A transition t ∈ T is enabled if and only if all input places p of t contain at least W(p, t) tokens. The firing of t removes W(p, t) from all input places p and adds W(t, p′) to each output place p′. Notation $m\stackrel{t}{\to }t\mathrm{\prime }$ indicates the firing of t in marking m, which leads to marking m′. A firing sequence σ for marking m is a sequence of transitions t₀, t₁,… for which it exists a sequence of markings m₀, m₁,… such that m = m₀ and, $\forall i\ge 0:m_i\stackrel{t}{\to }{m}_{i+1}$. Accordingly, the sequence of markings π = m₀, m₁,… identifies a path of $NS$ starting in m₀, with π[i] = m_i. We indicate with π[i··] the suffix of π starting in m_i. We define ${\mathcal{P}}_M^{\mathrm{\infty }}(m)$ as the set of all infinite paths starting in m. From now on we shall interchangeably use the terms “marking” and “state”. The reachability set (RS) of a Petri net system $NS$ is the set of all markings reachable from $m_{init}$, $RS(NS)=\{m|{\mathrm{\exists }}_{s\in T\ast }{m}_{init}\stackrel{\sigma }{\to }m\}.$ A reachability graph (RG) of a PN is a tuple $RG(NS)=⟨RS,\mathcal{E},m_{init}⟩$ where: $RS$ is the set of all markings reachable from $m_{init}$, $\mathcal{E}=\{(m,t,m^{\mathrm{\prime }})\in RS\times T\times RS|m,m^{\mathrm{\prime }}\in S,t\in T,m\stackrel{t}{\to }{m}^{\mathrm{\prime }}\}$ is the set of state transitions. If there is no risk of confusion, the $NS$ indication is usually dropped and we write $RS$ and $RG$.

Figure 1A shows a Petri net with 3 places and 4 transitions, the RS built from the initial state $m_{init}$ = m₁, in Fig. 1B, and the corresponding RG, in Fig. 1C.

A colored Petri net is a P/T net in which tokens have identities, to allow for a parametric and more compact description of systems. Each colored Petri net can be unfolded into a P/T net of isomorphic RG.

Decision diagrams for state space representation

Decision Diagrams (DD) are a well-known data structure to efficiently encode functions as well as large sets of structured data through their characteristic function. A Binary DD (BDD) encodes a boolean function and a Multivalued DD (MDD) encodes an integer function. A MDD can be used to encode a RS by associating each place of the net to a level in the MDD and ensuring that each path in the MDD corresponds to a reachable state (and vice-versa). Given the example Petri net of Fig. 1A, the MDD of the initial state m₁ is depicted in Fig. 2A, and the one of RS in Fig. 2D. DD levels encode the token counts of each Petri net place, and appear in the order P2, P1, P0 (top-down). So the rightmost path in the MDD of RS corresponds to the reachable state [m(P2) = 1, m(P1) = 0, m(P0) = 0], or [1, 0, 0] for short. The depicted MDDs are fully-reduced (when all edges out of a node lead to the same down node, the node is removed and by-passed), so the leftmost path of this MDD encodes both [0, 0, 0] (no token in any place) and [0, 0, 1] (one token in P0).

A DD encoding a relation function is called matrix diagrams (M×D). These kind of DDs are used in starMC to represent transitions among states due to the firing of Petri net transitions. An M×D has twice the levels of an MDD, and each pair of levels (also called the unprimed and primed levels for a variable) encodes the before/after relations of a variable (i.e. a place of a Petri net or a location of an automaton). Each of the four M×Ds in Fig. 2B encodes the transformation performed by a single transition of the example net while the M×D in Fig. 2D is the Next-State-Function (NSF), the union of all the single transition M×Ds. The depicted M×D are identity-reduced, i.e. skipping a level pair means that the encoded before-after relation is the identity. The image of this M×D applied to an MDD encoding a state m returns an MDD with all states that can be reached from m by the firing of a single transition. Its fixed point application results in the MDD of the RS. In practice the more efficient saturation technique (Ciardo, Lüttgen & Siminiceanu, 2001), also based on the NSF, is usually preferred. Note that usually the M×D of the RG is never built and stored, as any info in the RG can be retrieved through the DDs of NSF and RS.

In Fig. 2 places are allocated to MDD (and M×D) levels in alphabetical order. It is well known that the DD variable order (the order in which variables are assigned to levels) may greatly influence the size of the resulting DD. The interested reader may find in Amparore, Donatelli & Ciardo (2020a) a recent study on various variable order heuristics that have been successfully applied to encode state spaces of Petri nets.

The temporal logic CTL ${}^{\ast }$

CTL ${}^{\ast }$ (Emerson & Halpern, 1983, 1986) is a temporal logic in which path operators can be quantified or not, and it therefore allows to freely mix linear and branching reasoning. It subsumes the linear logic LTL (Pnueli, 1977) and the branching logic CTL (Clarke & Emerson, 1981), that are known to have a different expressive power, although with non-empty intersection.

Definition 1 (CTL ${}^{\mathbf{\ast }}$ syntax). CTL $\ast$ formulae are the formulae ${}^{\psi }$ inductively defined by:

$$\mathrm{\Psi }::=a|\mathrm{\Psi }\wedge \mathrm{\Psi }|\mathrm{\neg }\mathrm{\Psi }|E\varphi |A\varphi$$

$$\varphi ::=\mathrm{\Psi }|\varphi \wedge \varphi |\mathrm{\neg }\varphi |X\varphi |\varphi U\varphi$$where a ∈ AP and AP is a set of atomic propositions. The rules ${}^{\psi }$ and ϕ define state and path formulae, respectively.

Note that in principle just one type of quantification is needed, as $A$ϕ = ⫬ $E$⫬ ϕ. The BNF grammar of CTL ${}^{\ast }$ is ambiguous, as boolean expressions appear both as state and path formulae. For instance, the expression a₁ ∧ a₂, with a₁, a₂ ∈ AP, is both a state formula (rule ${}^{\psi }$ ∧ ${}^{\psi }$) and a path formula (rule ϕ ∧ ϕ where ϕ ⇝ ${}^{\psi }$ and ${}^{\psi }$ ⇝ a).

The satisfaction relation for a CTL ${}^{\ast }$ formula is given over a Kripke model M = (S, E, L) with an associated set AP of atomic propositions, where S is a finite and non-empty set of states, $E:S\to 2^S$ is the total successor function, $L:S\to 2^{AP}$ is a labelling function, and L(s) is the set of atomic propositions that holds in s. The set of paths starting in s, ${\mathcal{P}}_M^{\mathrm{\infty }}(s)$ is defined as for Petri nets. It is straightforward to map a reachability graph into a Kripke model.

Definition 2 (Kripke model of an RG). The Kripke model of a reachability graph $RG=⟨RS,\mathcal{E},m_{init}⟩$ is the Kripke model $M(RG)=⟨RS,{\mathcal{E}}^{\prime },L⟩$, with ${\mathcal{E}}^{\prime }=\mathcal{E}\cup \{(m,m):\mathrm{\forall }deadlockstatesm\in RS\}$ being the stuttering of $\mathcal{E}$, and L a labelling function defined over the markings.

Stuttering of deadlock states ensures that the paths in the RG are infinite, as required by a Kripke model. Figure 1D shows the Kripke model derived from the RG in Fig. 1C when considering a set of atomic propositions AP = {β: (#P1 > 0 or #P2 > 0)} and the labelling function depicted next to the states. The two states s₃ and s₄ are stuttered, since they correspond to the deadlock markings m₃ and m₄.

Definition 3 (CTL ${}^{\mathbf{\ast }}$ semantics). The satisfaction relation of CTL ${}^{\ast }$ state formulae is defined by:

$$\begin{array}{ccc}s\models a& \mathrm{i}\mathrm{f}\mathrm{f}& a\in L(s)\\ s\models {\mathrm{\Psi }}_1\wedge {\mathrm{\Psi }}_2& \mathrm{i}\mathrm{f}\mathrm{f}& (s\models {\mathrm{\Psi }}_1)\mathrm{a}\mathrm{n}\mathrm{d}(s\models {\mathrm{\Psi }}_2)\\ s\models \mathrm{\neg }\mathrm{\Psi }& \mathrm{i}\mathrm{f}\mathrm{f}& \mathrm{n}\mathrm{o}\mathrm{t}s\models \mathrm{\Psi }\\ s\models A\varphi & \mathrm{i}\mathrm{f}\mathrm{f}& \pi \models \varphi ,\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l}\pi \in {\mathcal{P}}_M^{\mathrm{\infty }}(s)\end{array}$$

while the satisfaction relation of path formulae for a path π is:

$$\begin{array}{ccc}\pi \models \mathrm{\Psi }& \mathrm{i}\mathrm{f}\mathrm{f}& \pi [0]\models \mathrm{\Psi }\\ \pi \models {\varphi }_1\wedge {\varphi }_2& \mathrm{i}\mathrm{f}\mathrm{f}& (\pi \models {\varphi }_1)\mathrm{a}\mathrm{n}\mathrm{d}(\pi \models {\varphi }_2)\\ \pi \models \mathrm{\neg }\varphi & \mathrm{i}\mathrm{f}\mathrm{f}& \mathrm{n}\mathrm{o}\mathrm{t}\pi \models \varphi \\ \pi \models X\varphi & \mathrm{i}\mathrm{f}\mathrm{f}& \pi [1\cdots ]\models \varphi \\ \pi \models {\varphi }_{1}U{\varphi }_2& \mathrm{i}\mathrm{f}\mathrm{f}& \mathrm{\exists }j\ge 0:(\pi [j\cdots ]\models {\varphi }_2\wedge \\ & & (\mathrm{\forall }0\le k<j:\pi [k\cdots ]\models {\varphi }_1))\\ & & \end{array}$$

A CTL ${}^{\ast }$ formula in which all path operators are prefixed by a quantifier is also a CTL formula. To express the relation between CTL ${}^{\ast }$ and LTL it is convenient to introduce the following definitions:

Definition 4 (LTL formula). An LTL formula φ is inductively defined by:

$$\phi ::=a|\phi \wedge \phi |\mathrm{\neg }\phi |X\phi |\phi U\phi$$

It is usually assumed that path formulae over a Kripke model are “implicitly universally quantified”. To avoid any confusion we assume that the quantifier is explicitly indicated.

Definition 5 (Quantified LTL formula). Given an LTL formula φ a quantified LTL formulae ${}^{\psi }$ is defined by ${}^{\psi }$ :: = $E$φ | $A$φ

The GreatSPN tool

GreatSPN (Amparore et al., 2016; Amparore & Donatelli, 2018b) is a collection of Petri nets tools that are integrated in a common framework, with a unified graphical user interface (Amparore, 2014). Initially developed for performance evaluation of stochastic Petri nets, the tool was enriched over the years with algorithms for qualitative analysis.

From the graphical interface it is possible to draw a Petri net (colored or P/T), to compose Petri nets, to play the token game to familiarize with the net behaviour, and to verify the model using techniques specific to Petri nets (structural properties like P- and T-semiflows) or to check standard Petri net properties (like absence of deadlocks, boundedness and liveness) as well as CTL properties. The state space exploration algorithm and the CTL model-checker are fully symbolic, built using the MDD library Meddly (Babar & Miner, 2010), and a large set of heuristics for variable ordering (Amparore, Donatelli & Ciardo, 2020a) that exploit the net structure. Qualitative verification can be integrated by a stochastic verification, as GreatSPN includes a model-checker (Amparore & Donatelli, 2018a) for the stochastic logic CSL^TA (Donatelli, Haddad & Sproston, 2009), and the computation of standard performance indices based on efficient analytical solutions of Markov chains and Markov Regenerative Processes, as well as simulation. The tool is open source and it is available from GitHub (https://github.com/greatspn/SOURCES).

The supporting libraries: meddly and spot

Spot 2.0 (Duret-Lutz et al., 2016) is a framework which provides, among other features, a wide range of tools to manipulate automata over infinite words and to translate them from LTL propositions. The framework has been maintained for decades and can be considered the state-of-the-art of LTL-to-Büchi translation. Our model checker uses the Spot library to construct a Generalized Büchi Automaton with state-based acceptance (SGBA) from each LTL formula encountered during the verification process. We chose to use the Hanoi-Omega Automata (HOA) format (Babiak et al., 2015) to represent such automata. The HOA format is a text-based encoding which supports various derivation of ω-automata. Spot can be downloaded from its home page (https://gitlab.lrde.epita.fr/spotandspot.lrde.epita.fr).

Meddly (Babar & Miner, 2010), Multi-terminal and Edge-valued Decision Diagram LibrarY, is an open-source library (https://github.com/asminer/meddly) for decision diagrams, developed at Iowa State University. It supports the construction and manipulation of various kinds of decision diagrams: Binary and Multivalued decision diagrams, matrix diagrams (M×D) and edge-valued DDs, including a built-in function to support efficient state-space construction based on saturation.

Background: LTL model checking

It is well-known, as in Vardi (1995) or Baier & Katoen (2008, p. 429), that for every LTL formula $A$φ it is possible to generate a Generalized Büchi Automaton that accepts all and only the paths that satisfy ϕ, so we shall define them next.

Definition 6 (GBA). A Generalized Büchi Automaton (GBA) is a tuple $\mathcal{A}=⟨Q,AP,\delta ,Q_0,\mathcal{F}⟩$, where Q is a finite set of locations, AP is a set of atomic proposition labels, $\delta \subseteq Q\times AP\times Q$ is a total transition relation, $Q_0\subseteq Q$ is the set of initial locations, and $\mathcal{F}=\{F_i|F_i\subseteq Q{\}}_{i=1}^n$ is the set of accepting sets of locations.

A GBA in which there is a single accepting set ( $|\mathcal{F}|=1$) is called a Büchi automaton (BA tout-court).

The language $\mathcal{L}$( $\mathcal{A}$) of a GBA $\mathcal{A}$ is the set of infinite words on AP recognized by a run of $\mathcal{A}$ that visits infinitely often at least one location in each $F_i\in \mathcal{F}$.

The standard automata-theoretic approach for model checking LTL formulae over a Kripke model M, following the schema of Vardi & Wolper (1986, 1994) is outlined in Algorithm 1, that checks if the quantified LTL formula $A$ϕ is satisfied for all paths π starting in the initial state m₀. To do so it builds the GBA $\mathcal{A}$ of ⫬ϕ (line 3), the BA $\mathcal{M}$ of the Kripke model M (line 4), and the synchronized product GBA $\mathcal{M}\otimes \mathcal{A}$ (line 5), whose language is the intersection of the languages of M and $\mathcal{A}$. If this language is empty, no path π satisfies ⫬ϕ and therefore ϕ holds on all paths $\pi \in {\mathcal{P}}_M^{\mathrm{\infty }}(m)$.

When dealing with Petri nets, the model to be checked is a stuttered RG, which is translated into the Kripke model M( $RG$) according to Definition 2. The corresponding automaton $\mathcal{M}$ is then built (line 4). Note that Büchi automata recognize words over edges, while Kripke models define a language of sequence of states (and associated atomic propositions). The transformation takes as accepting set $\mathcal{F}=\{RS\}$ and moves the labelling function of a state to all the incoming arcs, which requires a translation, based on Vardi & Wolper (1986), for the initial state (that has no incoming arc) that consists in adding a fictitious initial state s_pre, leading to the following definition.

Definition 7 (BA of a Kripke model). The Büchi automaton $\mathcal{M}=⟨Q,AP,\delta ,Q_0,\mathcal{F}⟩$ of a Kripke model M = 〈S, E, L〉, with $L:S\to 2^{AP}$, for the initial states $S_0\subseteq S$, is defined as follows: Q = S ∪ {s_pre}, with s_pre ∉ S; Q₀ = {s_pre}; ∀s, s′∈ Q\ s_pre: δ(s, s′, l) ∈ δ iff (s,s′) ∈ E and l = L(s′); ∀s′∈ S₀, δ(s_pre, s′, l) ∈ δ iff l = L(s′); $\mathcal{F}=\{S\}$.

Figure 5A shows the Büchi automaton built for the Kripke model in Fig. 1D, for S₀ = s₁.

We now define a simplified synchronized product of Büchi automata, for the case in which one automaton is a GBA and the other is a BA derived from a Kripke model (therefore with a single acceptance set that includes all states).

Definition 8 (Synchronized product GBA $\mathcal{M}\otimes \mathcal{A}$). Given a BA $\mathcal{M}=⟨Q^M,AP,{\delta }^M,Q_0^M,{\mathcal{F}}^M=\{Q^M\}⟩$ and GBA $\mathcal{A}=⟨Q^A,AP,{\delta }^A,Q_0^A,{\mathcal{F}}^{\mathcal{A}}⟩$, their synchronized product is the GBA $\mathcal{M}\otimes \mathcal{A}=⟨Q,AP,\delta ,Q_0,\mathcal{F}⟩$ where

$$Q=Q^M\times Q^A$$

$$Q_0=Q_0^M\times Q_0^A$$

$$\mathcal{F}=\{F_i{\}}_{i=1}^n:\mathrm{\forall }{q}^A\in F_i^A,\mathrm{\forall }{q}^M\in Q^M:(q^A,q^M)\in F_i$$

$$\delta =\subseteq Q\times AP\times Q:((q_i^A,q_i^M),a,(q_j^A,q_j^M))\in \delta \mathrm{i}\mathrm{f}\mathrm{f}$$

$$(q_i^A,a,q_j^A)\in {\delta }^A\mathrm{a}\mathrm{n}\mathrm{d}(q_i^M,a,q_j^M)\in {\delta }^M$$

It is known (e.g. Baier & Katoen, 2008, p. 156) that if $\mathcal{A}$ = $\mathcal{A}$ $\mathcal{A}$₂, then $\mathcal{L}$( $\mathcal{A}$) = $\mathcal{L}$( $\mathcal{A}$₁) ∩ $\mathcal{L}$( $\mathcal{A}$₂). The Büchi automaton for the LTL formula ⫬Φ with Φ = $G$ ⫬ β is shown in Fig. 5B; its synchronized product with the Büchi automaton in Fig. 5A is shown in Fig. 5C. The product BA has always a single initial state (s_pre, q₀). The set of accepting subsets $\mathcal{F}$ for the example is $\mathcal{F}=\{F_1\}$, with F₁ = {(s, q₁), ∀s ∈ $RS$}. Since, indeed, there is an infinite path that starts in (s_pre, q₀) and that visits infinitely often at least one state in F₁, then $\mathcal{L}$ ( $\mathcal{M}\otimes \mathcal{A}$) is not empty and therefore the initial state of the net does not satisfy the LTL formula $G$ ⫬ β. Note that the initial state of the net is the successor of the initial state of $\mathcal{M}\otimes \mathcal{A}$.

From LTL to Sat∃LTL

A limitation of Algorithm 1 is that it only computes if the initial state $m_{init}$ satisfies the formula $A$φ. For CTL ${}^{\ast }$, we shall need instead the entire sat-set. Moreover, we switch to existentially-quantified LTL formulae, since they are easy to compute in symbolic form, noting that a state (s |= $A$φ) ≡ (s |=⫬ $E$ ⫬φ). In this section we describe the extension of Algorithm 1 for sat-set computation as Sat∃LTL(M, φ).

A remarkable result of Clarke, Grumberg & Hamaguchi (1997), Theorem 2, p. 54) shows that LTL model checking can be actually performed using a fair CTL model checker, not on the Kripke structure $\mathcal{M}$ but on the product GBA $\mathcal{M}\otimes \mathcal{A}$. In particular, only a single CTL operator, $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}$ $G$ $true$, is needed to check arbitrary LTL formulae.

We briefly recall the definitions of CTL fairness. A fairness condition F is a subset of states that must be visited infinitely often. A path $\pi \in {\mathcal{P}}_M^{\mathrm{\infty }}(s)$ satisfies a fairness condition F if the states of F appear infinitely often on π. A fairness set $\mathcal{F}$ is a set of fairness condition F, that must all be met together. A path π is a fair path w.r.t. $\mathcal{F}$ iff π statisfies all fairness conditions of $\mathcal{F}$. A state s satisfies the fair CTL formula $E$ϕ w.r.t. fairness set $\mathcal{F}$ iff (1) it satisfies the CTL formula $E$ϕ, and (2) $\mathrm{\exists }\pi \in {\mathcal{P}}_M^{\mathrm{\infty }}(s)$ that is a fair path w.r.t. $\mathcal{F}$.

A state s satisfies a quantified LTL formula $E$φ if in the product GBA $\mathcal{M}\otimes \mathcal{A}$ there is a path from s, q₀ that visits infinitely often the states in the acceptance sets $\mathcal{F}$, i.e. the same definition of the fairness condition acceptance in fair CTL. Thus if we consider the GBA as a Kripke structure, finding the states that satisfy $E$φ becomes equivalent to finding the locations that originate fair paths, i.e. finding the states on the equivalent Kripke structure that satisfy $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}$ $G$ $true$ w.r.t. the fairness set $\mathcal{F}$. The definition of Sat∃LTL(M, φ) that performs LTL model checking using fair CTL is described in Algorithm 2.

The procedure first translates the path formula φ into a GBA $\mathcal{A}$ (line 2), it then builds the BA $\mathcal{M}$ of the input model M (line 3), described by its RG. Note that, since the goal is the sat-set computation, we need to consider all states as possible initial states, by taking S₀ = $RS$ in definition 7. Figures 6A and 6B show the BA $\mathcal{M}$ built from the RG of the net of the previous example, and the BA $\mathcal{A}$ for the formula $F$β. The algorithm then builds (line 4) the GBA $\mathcal{M}\otimes \mathcal{A}$, as in Definition 8. The $\mathcal{M}\otimes \mathcal{A}$ automaton is then translated into the Kripke model M( $\mathcal{M}\otimes \mathcal{A}$) (line 6 and 7). This translation is trivial due to the particular structure of $\mathcal{M}\otimes \mathcal{A}$, since all arcs that enter a state carry the same set of atomic propositions, that can then be associated to the state itself. The sat-set of the CTL formula $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}$ $G$ $true$ for M( $\mathcal{M}\otimes \mathcal{A}$) is then computed (lines 8 to 14), which is then mapped back to the states of the initial model (the RS of the Petri net) in line 15.

The $\mathcal{M}\otimes \mathcal{A}$ of the example is shown in Fig. 6C: there is a single initial state (s_pre, q₀) and the accepting set is $\mathcal{F}=\{F_1\}$, where F₁ = {(s,q₁), ∀s ∈ $RS$}, therefore, in this case, $\mathcal{M}\otimes \mathcal{A}$ is a BA. The Kripke model M( $\mathcal{M}\otimes \mathcal{A}$) of the synchronized product $\mathcal{M}\otimes \mathcal{A}$ is shown in Fig. 6D, together with all the sets computed by Algorithm 2.

Some aspects of this procedure need a deeper explanation, in particular the three different ways of computing the sat-set (lines 8–14) and the Z₀ construction and its use (lines 6 and 15).

[Lines 8–14] This part computes the set of states that originate an infinite path accepted by the $\mathcal{M}\otimes \mathcal{A}$ BA. Line 14 is the general case: the results in Burch et al. (1992) and Clarke, Grumberg & Hamaguchi (1997) show how to compute the sat-set through the model-checking of the fair CTL formula $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G(true,fair=\mathcal{F})$ on $\mathcal{M}\otimes \mathcal{A}$. The procedure for the sat-set computation of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G(true,fair=\mathcal{F})$ for a Kripke model is reported in Algorithm 3. Line 10 and 12 correspond to two specific cases, as it was shown (Bloem, Ravi & Somenzi, 1999) that if $\mathcal{M}\otimes \mathcal{A}$ is a weak or terminal BA, the computation of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}$ $G$ $true$ can be simplified.

A BA is weak if (1) it is possible to find a partition {Q_i} of its set of locations Q so that each Q_i is either contained in the accepting set or it is disjoint from it and (2) the {Q_i} are partially ordered s.t. the transitions of the automaton never move from Q_i to Q_j unless Q_i ≤ Q_j. In this case the only way for a path to visit infinitely often a state of the accepting set F₁ is to eventually be confined inside one $Q_i\subseteq F_1$, and an accepting run is a witness for the CTL formula EF EG F₁. A terminal BA is a weak BA in which the {Q_i} are maximal elements in the partial order. In this case an accepting run is a witness for the CTL formula EF F₁. The time complexity for model checking the two CTL formulae is linear in the size of the model, while the cost for the general case in line 14 is instead quadratic. Note that Algorithm 2 tests the weak and terminal condition on $\mathcal{A}$, and not on the (usually) much larger $\mathcal{M}\otimes \mathcal{A}$, as it was proven in Bloem, Ravi & Somenzi (1999) that if $\mathcal{A}$ is weak (terminal) so is $\mathcal{M}\otimes \mathcal{A}$. When this is the case the simplified procedure obviously applies also to M( $\mathcal{M}\otimes \mathcal{A}$), as we do.

[Lines 6 and 15]. Z₀ computed in line 6 is the set of immediate successor of the initial state(s) of the $\mathcal{M}\otimes \mathcal{A}$. This step is needed because of the fictitious s_pre state introduced in $\mathcal{M}$ by the translation of Definition 7. The map operation in line 15 maps pairs (s_i, q_j) onto the marking of s_i.

The sat-set of the EFβ for the Petri net system of the example is also listed in Fig. 6D. Note that (s₃, q₁) |= $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G$ $true$, as there is indeed an infinite loop on the (s₃, q₁) state, which is an accepting state. But s₃ corresponds to the Petri net marking m₃, a deadlock state that does not satisfies β, so certainly $s_3\text{⧸}\models EF\beta$. Indeed (s₃, q₁) ∉ Z₀ since the witness path of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G$ $true$ for (s₃, q₁) corresponds to a path in the automata of the formula that does not start from the initial location q₀.

The work in Emerson & Lei (1987) gives a symbolic algorithm (known as Emerson-Lei algorithm) for model checking fair CTL properties, while (Burch et al., 1992) provides its fixed point characterization. Its time complexity is quadratic in the size of the automaton being checked. Algorithm 3 is an high-level view of the this fixed point characterization for the sat-set computation of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G$ψ formulae for a Kripke model M, with a set of fairness constraints $\mathcal{F}$. In the algorithm, the call to SATCTL(M, Φ) returns the sat-set of the CTL formula Φ (without fairness constraints) on the Kripke model M. The Until and next path operators at line 6 and 7 use sets instead of atomic propositions or state formulae, for simplicity.

CTL ${}^{\ast }$ model-checking procedure

To understand the CTL ${}^{\ast }$ model checking procedure of starMC, we have to explain how a single CTL ${}^{\ast }$ formula is divided into multiple quantified LTL formulae that can be checked using Sat∃LTL only. It may be worth starting from an example. Let’s consider the CTL ${}^{\ast }$ formula

$$\mathrm{\Psi }=EFGE(\alpha U(AX\beta ))$$where $\alpha ,\beta$ are atomic propositions, and define the following (sub-)formulae:

$$\begin{array}{cc}{\mathrm{\Psi }}_1=AX\beta & {\mathrm{\Psi }}_2=E(\alpha U(AX\beta ))\\ {\mathrm{\Psi }}_3=E(\alpha U{a}_{{\mathrm{\Psi }}_1})& {\mathrm{\Psi }}_4=EFG{a}_{{\mathrm{\Psi }}_3}\end{array}$$

Let Sat(Ψ) be the sat-set of Ψ. Assume that we have a model checking procedure SATLTL(Ψ) that computes Sat(Ψ) if Ψ is a quantified LTL formula. Similarly, assume that a procedure SATCTL(Ψ) computes Sat(Ψ) if Ψ is a CTL formula. Formulae Ψ₁ and Ψ₃ can be considered as both quantified LTL and CTL formulae, while Ψ₂ is not LTL, and Ψ₄ is not CTL. If we assume that aΨ_i is an atomic proposition that holds in state s iff s |= Ψ_i, then we can compute Sat(Ψ) using only SATLTL on the sequence of formulae: Ψ₁, Ψ₃ and Ψ₄. Vice-versa, Sat(Ψ) cannot be computed using only SATCTL, since SATCTL(Ψ₂) allows to rewrite Ψ into Ψ₄ which is an LTL formula for which it is well-known that no equivalent CTL one exists. According to Emerson & Lei (1987), the idea of substitution was already present in Emerson & Sistla (1984) and it was later used in Visser & Barringer (2000).

Algorithm 4 is the logical view of the CTL ${}^{\ast }$ model-checking procedure implemented in starMC. The algorithm follows the ideas in Emerson & Lei (1987), and builds a CTL ${}^{\ast }$ model-checker based on SAT∃LTL. It contains one case per type of state formula (by definition a CTL ${}^{\ast }$ formula is a state formula). The only non-trivial case of Algorithm 4 is Eφ in line 7: if φ is an LTL path formula, SAT∃LTL can be directly applied to the formula, if instead φ contains additional quantifications, we first have to rewrite φ as an LTL path formula (by substituting quantified sub-formulae with newly created atomic propositions). Both cases are treated in a unified manner through the call of SAT∃LTL(REWRITE(φ)). Each formula identified for the substitution is a Maximal Proper Quantified Sub-formula (MPQS). A MPQS is defined in terms of the so-called Maximal Proper State Sub-formula (MPSS); see (Baier & Katoen, 2008), def.6.86.

Definition 9 (Maximal Proper State Sub-formula). State formula $\mathrm{\Psi }isaMPSSof\rho whenever\mathrm{\Psi }$ is a sub-formula of $\rho thatdiffersfrom\rho$ and that is not contained in any other proper state sub-formula of ρ.

Let MPSS(ρ) be the set of all maximal proper state sub-formula of ρ.

Example (from (Baier & Katoen, 2008)). If $\rho =E\phi$ and

$\phi =X(AGEFa)\wedge FGE(Xa\wedge Gb)$then MPSS( $\phi )=\{AGEFa,E(Xa\wedge Gb)\}$. Back to the initial example, $MPSS(\mathrm{\Psi })=\{{\mathrm{\Psi }}_2\}$ and $MPSS({\mathrm{\Psi }}_2)=\{{\mathrm{\Psi }}_1\}$.

Let’s consider this second example:

$$\mathrm{\Psi }=EFG\underset{{\mathrm{\Psi }}_1}{\underbrace{E(\alpha U\underset{{\mathrm{\Psi }}_2}{\underbrace{(\alpha \wedge \underset{{\mathrm{\Psi }}_3}{\underbrace{AX\beta }})}})}}$$than ${\mathrm{\Psi }}_2\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{M}\mathrm{P}\mathrm{S}\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{\Psi }}_1\mathrm{a}\mathrm{n}\mathrm{d}{\mathrm{\Psi }}_3\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{M}\mathrm{P}\mathrm{S}\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{\Psi }}_2$. In our model-checking procedure we consider instead a refinement of the MPSS definition, that we call Maximal Proper Quantified Sub-formula (MPQS). The motivations for introducing MPQS with respect to previous work and MPSS is given in “MPSS vs. MPQS”.

Definition 10 (MPQS). A Maximal Proper Quantified Sub-formula (MPQS) $\mathrm{\Psi }ofaformula\rho isaMPSSof\rho whichisaquantifiedstateformula,i.e.either\mathrm{\Psi }=E\varphi or\mathrm{\Psi }=A\varphi$.

The MPQS decomposition maximises the length of path formulae whenever possible. The MPQS decomposition of the example CTL ${}^{\ast }$ formula is:

$$\mathrm{\Psi }=EFG\underset{{\mathrm{\Psi }}_1}{\underbrace{E(\alpha U(\alpha \wedge \underset{{\mathrm{\Psi }}_3}{\underbrace{AX\beta }}))}}$$

The state sub-formula ${\mathrm{\Psi }}_2\mathrm{c}\mathrm{a}\mathrm{n}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{b}\mathrm{e}\mathrm{a}\mathrm{M}\mathrm{P}\mathrm{Q}\mathrm{S}\mathrm{o}\mathrm{f}{\mathrm{\Psi }}_1,since{\mathrm{\Psi }}_2$ is not a quantified state formula.

REWRITE is the procedure to identify MPQS; its logical view is described in Algorithm 5, which is based on the syntactical recognition of the input formula ρ. Operators and atomic propositions are left unchanged (lines 3–9), when $\rho =E\varphi$ (line 10) the formula ρ is rewritten as the atomic proposition $a_{\rho }\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{t}\mathrm{o}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{l}\mathrm{y}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}{a}_{\rho }$, the sat-set of ρ is computed through the indirect recursive call to SatCTL ${}^{\ast }$(ρ).

CTL ${}^{\ast }$ model-checking approaches

CTL ${}^{\ast }$ model checking procedures can be distinguished depending on whether they are conducted on the model itself or on an extended model (as in case of the synchronized product).

Sat-set computation for E_fairG Ψ

The work in Emerson & Lei (1986), already mentioned for its contribution in proving that μ₂-calculus can be efficiently model-checked, also provides a succinct translation from fair-CTL to μ₂-calculus, paving the way to efficient model checking of fair-CTL, in particular for the sat-set computation of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}$ $G$Ψ. The algorithm is provided in an abstract form, regardless of the specific data structures choice. It is readily suitable for a symbolic implementation, which is what we have used for starMC.

A different symbolic solution for the computation of a fair cycle is provided by the OWCTY algorithm (Černá & Pelánek, 2003), which does not compute sat-sets but aims at building counter-examples as quickly as possible. A comparison of the two approaches for LTL model checking can be found in Duret-Lutz et al. (2011).

Sat-set computation for ELTL

Buchi-based model checking of LTL of a formula Φ is defined in Vardi & Wolper (1986), as recalled in “StarMC: The Logical View”. The paper in Burch et al. (1992) shows a symbolic model checking procedure for LTL. It is based on an implicit tableau construction and on its symbolic representation. It is then shown that the sat-set of an LTL formula can be reduced to the fair CTL symbolic model-checking for the μ-calculus expansion of EG true, under appropriate fairness constraints.

A later work (Clarke, Grumberg & Hamaguchi, 1997) encodes the tableau of the LTL formula along with the model variables in a single SMV (Clarke et al., 1996) model, and explicitly reduces the LTL model checking to a fair-CTL formula, using the existing and unmodified CTL model checker of SMV. Model checking fair CTL is based on the Emerson-Lei algorithm (Emerson & Lei, 1986). This procedure, together with its implementation in BDD is well explained in the survey paper by Rozier (2011). The survey also cover many other related topics, including a comparison of the expressiveness of LTL, CTL and CTL ${}^{\ast }$ both in theory and in practical applications.

starMC exploits the Buchi-based approach in Vardi & Wolper (1986), that makes the synchronized product of the Büchi automaton and the model, and uses the algorithm in Emerson & Lei (1986) for the detection of the accepting runs and the consequent construction of the set of states that satisfy the formula. The library Spot (Duret-Lutz, 2014; Duret-Lutz et al., 2016) provides different, efficient, ways to build a Büchi automaton from an LTL formula. The generated automaton may have state-based or transition-based acceptance. This latter choice, based on the translation defined in Couvreur (1999), implemented in Spot using decision diagrams, is usually the most efficient. starMC nevertheless uses state-based automata, since it allows an easier re-use of the SatCTL module of GreatSPN.

A number of possible optimizations have been presented in the literature for LTL model-checking. For example in Duret-Lutz et al. (2011) a modified algorithm called Self-Loop Aggregation Product (SLAP) is proposed, where instead of building a synchronized product in an extended space (model and automaton), the joint state is kept into separate aggregates. Accepting runs are then searched only in those aggregates that may contain cyclic runs over the accepting states. For the time being starMC does not include this kind of algorithmic optimization, although a certain attention has been taken to avoid inefficiencies in the symbolic implementation.

We do not review here the numerous techniques for the on-the-fly model-checking of LTL, since these techniques concentrate on establishing the truth value of a formula in the initial state, while the Sat∃LTL procedure needed for CTL ${}^{\ast }$ model checking requires the construction of the sat-sets of the LTL sub-formulae.

CTL ${}^{\ast }$ model checkers

According to a relatively old paper (Barringer et al., 2002) there was a CTL ${}^{\ast }$ model-checker available in the Rainbow verification system (Barringer et al., 2002): the tool is not available any longer and we could find no published details on how the model checker was built, only that it was based on HAA. Previous papers by the same authors evaluated whether it was the case to enhance the tool SPIN with CTL ${}^{\ast }$ (Visser & Barringer, 1998; Visser & Barringer, 2000), but apparently it was never done and/or reported in the literature.

There is an implementation of μ-calculus for the nuXmv tool (Cavada et al., 2014), which, according to its website, could lead to a CTL ${}^{\ast }$ model-checker but, for the time being, only CTL ${}^{\ast }$ formulae that are either LTL or CTL are actually processed.

The LTSmin tool (Kant et al., 2015) includes a CTL ${}^{\ast }$ model checker. It allows to model check a Petri net by translating the CTL ${}^{\ast }$ formula to be checked into an equivalent μ-calculus one, using the procedures defined in Dam (1990). A model checker of μ-calculus for Petri nets is available also in TINA (Berthomieu, Ribet & Vernadat, 2004), but no translator from CTL ${}^{\ast }$ to μ-calculus is provided.

MPSS vs. MPQS

In the literature there have been slightly different definitions of the CTL ${}^{\ast }$ sub-formulae to be substituted and, although they are all correct and adequate for CTL ${}^{\ast }$ model-checking based on Sat∃LTL, different definitions may lead to a different number and/or to a different structure of the Büchi automata being built. The work in Emerson & Lei (1987) does not contain a definition in strict sense of the formulae to be substituted, but it states that, for each formula $E$ϕ, where ϕ is a path formula, then the set of formulae to be substituted by atomic propositions are the “top-level proper existential sub-formulae $E$q_i of ϕ that are not sub-formulae of any other sub-formula Er of Eϕ, where Er is different from Eϕ and from Eq_i”. The Eq_i are therefore the largest quantified sub-formulae of the formula ϕ.

In later times the substitution of formulae with atomic propositions was based on the definition of MPSS (Visser & Barringer, 2000), which is equivalent to the one given by def.6.86 in Baier & Katoen (2008), reported in this paper as Definition 9. The definition of MPSS encounters two problems when deriving an algorithm, it may re-label more formulae than strictly necessary, and it may result in a non-deterministic algorithm since the CTL ${}^{\ast }$ grammar is ambiguous. Indeed in the considered example:

$$\mathrm{\Psi }=EFG\underset{{\mathrm{\Psi }}_1}{\underbrace{E(\alpha U\underset{{\mathrm{\Psi }}_2}{\underbrace{(\alpha \wedge \underset{{\mathrm{\Psi }}_3}{\underbrace{AX\beta }})}})}}$$

Since ψ₂ is both a state and a path formula due to the ambiguity of the CTL ${}^{\ast }$ language, there are two evaluation strategies:

1. Evaluate Ψ₁ as $E$ (α $U$aψ₂);

2. Evaluate Ψ₁ as $E$ (α $U$(α ∧ aψ₃));

Both strategies are correct, but they are not equivalent. The former strategy is based on the identification of MPSS sub-formulae. While correct, it inhibits possible reductions and optimizations for Büchi Automata generation, since the label aψ₂ clearly carries less information than the (α ∧ aψ₃) formula. This example also shows that MPSS are different from “top-level proper existential sub-formulae” which are at the base of the CTL ${}^{\ast }$ model checking algorithm in Emerson & Lei (1987) and that we have formally defined as MPQS in Definition 10. The MPQS decomposition of the example CTL ${}^{\ast }$ formula is:

$$\mathrm{\Psi }=EFG\underset{{\mathrm{\Psi }}_1}{\underbrace{E(\alpha U(\alpha \wedge \underset{{\mathrm{\Psi }}_3}{\underbrace{AX\beta }}))}}$$

The state sub-formula ψ₂ cannot be a MPQS of Ψ₁, since Ψ₂ is not a quantified state formula.

CTL ${}^{\ast }$ model checking

Implementation of $\mathcal{M}$ construction

The generation of the decision diagrams that encode the $\mathcal{M}\otimes \mathcal{A}$ automaton is summarized in Algorithm 6. The MDD (M×D) of $\mathcal{M}\otimes \mathcal{A}$ is defined to have n + 1 (2n + 2 resp.) levels: one level each for the n places of the net and one for the location of $\mathcal{A}$. We indicate with $⟨m,q⟩$ a joint state of $\mathcal{M}\otimes \mathcal{A}$, where m is a model state and q is an automaton location.

Algorithm 7:

Implementation view of SAT∃LTL.

1: procedure SAT∃LTL(M, φ)

2:    $\mathcal{A}$ ← translate φ into a Büchi Automaton

3:    $⟨Z_0,NSF,S_{\mathcal{F}}⟩$ ← BUILDSYNCHPRODUCT(M, $\mathcal{A}$)

4:   S ← Saturate(Z0, NSF)

5:   K = (S, NSF ) is a Kripke model

6:   AS ← MDD()

7:   switch type( $\mathcal{A}$) do

8:      case weak:

9:       AS ← SATCTL $AS\leftarrow SATCTL(K,EFEG(S_{\mathcal{F}}[1]))$

10:     case terminal:

11:       AS ← SATCTL $AS\leftarrow SATCTL(K,EF(S_{\mathcal{F}}[1]))$

12:     case otherwise:

13:       AS ← SATEFAIR $AS\leftarrow {\mathrm{S}\mathrm{A}\mathrm{T}\mathrm{E}}_{\mathrm{F}\mathrm{A}\mathrm{I}\mathrm{R}}G(K,\mathbf{t}\mathbf{r}\mathbf{u}\mathbf{e},fair=S_{\mathcal{F}}):$

14:   Sat(∃φ) ← RemLoc(AS ∩ Z0)

15:   return Sat(∃φ)

16: end procedure

DOI: 10.7717/peerj-cs.823/table-10

The algorithm builds the MDD and the M×D of $\mathcal{M}\otimes \mathcal{A}$ taking as input a graph description of the formula automaton $\mathcal{A}$ and the M×D for the NSF of $\mathcal{M}$. There is no MDD for the initial states of $\mathcal{M}$ since all states are considered as initial ones. The algorithm implements at the same time the translation of $\mathcal{M}$ into a Büchi automaton (Definition 7) and the $\mathcal{M}\otimes \mathcal{A}$ construction (Definition 8). The algorithm directly builds the Kripke model, without passing through the $\mathcal{M}\otimes \mathcal{A}$ BA as described in “StarMC: The Logical View”. The DD representation of the $\mathcal{M}\otimes \mathcal{A}$ automaton of Algorithm 6 consists of three elements: M×D NSF for the δ function; an array $S_{\mathcal{F}}$ of MDDs of the accepting sets; and the MDD for the set Z₀ of states as defined by Algorithm 2.

Figures 7B–7G shows the DDs built by Algorithm 6 for the net of Fig. 1A and for the formula expressed by the Büchi automaton in Fig. 7A. These DDs represent the symbolic encoding of the information (models and sat-sets) of Fig. 6.

CT test based on MCC results

starMC results are checked against the available truth values for CTL and LTL properties. For each instance we computed the result for 32 CTL and 32 LTL properties, for a total of 27,776 queries (434 ${}^{\ast }$ 64) queries. With the test resource limitations, starMC was able to terminate about 80% of the queries. All unanswered queries did not finish in the time bound. None of the query evaluation generated a not-enough-memory error. For 7 of these 434 models, the query format was corrupted due to bad namings. These 7 models were thus dropped. We found no mismatch w.r.t. the MCC published values for the remaining 427 models.

To get a better accuracy we need to verify more precise information like the sat-set cardinalities (the CC test), as in the next steps, but before proceeding to sat-set computation we have investigated the possible overhead caused by the (exponential) size of the Büchi automata. We have computed the size of the Büchi automata for about 14 k LTL queries. The largest automata produced by Spot has 70 locations and 356 edges. The average number of locations (edges) is 3.95 (7.7). This correspond to a maximum and average number of path operators equal to 12 and 4.7 respectively. Note that starMC may build more than one Büchi automata per formula when model-checking CTL and CTL ${}^{\ast }$ formulae, due to nesting.

CC and PE tests based on RGMeDD

This test compares the results of starMC and RGMeDD on CTL formulae. RGMeDD is the CTL model checker (Amparore, Beccuti & Donatelli, 2014) of GreatSPN, based on the standard recursive sat-set computation of CTL. The test therefore compares two different approaches to CTL model checking: a standard fixed point implementation (RGMeDD), and an automata-based implementation (starMC). Both tools use the same library for DD manipulation, moreover the same variable orders have been used. The benchmark is again on 427 instances, for a total of 13,664 CTL queries (427 ${}^{\ast }$ 32).

Table 1 summarizes the behaviour of the two tools, while the plots in Fig. 9 report the execution times in linear form Fig. 9A and log form Fig. 9B: dots below the diagonal are queries for which RGMeDD is faster than starMC and vice-versa. Executions that have timed out are marked as TO.

RGMeDD completes 10,740 queries (79%) from 427 different model instances, while starMC completes 10,223 queries (75%) from 399 different model instances. All the 9,747 queries completed by both tools (71%) produced sat-sets of equal cardinality (CC test).

We have also analyzed the different behaviour in terms of solved queries and model instances, with RGMeDD being able to solve more queries but especially for more diverse instances.

Moreover, on the queries completed by both tools (blue dots in the plot), RGMeDD was faster than starMC on 87% of the queries, slower on the 10% and equal time (up to millisec) on 3%. There are a number of cases in which the execution times are rather different (values close to only the x or the y axis). This result for the PE test is somehow unexpected: considering that the same DD library is used, and the same variable orders, the differences are possibly due to the different model-checking approaches, and we actually expected the standard CTL procedure of RGMeDD to be more efficient in almost all practical cases. Factors that could make starMC faster than RGMeDD are the optimization of the formula done by Spot when building the Büchi automaton and a different use of the potential state space in the two implementations.

Tests based on LTSmin

The last tool assessment is based on a comparison with LTSmin (Kant et al., 2015): CC and PE are assessed for LTL, CTL and CTL ${}^{\ast }$ formulae. LTSmin is run using the pins2lts-sym interface with the –ctlstar option, which first converts the input formula into μ-calculus and then applies the μ-calculus model-checker of LTSmin. It is worth noting that this translation may incur an exponential cost since LTSmin uses the translation described in Dam (1990), which is in theory less efficient than using Büchi automata. LTSmin is also based on decision diagrams, provided by the multi-core library Sylvan (van Dijk & van de Pol, 2017). To make a meaningful comparison, for each model instance we have enforced the use of the same variable order for the two tools. We have initially checked that both tools, when given the same variable order, produce exactly the same DD in output. Thus the choice of the variable order does not penalize one tool or the other, as long as it is the same for both tools. We give LTSmin the same time constraints as the other tools, i.e. 60 s to generate the state space, and 60 s to translate the formula in μ-calculus and perform the model checking. Since we have experienced inconsistencies of the LTSmin tool when dealing with models with deadlocks, we have considered deadlock-free models only. Moreover, only P/T models have been included in the analysis, since LTSmin does not treat colored models, leaving 222 model instances from 29 distinct models. For what concerns the type of queries, only LTL/CTL/CTL ${}^{\ast }$ queries of the “Cardinality” category have been checked, since LTSmin cannot express atomic propositions based on the enabling of transitions. We did not consider model instances for which none of the two tools was able to compute the state space, leaving 123 model instances from 28 different models. Of the 123 models, LTSmin builds the state space of 86 and starMC of 116. In all tests, LTSmin ran on four cores (to allow it to exploit the parallelism of the Sylvan library) and starMC on a single one, similar to the settings used for the MCC competition.

The first test (CC and PE for LTL and CTL formulae) compares the sat-set cardinalities and the execution times for LTL and CTL formulae, while the second test (CC and PE for CTL ${}^{\ast }$ formulae) extends the analysis to CTL ${}^{\ast }$ formulae, the final target of the whole testing procedure.

CC and PE for LTL and CTL formulae

We computed the sat-sets generated by starMC and LTSmin (running in –ctlstar mode) on LTL and CTL formulae on the 123 model instances selected as explained above, with 16 CTL formulae and 16 LTL ones for each model instance. The objective is to check the performance of the two model checkers (PE) and, for all queries solved by both tools, check that the same sat-sets’ cardinalities are computed (CC). The results are summarized in Table 2, while a comparison of the execution times in log and linear form (as for the previous experiment) is reported in Fig. 10.

Table 2:

Summary of the experiments with starMC and LTSmin on CTL and LTL queries.

Characteristics	Value	Characteristics	Value
No. of queries	3,936 (123*32)	Same Time both	1
starMC terminates	2,990	Both timed-out	895
LTSmin terminates	1,758	Only LTSmin timed-out	1,277
Both terminates	1,708	Only starMC timed-out	50
starMC faster	485	Mismatches in sat-set cardinality	7
LTSmin faster	1,222

DOI: 10.7717/peerj-cs.823/table-2

PE results: in the 60 s limits, starMC solves 50% more queries than LTSmin, although, in the subset of queries solved by both, LTSmin is faster than starMC on 2 out of 3 queries.

CC results: of the 1,708 queries computed by both tools, we have a mismatch in the cardinalities of the sat-sets of 7 formulae (from different model instances). These are all LTL formulae. All results computed by starMC are consistent with the MCC known truth values. Since truth values of MCC are assigned according to majority of the output of the tools that participate in the competition, they may not be 100% reliable. We have therefore computed the sat-sets of the sub-formulae, which indicate that LTSmin is computing the wrong sat-sets, most likely due to a wrong translation to μ-calculus. For example, the sat-set of the formula named LTLCardinality-08 for the model BART-PT-002 has a structure “ $X$ $F$ $G$ $F$a”, where a is an atomic proposition and the formula is implicitly quantified for all paths. LTSmin reports that the formula has an empty sat-set, although the sat-set of the formula “ $F$ $G$ $F$a” is equal to the full state space, which clearly indicates contradictory results. The translation in μ-calculus of the former query leads to a recursive formula with 7 μ and 18 ν fixed point operators, and it is not trivial to assess whether there is an error in the translation of the formula into μ-calculus or in the model-checking procedure. This behaviour has been reported to the LTSmin developers.

CC and PE for CTL ${}^{\ast }$ formulae

The tests of “CC and PE for LTL and CTL formulae” are here repeated for the set of generated CTL ${}^{\ast }$ formulae. The results are summarized in Table 3, while a comparison of the execution times in log and linear form (as for the previous experiments) is reported in Fig. 11.

Table 3:

Summary of the experiments with starMC and LTSmin on CTL* queries.

Characteristics	Value	Characteristics	Value
No. of queries	1,968 (123*16)	LTSmin faster	536
starMC terminates	1,442	Both timed-out	505
LTSmin terminates	775	Only LTSmin timed-out	688
Both terminates	754	Only starMC timed-out	21
starMC faster	218	Mismatches in sat-set cardinality	10

DOI: 10.7717/peerj-cs.823/table-3

PE: in the 60 s limits, starMC solves the highest number of queries (73%), almost twice those solved by LTSmin. In the subset of queries solved by both, LTSmin is faster than starMC on almost 4 out of 5 queries.

CC: of the 753 queries computed by both tools, we have a mismatch in the cardinalities of the sat-sets of 10 formulae (from different model instances). We have checked these formulae one by one, by computing the sat-set of sub-formulae, when meaningful and useful. In all but one case, the two tools differ radically: the sat-set being the whole state space for one and the empty set for the other, or vice-versa. In these cases it was not difficult to check that LTSmin is computing results that are inconsistent with the sat-sets of the sub-formulae computed by the tool itself. For the remaining case a special procedure was put in place that indicates that it is likely that LTSmin is computing the wrong result² .

Petri nets only?

starMC has been developed for Petri nets, either in GreatSPN format or in the PNML standard format. Although having a model-checker fully integrated into a specification and verification tool is an advantage for the user, it may hamper its reuse in the research community. There are indeed tools like LTSmin that try to be as general as possible, by providing a (formalism agnostic) intermediate language. starMC exploits Petri nets to collect useful information: computation of place bounds (so the user does not have to identify them beforehand) and heuristics for variable orders, which are based on P-semiflows (a structural property of the Petri net). Both issues relates to performance and do not hamper the application of the results presented in this paper to other formalisms for DEDS; nevertheless the tool is not currently structured for direct reuse as a stand-alone multi formalism model-checker.

Tool’s limitations

starMC builds the whole state space beforehand even for properties that could be proved by inspection of a part of the state space (no on-the-fly verification), and it does not build counter-examples.

Answer to research questions

(R1): our testing campaign has shown that it is possible to realize an efficient and fully symbolic implementation of the computation of sat-sets of LTL and CTL ${}^{\ast }$ properties based on Büchi automata. (R2): the experiments conducted in the testing phase indicate that starMC can solve (significantly) more formulae than LTSmin, although, on the formulae solved by both tools, LTSmin is generally faster. (R3): an extensive comparison of different variable order heuristics has not been conducted yet, but the size of the system that we were able to solve show that, indeed, the variable order heuristic used, that was the best one for state-space exploration, as reported in Amparore et al. (2019), is at least adequate for CTL ${}^{\ast }$ model-checking. (R4): the construction of counter-examples and witnesses has not been addressed in this paper.

Current and future work

starMC builds the whole state space beforehand even for properties that could be proved by inspection of a part of the state space (no on-the-fly verification). We are working on a model-checker that keeps the NSF in implicit form: the M×D of the NSF is substituted by a function that performs the firing of the Petri net transitions by directly manipulating the MDD of the state space being built. With a NSF in implicit form it is possible to implement an on-the-fly approach, in which the state space is built incrementally up to the point where the property can be (dis-)proven. The results could be extended to CTL ${}^{\ast }$ following the work in Bhat, Cleaveland & Grumberg (1995), that develops an on-the-fly procedure for LTL and then extends it to CTL ${}^{\ast }$.

No model-checker is fully useful if it does not produce meaningful counter-examples. GreatSPN already produces CTL counter-examples or witnesses for the initial state. Extending the current approach to CTL ${}^{\ast }$ may require a definite shift in the approach, for example by looking into the creation of evidence for μ-calculus (Cranen, Luttik & Willemse, 2015), while keeping an eye on the efficiency of the construction, as in Jiang & Ciardo (2018).

Although CTL ${}^{\ast }$ allows to express fairness constraints in the formula, we plan to develop also a model checker for the fair variant of CTL. Although this may seem a straightforward task, considering the already existing implementation of $E_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{r}}G$ ψ, further analysis is needed to understand the most adequate form of fairness, in particular if we want to consider not only fairness based on visited states (and therefore among enabled actions), but also fairness of taken actions, to include, for example, the “Fairness running” clause of the NuSMV tool (Cavada et al., 2014).

The example in “CTL ${}^{\ast }$ Model-checking Procedure” points out that a CTL ${}^{\ast }$ model-checker could be realized based on a mixture of Sat∃LTL and SatCTL. Beyond the theoretical differences in complexity, the construction of such a CTL ${}^{\ast }$ model-checker requires more experiments to understand if there is a practical advantage in doing so. Our comparison in “CC and PE Tests Based on RGMeDD” points out that it could be so.

Availability

• A virtual machine, with the reproducible benchmark (scripts, models, tools, instructions) is available as a Zenodo permalink. The provided benchmark can be run either in quick or in full mode, which will take about 30 minutes/10 days, respectively, to complete. Instructions are provided inside the VM, available at: https://zenodo.org/record/5752419 with the name starMC-benchmark.ova.

• A second virtual machine with the starMC tool pre-installed can also be found at the same link, with the name starMC.ova.

• The source code of GreatSPN is available at https://github.com/greatspn/SOURCES.

• The benchmark data is available at https://github.com/amparore/starMC-benchmark.