UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Probabilistic Constraint Nets : a formal framework for the modeling and verification of probabilistic… St-Aubin, Robert Jr. 2005

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_2005-105720.pdf [ 19.48MB ]
JSON: 831-1.0051506.json
JSON-LD: 831-1.0051506-ld.json
RDF/XML (Pretty): 831-1.0051506-rdf.xml
RDF/JSON: 831-1.0051506-rdf.json
Turtle: 831-1.0051506-turtle.txt
N-Triples: 831-1.0051506-rdf-ntriples.txt
Original Record: 831-1.0051506-source.json
Full Text

Full Text

Probabilistic Constraint Nets: A Formal Framework for the Modeling and Verification of Probabilistic Hybrid Systems. by Robert Jr. St-Aubin B.Math. Universite de Montreal, 1996 M.Sc. University of British Columbia, 1998 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  Doctor of Philosophy in THE FACULTY OF GRADUATE STUDIES (Computer Science)  The University of British Columbia June 2005 © Robert Jr. St-Aubin, 2005  Abstract The development of autonomous agents, such as mobile robots or software agents has generated considerable research in recent years. Robotic systems, which are usually built from a mixture of continuous (analog) and discrete (digital) components, are often referred to as hybrid dynamical systems. The modeling and analysis of hybrid dynamical systems is becoming more and more important as such systems are now widely used to reason about complex physical systems. Ying Zhang and Alan Mackworth developed a semantic model for dynamical systems, called Constraint Nets (CN) [ZM95a]. C N introduces an abstraction and unitary framework to model hybrid systems. Furthermore, specification and verification methods were introduced for deterministic system. Traditional approaches to real-time hybrid systems usually define behaviors purely in terms of determinism or sometimes non-determinism. The CN framework was developed to model and verify deterministic systems, with the capability to model non-determinism. However, real-time dynamical systems very often behave unpredictably and thus exhibit (structured) uncertainty. It is therefore important to be able to model and analyze real-time probabilistic systems. Hence, a formal framework to model systems with unpredictable behaviors is essential. We extend the work previously done on Constraint Nets by developing a new framework that we call "Probabilistic Constraint Nets" (PCN). The PCN framework allows for the modeling and simulation of any dynamical system, whether it is deterministic, non-deterministic or probabilistic. We introduce formal syntax and semantics for the framework that ensure the correctness of the models. We also provide a graphical representation that simplifies the task of modeling complex systems. Moreover, we show that our framework is a generalization of many commonly used frameworks such as Markov processes and Markov Decision Processes (MDP). This allows the user to take advantage of a unified framework encompassing most popular modeling paradigms. We have also developed two specification languages (average-time timed V-automata and PATTL) along with verification algorithms that allow us specify some behavioural constraints on the system and enables us to proceed to on average and to probabilistic verification of these requirements. Finally, we also provide, for a subclass of PCN models algorithms for control synthesis. Moreover, we investigate the use of stochastic and robust control for handling the control synthesis task within PCN. With such control synthesis techniques, a designer can automatically construct an optimal controller for his system, hence greatly facilitating his task.  ii  Contents Abstract  ii  Contents  iii  List of Tables  vii  List of Figures  viii  Acknowledgements  xi  Dedication 1  Introduction and Motivation  1.1 1.2 1.3  2  xii 1  Considered Problems Proposed Solutions to the Considered Problems Related Work 1.3.1 Review of Constraint Nets 1.3.2 Modeling Probabilistic Systems 1.3.3 Requirement Specifications and Verification Techniques  3 10 14 14 19 21  Measure-Theoretical and Topological Structure of Dynamics  26  2.1  26 30 33 36 40 43 46 49 53 53 54  2.2 2.3 2.4 2.5  General Topology, Partial Order, Metric Space and Measure Theory 2.1.1 Partial order 2.1.2 Metric space 2.1.3 Random Variables 2.1.4 White Noise, Brownian Motion and Stochastic Integrals Time Structures Domain Structures Stochastic Traces and Events Transductions 2.5.1 Transductions: general concepts 2.5.2 Primitive transductions  iii  .  2.6 3  4  62  3.1  62 63 69 73 74 75 79 81  Probabilistic Constraint Nets 3.1.1 Syntax of Probabilistic Constraint Nets 3.1.2 Semantics of Probabilistic Constraint Nets 3.1.3 Fixpoint in distribution of partial orders 3.1.4 Semantics of Probabilistic Constraint Nets 3.1.5 Semantics of Modules 3.1.6 Family of Probabilistic Constraint Nets 3.1.7 Stochastic temporal integration  Modeling in P C N  4.2 4.3  5.2  5.3  5.4 5.5  86 87 90 90 91 91 92 97  Discrete time and discrete domain systems 5.1.1 DTMC to PCN conversion 5.1.2 DD-PCN equivalence to DTMC Discrete time and continuous domain systems 5.2.1 DTMP to PCN conversion 5.2.2 DC-PCN equivalence to DTMP Continuous time and discrete domain systems 5.3.1 CMTC to PCN conversion 5.3.2 P C N t o C T M C Continuous time, continuous domain Other models of interest  Introduction to Behavioural Verification  6.1 6.2 6.3  . . .  86  Events 4.1.1 Event generators 4.1.2 Event synchronizers Types of Uncertainties Computation in Probabilistic Constraint Nets 4.3.1 Sequential and Analog computation 4.3.2 Stochastic Taylor expansion  Models Subsumed by P C N  5.1  6  57 58  Probabilistic Constraint Nets  4.1  5  2.5.3 Event-driven transductions Dynamics Structures  97 97 98 104 104 104 105 107 107 109 110 111  Average-Timed V-Automaton Our Probabilistic Temporal Logic: PATTL Behavioural Verification  iv  112 112 113  7  Behavioural Verification with Average Timed V-Automaton 7.1 V-Automata 7.1.1 Average-timed automata 7.2 Model-Checking Approach 7.2.1 Behavioural Constraint Verification Rules for Discrete-time 7.2.2 Automatic Behaviour Verification 7.3 Generalizing the Approach  116 116 119 123 127 131 133  8  Behavioural Verification with a Probabilistic Temporal Logic 137 8.1 Introduction to C T L and PCTL 138 8.2 Probabilistic Arbitrary-time Temporal Logic (PATL) 140 8.2.1 Common Logical Connectives and Temporal Operators 145 8.3 Probabilistic Arbitrary-time Timed Temporal Logic (PATTL) 146 8.4 Time Structure and Verification Issues 149 8.4.1 The Structure of Time 149 8.5 Model Checking of PATTL over finite PCN and non-continuous time structure . . . 151 8.5.1 Verifying modal formulae in PATTL 152 8.5.2 Model checking of the X operator 153 8.5.3 Model checking of the U operator when r — 0 153 8.5.4 Model checking of the U operator when 0 < r < o o 153 8.5.5 Model checking of the U operator when r = oo 155 8.5.6 Model checking of the <S operator 159 8.6 Model checking of PATTL formulae with continuous time structures 161 8.7 First Order PCN timed temporal logic (FPATTL) 164 T  T  T  r  9  Control Synthesis 9.1 Introduction 9.2 Stochastic and Robust Control 9.2.1 Robust Control of Package Delivery Robot 9.3 Planning under uncertainty 9.4 Introduction to Markov Decision Processes 9.5 MPDs correspondence to PCN 9.5.1 MDP to PCN Conversion 9.5.2 PCN to MDP conversion 9.6 Introduction to Partially Observable MDPs 9.7 POMDPs correspondence to PCN  10 Problems, Solutions and Contributions 10.1 Summary of Contributions 10.2 Future Work 10.2.1 Further Desired Theoretical Development 10.2.2 Further Practical Applications v  169 169 172 174 176 178 180 181 189 202 205 209 209 212 212 213  Appendix A Proofs of Theorems A . l Measure-Theoretical and Topological Structure of Dynamics A.2 Probabilistic Constraint Nets A.3 Modeling in PCN A.4 Behavioural Verification with Average Timed V-Automaton A. 5 Behavioural Verification with a Probabilistic Temporal Logic  215 215 225 230 231 241  Appendix B Modeling and Verifying an Elevator System with Uncertainty B. l Augmented Model B . l . l Continuous model B.1.2 Discrete model B.1.3 Hybrid model B.1.4 Control Design B.1.5 Example of Behavioural Constraint Verification  250 251 252 253 253 254 256  Appendix C Code and output for the MDP systems C l VAMP system C. 2 V A M system C.3 Cat Mouse system  260 260 261 263  Bibliography  273  Index  285  vi  List of Tables 4.1  Generic Types of probabilistic/stochastic models  8.1  Comparing the diverse cases  86 152  vii  List of Figures 1.1 1.2 1.3 1.4 1.5 1.6  The structure of a constraint-based agent system 4 a) Dynamics of a mobile car with uncertain actuators; b) PCN model of Equation 1.1 6 Autonomous robot chasing a series of targets. Various effects of uncertainty in the odometry 7 The problems and proposed solutions 10 Transduction Type Hierarchy 12 The constraint net representing s — f(s) 16  2.1 2.2 2.3  An event trace: each dot depicts a time point Event logic for "or" Simple Transduction with uncertainty  52 56 57  3.1 3.2 3.3 3.4 3.5 3.6 3.8 3.9 3.10 3.11  Gaussian probability distribution as a Generator and random location The probabilistic constraint net representing a Markov model PCN model of an Ito process Combined Operations: cascade, parallel and feedback PCN model of an input/output probabilistic automaton (state denotes either s' or s). Simple PCN for a probabilistic sum Density of dX = -X(X - \)(X - 2)dt + dB Sample path of the system in Figure 2.3 from page 57 Empirical Distribution of Example 2.3 Evolution of the distributions of f(x)  65 65 66 68 68 70 72 75 76 77  4.1 4.2  Two basic modules for event logics: NotEqual and F. A sequential module  88 91  5.1 5.2 5.3 5.4 5.5 5.6  Representation of a DTMC as a PCN Simple DD-PCN with three locations Bayesian Network representation of the temporal dependence Non-Unit delays for the X Y Z DD-PCN Representation of a DTMP as a PCN PCN equivalent of a C T M C  t  viii  98 100 102 103 104 107  6.1  Robot Delivery V-Automaton Specification  7.1 7.2  V-Automaton Specifications (a) goal achievement (b) safety (c) bounded response . 120 Timed V-Automaton Specifications: real-time bounded response 123  8.1 8.2  Comparing the logics extending CTL Algorithm for calculating <%(•)  140 155  9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 9.13 9.14 9.15 9.16 9.17 9.18 9.19 9.20 9.21 9.22 9.23  Comparing a) stability, b) attraction, and c) asymptotic stability Package delivery robotic system One realization of the package delivery system Similarities between MDP and PCN frameworks Control synthesis for the synchfin-PCN class PCN model of an extensional MDP PCN model of an intensional (factored) MDP Value for a state space with two boolean variables pcn2mdp algorithm Control Synthesis algorithm V A M P problem in PCN framework 2TBN for VAMP problem Policy for the vamp PCN VAMP problem in PCN framework: after elimination of variable P 2TBN of the VAMP example: after elimination of variable P Policy for the V A M PCN cat mouse example PCN model for the cat mouse system 2TBN for cat mouse system Policy for the cat mouse PCN Similarities between POMDP and PCN frameworks PCN model of an extensional POMDP Intensional POMDP  172 175 177 180 181 182 184 190 195 196 196 197 198 199 199 200 200 202 203 204 206 207 208  A. 1 Two different distributions generating a stationary distribution with transcendental values B. l B.2 B.3  114  247 251 252  B.4 B.5  The BODY module of the elevator system The Elevator module: continuous components of the elevator system The hybrid model: combining continuous and discrete components of the elevator system The module of the discrete controller: ControM Stochastic State Transition System of the elevator behaviour  Cl  Value for the vamp PCN  262 ix  253 257 258  C.2 Value for the vam PCN C.3 Value function for the cat mouse PCN  '.  x  264 272  Acknowledgements First and foremost, I would like to thank the members of my supervisory committee: Alan Mackworth, Joel Friedman, David Poole, Alan Hu, Marty Puterman, Mark Greenstreet, Paul Gustafson, Dinesh Pai and Harry Joe (chair). Without their contribution and direction, this thesis would not have been what it is now. I am incredibly grateful that I had the opportunity to work with Alan Mackworth as my thesis supervisor and research collaborator. I could not have asked for a better supervisor. Alan's unparalleled insight and perspective inspired me to tackle stimulating new research problems. His continued support (intellectual, emotional and financial) and trust in my abilities also greatly contributed to the success of this thesis. Joel's contribution to this thesis is incalculable. Without his direction and knowledge in the field of mathematics and topology, this work would not have been possible. I would also like to thank David for many challenging discussions which kept me on my toes and forced me to reconsider many concepts and to see them under a different light. Many thanks go to Craig Boutilier, my first research supervisor in LCI. Craig believed in me when I had no experience in the field and he gave me the opportunity to enter the challenging world of computer science and artificial intelligence. He proved to be a great supervisor and friend. Valerie McRae, the LCI secretary acted like a surrogate mother throughout the duration of my studies. She was always there for me, providing help and supportive comments on a regular basis. I am very grateful to her. I thank my family and close friends (you know who you are), for their wishes, belief, support and understanding during what proved to be a longer than expected process. Finally, I thank the University of British Columbia for the Graduate Student Fellowship, NSERC for the Post Graduate Fellowships, FCAR for the Doctoral Fellowship and PRECARN for the scholarships they granted me over the course of my academic career. Financial security throughout my graduate studies provided me with peace of mind that allowed me to devoted my time to my research and I am very thankful for that. ROBERT JR. ST-AUBIN  The University of British  Columbia  June 2005  xi  Je dedis cette these a ma famille et a mes amis qui m'ont offert tout le support dont j'aurais esperer et meme plus.  xn  Chapter 1  Introduction and Motivation Computer-controlled robotic systems are becoming ubiquitous. Until recently, computer controls were mainly used in large industrial applications such as nuclear and chemical plants, assembly lines and forest industries. However, now such systems are everywhere, whether it is in our cars, our refrigerators or in our children's toys. As a larger portion of the population is exposed to these systems, there is a growing need for the design of intelligent, robust, reliable and safe robotic systems. In this body of work, we are interested in the modeling and verification of robotic systems where the inherent uncertainty is explicitly modeled. A robotic system is a dynamical system which can be extremely complex. No tractable model can represent its dynamics and environment perfectly. Due to the limitations in modeling and sensing of such systems, they exhibit uncertainty and very often behave probabilistically. It is therefore important to be able to model and analyze real-time systems while taking into consideration the underlying uncertainty. To address this question, we have developed Probabilistic Constraint Nets (PCN): a framework for the modeling and analysis of systems exhibiting uncertainty. However, before getting into the thick of the subject, it is important that we clarify what we actually mean by the term uncertainty and to define how we intend to model it. It is now well established that probability theory is a body of knowledge (although not the only one) that enables one to reason formally about uncertain events. Many different approaches on probability theory have arisen over time: the approaches that are most discussed in the literature are the objective (classical, frequentist) and subjective (Bayesian) approaches.  1  The key difference between the Bayesian and frequentist approach is in how probability is viewed and assessed. The frequentist approach views probability as a proportion: the probability of a coin landing heads-up is 0.5 because in a long series of coin tosses it lands heads-up half the time. On the other hand, the Bayesian approach views probability as a subjective measure which reflects personal belief. Somebody adopting a Bayesian approach might therefore decide that the probability of a coin landing heads-up is 0.5 because the evidence combined with their belief leads them to believe that the chance of obtaining a head or tail is equal. Bayes' ideas emerged at the same time as frequentist methods were being advanced. However, Bayesian statistics did not develop in parallel with traditional frequentist techniques because the application of these methods is complex and was until recently intractable due to the sheer number of computations often required. The application of Bayesian techniques is therefore relatively new and is growing as a consequence of advances in modern computers and the development of appropriate statistical software. Moreover, Bayesian techniques are gaining momentum in the scientific community as researchers agree more and more on the observation that subjective probability is a natural concept developed by the human mind to quantify the plausibility of events in conditions of uncertainty. One could argue that Bayes' theorem is in fact a natural way of reasoning in updating probability. There has been, and still is, disagreement between the frequentist and Bayesian community. This work does not intend on providing evidence on the "right" approach. Instead, we simply adopt the subjectivist (Bayesian) position regarding the interpretation of probability, which we see as more natural. The reader is referred to [BS94] for a thorough introduction to Bayesian theory. Throughout this dissertation, we will also assume deterministic causality within the world. Hence the uncertainty in the systems of interest can be viewed as arising from the observer's ignorance or his inability to derive accurate models. The probability distributions that will be part of our models will then reflect the observer's belief of the system behaviour. For example, one might try to model a certain system function /(•) but due to incomplete information, she might only be able to observe and reason about the function: g(-) = /(•) + "noise". This user might have a certain belief about the nature of the noise, and thus specify a certain probability distribution on it. Furthermore, we will express causal relationships in the form of deterministic, functional equations, and introduce probabilities through the assumption that certain variables in these equations  2  are unobserved (or random). This reflects Laplace's conception of natural phenomena, according to which nature's laws are deterministic and randomness surfaces due merely to our ignorance of the underlying boundary conditions. It contrasts with the modern (quantum mechanical) conception of physics, according to which all of nature's laws are inherently probabilistic and determinism is but a convenient approximation. However, it seems to be widely accepted within the scientific community that for macro systems, physical laws are deterministic. Quantum mechanics applies to micro-systems, systems which we shall not focus on in this dissertation. We are mainly interested in robotic systems, and we will assume that the laws of physics or mechanics that they follow are deterministic. Let us now conclude this discussion with an interesting passage on the defense of the subjective position, as given in de Finetti's Theory of Probability, [dF74]: The only relevant thing is uncertainty - the extent of our own knowledge and ignorance. The actual of whether or not the events considered are in some sense determined, or known by other people, and so on, is of no consequence. The numerous, different, opposed attempts to put forward particular points of view which, in the opinion of their supporters, would endow Probability Theory with a "nobler" status, or a "more scientific" character, or "firmer" philosophical or logical foundations, have only served to generate confusion and obscurity, and to provoke well-known polemics and disagreements - even between supporters of essentially the same framework. (de Finetti, 1970/1974, Preface, xi-xii)  1.1  Considered Problems  From a systemic point of view, we view a robotic system is the coupling of a robotic agent (or simply a robot or an agent) to its environment. Obviously, the most common agents are robots. However, our framework applies equally to software agents, embedded devices or any animate agents in this world. Formally, an agent consists of the coupling of two distinct constraint-based modules: a body which usually encompasses the various sensors and actuators, and a controller, which is the (mostly) software portion that controls the behaviour of the agent. We view constraint satisfaction  3  Figure 1.1: The structure of a constraint-based agent system as a dynamical process that approaches the solution set of the given constraints asymptotically; where a constraint is a (possibly) implicit relation on a set of variables. Generalizing, we view a constraint-based dynamical system as a dynamical system that approaches the solution set of the given constraints persistently. We will discuss this topic more in depth in the upcoming chapters. With its sensors, the agent's body senses the environment, and reports to the controller on the perceived state of the environment. In turn, the controller, now equipped with an updated report on the state of the environment, sends appropriate control signals to the actuators of the body to perform the required actions which change the state of the world. The corresponding coupled relationship between a robotic agent and its environment is shown in details in Figure 1.1. From this figure, one can see how the coupled agent and environment act on, and react to, each other in a closed-loop system evolving over time. It is important to notice how the system is affected at different levels by the various types of uncertainty, for example, originating from external disturbances, sensor noise and uncertainty in the dynamics. Hence, to design proper robotic systems, one needs to solve the constraint satisfaction problem resulting from joining the knowledge about the various types of uncertainty with the constraints-based modules composing the agent. The notions of constraint programming and of the Constraint Satisfaction Problem (CSP) are important paradigms that have been studied extensively. Typically, CSP's provide problem solvers,  4  which are, more often then not, off-line solvers. Despite the advances in the field of CSP, one area that has yet to be explored thoroughly is the problem of designing dynamical systems in a constraintbased fashion. Dynamical systems should not be handled in an off-line approach but rather should be seen as online constraint satisfying systems. One proposed solution, Constraint Net (CN), was developed for building deterministic hybrid intelligent systems as situated agents [ZM95a]. In CN, a deterministic dynamical system is modeled, in a unitary way, by the coupling of an agent with its environment. In order to handle embedded systems in an efficient way, we must move beyond the typical offline constraint satisfaction model and adopt a model where the solution to a constraint problem is a temporal trace of values, obtained via a non-anticipative transformational process, a transduction, of the input trace over time. Furthermore, the hybrid nature of most embedded systems necessitates that computations be performed on various time structures such as discrete, continuous or eventbased. In addition, the computations should be performed while taking account of the uncertainty present in the system under study. In this thesis, we show how the PCN framework is suited for modeling and reasoning about such systems. Before going any further, let us demonstrate key concepts by introducing an example based on a testbed for radio-controlled soccer playing car-like robots built at the Laboratory for Computational Intelligence (LCI) at the University of British Columbia. For each robot in the testbed, the controller is equipped with both digital and analog devices, hence making it a good example of an hybrid system [SM94]. This example is the stochastic version of the central example presented in the initial work on Constraint Net [Zha94], the framework which we build on. In the original testbed, the robots could move forward and backward with a throttle setting, and were able to make turns by steering their two front wheels. The robots were incapable of moving sideways and the turning radius was restricted by mechanical stops in the steering gear. Zhang and Mackworth modeled these robots as a deterministic system. Here we extend this model to include noise in the actuators, hence introducing uncertainty in the dynamics of the robots. We display, in Figure 1.2a), the configuration of one of the soccer playing robot-cars. In this model, v and a respectively represent the velocity of the car and the current steering angle of the front wheels. At this level of modeling, one can see v and a as control inputs of the system. Extending the deterministic dynamics presented in [Lat91, Zha94], the stochastic dynamics of the  5  X = VCOS(0)+W  ^ = vsin(9)+W 9 = tan(a)/L + W  a  Control Input  Output [•COS -  Ocy)  a) Body Dynamics  b) P C N Model  Figure 1.2: a) Dynamics of a mobile car with uncertain actuators; b) PCN model of Equation 1.1 car can be modeled by the following stochastic differential equations: x = vcos(6) + W , x  t  y = vsin{9) + W?, 6 = v/R + Wf  (1.1)  where (x, y) is the coordinate of the position of the tail of the car, 9 is the heading direction of the car, R = L/tan(a) is turning radius given the length of the car L and W}, i € {x, y, 6} is a one-dimensional "white noise" affecting the location of the car along with its heading angle. 1  So far, we have not yet directly addressed the importance of explicitly modeling a systems's uncertainty. However, many factors, such as industrial production variations, incomplete information or even numerical precision errors, lead to the conclusion that uncertainty is inherently present in any real physical system. Failing to model the effects of these various sources of uncertainty can have a drastic influence on the modeled system's behaviour. Although the uncertainty model might not mimic exactly the nature of the uncertainty within the system under study, completely ignoring it in the modeling task can be even more disastrous. For instance, when designing an airplane controller, it is crucial to take the uncertainty of the wind speed into account, since a given controller might be optimal for a certain wind condition but might behave dangerously for other 'We will formally discuss the notion of white noise when we introduce the concept of Brownian motion in §2.1.4  6  Figure 1.3: Autonomous robot chasing a series of targets. Various effects of uncertainty in the odometry. wind conditions. As another example, consider a simulation of the behaviour of an autonomous robotic system traveling to a series of goal locations, as shown in Figure 1.3 (goals are shown as o in the figure). In the situation where the uncertainty in the odometry of the robot is ignored, one can see that whether or not a "reset" function of the belief state is used to relocate the robot when it has reached an intermediate goal, the robot's belief of its location will diverge from its true position as time goes by, as it is increasingly affected by the odometric uncertainty. From this example, we can see that for the robot to "know" where it is and eventually localize itself, a model of uncertainty is needed. As shown in Figure 1.1, we consider three common ways in which the uncertainty can enter a system: 1. Uncertainty in the Dynamics: Uncertainty can enter a system through the model of the physical system. High-order and nonlinear dynamics can be unknown or deliberately ignored in order to keep the model simple and tractable. This lack of knowledge of the true underlying dynamics introduces some uncertainty in the model, thus rendering the behaviour somewhat  7  unpredictable. Let us revisit the previous mobile robot example of Figure 1.3.' Consider for instance that some nonlinear effects in the actuators of the robot need to be modeled in order to precisely capture their dynamics. To simplify the model, one might want to omit these effects. Yet, not accounting for the uncertainty introduced in the dynamics by the simpler model might lead to imperfect movement of the robot. As a result, the robot might have a false belief of where it is, even though it reaches the intermediate goals. Another example of such uncertainty is the high-order effects of evaporation of water in a steam boiler system which are usually ignored to simplify the model. 2. External Disturbances: Disturbances (not attributed to the inaccuracy of the model of the agent itself), can also enter in the dynamics of the robot. Such uncertainty is caused by external disturbances that are outside of the control of the system. E.g., barometric pressure or wind speed are elements outside of the body's dynamics which can influence the behaviour of an airplane system. 3. Sensor Noise: This type of uncertainty enters the system through imprecise sensor measurements. For example, sensor noise could come from the sonar readings of a mobile robot. Now that we have briefly introduced the notion of uncertainty, let us discuss the dynamical nature of the systems we are interested in. Considering the dynamic aspect of the problem, we observe that the relationship of a robot with its environment is one that changes (somewhat unpredictably due to the underlying uncertainty) over time. Therefore, to reason about the overall behaviour of a robotic system, we need a model which characterizes the behaviours of its various components (body, controller and environment) and derives the overall behaviour of the complete robotic system. As described above, the actions of a robotic agent are controlled by a software component called the controller. Robotic systems are in fact controlled dynamical systems. Such systems are designed to meet certain requirements such that their behaviour must satisfy certain behavioural constraints. Typical requirements on the behaviour of robotic systems include safety: a system should never be in a certain undesirable situation; reachability: a system should eventually reach a given predefined  8  goal and persistence: a system should approach a given goal infinitely often. In order to characterize the desired properties of a system, it is important to be able to represent such behavioural constraints. Therefore, a formal language for specifying behavioural constraints is needed. Moreover, given a representation of a behavioural constraint, a formal method for behaviour verification is essential for ensuring the correctness of the system with respect to these specifications. However, both the specification and the verification task must be performed while taking into account the uncertainty in the systems. Behavioural constraint specifications must be able to express uncertain (or probabilistic) requirements, while the verification methods must be able to handle the fact that while some behaviours might diverge from the specifications, the system will obey them with respect to some given measure (e.g., on average or according to some probability threshold). Another challenge in the modeling and analysis of robotic systems is the design of the controller, most often referred to as the control synthesis problem. Given a model for the dynamics of the robotic agent and its environment, the task is to automatically generate a controller ensuring that the system's behaviour will follow a certain specification. This problem is especially complex in the presence of uncertainty. Although the field of stochastic optimal control has made great leaps in the recent past, the problem of control synthesis on uncertain robotic systems is still mostly unexplored. In summary, we propose to address four problems that are at the core of designing and analyzing uncertain robotic systems: 1. Modeling: how to model a robotic system exhibiting uncertainty? 2. Behavioural Constraint Specifications: how to specify properties that the uncertain behaviour of a robotic system should display? 3. Behavioural Verification: how to guarantee that a robot will do the right thing, where the right thing is defined according to the designer's favorite measure (e.g., safety or reachability) for unpredictable behaviour? 4 . Control Synthesis: how to automatically generate a controller that will act "optimally" given the uncertainty present in the robotic system? In Figure 1.4 we present a visual representation of the summary of the problems we consider in this thesis along with the proposed solutions. These proposed solutions are discussed in the 9  Will the robot do therightthing? On average? Probabilistically?  Behaviour! Vcrii.ca.um  Uncertain Koholii'l  Requirements S p e c i f i c a t i o n  SVSIL'IIIS  Average timed V-Automata  PCN Model  PATTL  What is the possible^ behaviour of the robot?  N  Control Synthesis  What is the right thing for the robot to do considering its uncertain environment?  Stochastic Control  Robust,  How to make the robot do therightthing, even under uncertainty  Figure 1.4: The problems and proposed solutions following section.  1.2  Proposed Solutions to the Considered Problems  When initially considering the task of modeling uncertain hybrid systems, our approach was motivated by numerous considerations. The first one corresponded to the fact that hybrid systems consist of interacting discrete and continuous components, both in terms of time and domain. Therefore, rather than simply developing a model with fixed time and domain structures, an encompassing model for hybrid systems should be developed on both abstract time structures and abstract data types. Secondly, most hybrid systems are complex and need to be specified with multiple interacting components. Hence, an efficient model for hybrid systems should allow hierarchical and modular modeling. Third, by combining discrete and continuous components, hybrid systems are generalizations of those basic systems. Therefore, any framework modeling a hybrid system should be at least as powerful as the existing computational model used to model the basic systems. Finally, uncertainty is inherent in any realistic hybrid system. Whether it is through sensor noise, external disturbances or imprecision in the model of the dynamics, uncertainty has an important effect on the behaviour of hybrid systems. Therefore, a model for a hybrid system should support probabilistic  10  and non-deterministic modeling. In short, a model for hybrid systems should be unitary, modular, powerful and allow for explicit modeling of uncertainty within the system. As time is an intrinsic component of a dynamical hybrid system, we start with a general definition of time. We view time as a linearly ordered set. In order to reason about the evolution of time, we associate a metric distance with any two time points and a measure with some intervals of time points. By using such a general time structure, we create an abstraction for event-based as well as discrete and continuous time. In order to study discrete and continuous domains in a unitary fashion, we cast domain structures within abstract algebra topology and measure theory. Given such a time structure and a domain structure, we define two basic types of element in dynamical systems: stochastic traces that are stochastic functions from time to domains, and transductions that are mappings from stochastic traces to stochastic traces with the causal restriction (which is called adapted in the measure theory literature), i.e., the output value at any time is (stochastically) determined only by its input values up to (and including) that time. We will define these notions in detail in Chapter 2. Using these notions, we develop the Probabilistic Constraint Net (PCN) model, an extension of the Constraint Net (CN) model [ZM95a]. PCN is built on an abstract dynamics structure composed of a multi-sorted set of stochastic trace spaces and a set of basic transductions. Basic transductions are the building blocks of our framework. By combining basic transductions we are able to build complex models of systems. The set of basic transductions is made of the following components which will be formally defined in Chapter 2: transliterations (memory-less combinational processes), unit and transport delays and generators. Generators allow for the modeling of uncertainty by introducing random variables in the model. We differentiate between deterministic and probabilistic transductions depending on whether or not they encompass a generator. Compound transductions of each type are built by combining simple transductions of the same type with transliterations and delays. Figure 1.5 shows the hierarchy of transductions within the PCN framework. Note that non-deterministic transductions are not represented on this diagram as these transductions are obtained from any transduction with a hidden input. We will clearly explain the concept of hidden inputs when we describe the PCN syntax in Chapter 3. As an example of what a transduction can be, consider the function f[x) = 2x. This function is a deterministic transduction (in fact it is simply a transliteration) while g(x) = f(x) + N (where N is a Gaussian random vari-  11  Compound ]  [ Simple '  Compound  Simple  }  Basic Transductions  Figure 1.5: Transduction Type Hierarchy able), is a probabilistic transduction made of the transliteration / along with the generator TV for a Gaussian random variable. Our framework also includes event-driven transductions. Event-driven transductions act as bridges between continuous and discrete time components, or as synchronizers among asynchronous components. This will allow the user to model hybrid systems possessing multiple clocks. Syntactically, a probabilistic constraint net is represented as a bipartite graph with two types of nodes: locations and transductions, and with a set of connections between locations and transductions. Locations can be seen as variables of the system while transductions are the transformational processes (functions) that are applied to these variables. Semantically, a probabilistic constraint net represents a set of stochastic equations, with locations as variables (possibly random variables if they are output of probabilistic transductions), and transductions as (possibly random) functions. The semantics of a probabilistic constraint net model, with the values over time of each location denoting a stochastic trace, is the least solution of the set of equations. Since uncertainty is present in the model, a solution to the set of equations is a random process that can be depicted by its probability distribution. As systems grow in complexity, it becomes essential to divide them into simpler components interacting together to create the system as a whole. In the PCN framework, we define a module as a probabilistic constraint net with a set of locations serving as its interface. Using modular and aggregation operators on modules, it is then possible to obtain a hierarchical model of the system  12  under study. The semantics of hierarchical systems can then be obtained from the semantics of their subsystems (modules) and their connections. As a simple example of a PCN model, one can denote Equation 1.1 as a probabilistic constraint net in which sin, cos, tan and * are transliterations (basic transductions). The graphical representation of this PCN is shown in Figure 1.2(b). We will clearly define the meaning of each component of the graphical representation of a PCN when we formally introduce the syntax in § 3.1.1. For this system, one can build a PCN module using the locations (variables) v, a, x, y, 9 as its interface. In general, we can model a control system as a module that can be further decomposed into a hierarchy of interactive modules. The higher level control signals are built on event-driven transductions while the lower levels are analog control components. Note that in our framework, unlike in most modeling paradigms, the environment of the robot can also be modeled as its own module. From Figure 1.1, one can observe that a robotic system as a whole can be obtained by integrating a module for the body, the controller and the environment, which can be represented, in general, by the following equations:  X = BODY(U, Y), U = CONTROLLER(X,  Y), Y =  ENVIRONMENT(X).  Contributions The general contribution of this thesis is to augments Constraint Nets with probabilistic event generators to produce Probabilistic Constraint Nets (PCN), an intuitive and very general framework for modeling and verifying probabilistic hybrid systems. More specifically, we establish a unified foundation for hybrid dynamical systems exhibiting uncertainty and further propose that an integrated approach to the design and analysis of robotic systems and behaviours should be taken and that uncertainty within the systems should be expressed explicitly and considered throughout the whole analysis. PCN introduces an abstraction and a unitary framework to model hybrid dynamical systems exhibiting uncertainty. PCN is modular and hierarchical, that is, the dynamics of the environment as well as the dynamics of the system can be modeled individually, and then, integrated using aggregation operators provided by the framework. Moreover, PCN supports multiple levels of abstraction, based on abstract algebra, topology and measure theory, to model and analyze a system and its underlying uncertainty at different levels of detail. Due to its rigorous measure theoretic and alge13  braic foundations, PCN can be used to define programming semantics of real-time languages for uncertain controlled dynamical systems. Second, to address the specification of behavioural constraints, we develop both a probabilistic arbitrary-timed branching time logic (PATTL) and average timed V-automata as specification languages. PATTL is a probabilistic branching time logic developed on abstract time and domain structures which extends already defined logics such as the Computation Tree Logic (CTL) [Eme90] and the Probabilistic Computational Tree Logic (PCTL) of Hansson and Jonsson [HJ94]. Average timed V-automata extend timed V-automata to accept timed traces where the average behaviours of the system is considered. These finite automata are powerful enough to specify properties of sequential and timed behaviours of hybrid systems, such as safety, reachability, persistence and real-time response corresponding to the on average constraint. Third, we develop a formal verification method for average timed V-automata specification, by combining a generalized model checking technique for automata with a generalized stability analysis method for uncertain dynamical systems. This verification method can be semi-automated for discrete time systems and further automated for finite domain systems. Furthermore, we show that existing verification methods can be extended to apply to the task of verifying PATTL requirements on PCN models. Finally, we discuss an approach to control synthesis for uncertain dynamical system emerging from stochastic and robust control as well as from the dynamic programming field of research.  1.3  Related Work  In this section, we present what the author believes to be some of the most relevant work done in the area of modeling and verification of probabilistic systems. We first provide a survey of the framework on which PCN is based: Constraint Nets. Following this short introduction, we present the notion of probabilistic systems and some of the most common models used to represent such systems. We conclude with a survey of the research performed on the verification of behavioural constraints of probabilistic systems.  1.3.1  Review of Constraint Nets  The C N modeling framework is built on a topological view of time and domain structures along with notions of traces, events and transductions. C N is a modeling framework for deterministic 14  system. It does not allow for the explicit modeling of uncertainty. The set of basic transductions in C N is limited to transliterations and delays. For the remainder of this dissertation, when we refer to a deterministic system within the PCN framework, we will mean a system composed of only transliterations and delays, i.e., a system which does not include any generators. The following definitions, which we review below for the sake of clarity, represent the foundation for the notion of time and domain in the Constraint Net framework. We retain these concepts and extend them to develop the Probabilistic Constraint Net model. We refer the reader to Chapter 3: Topological Structure of Dynamics, from [Zha94], for a thorough introduction to these concepts and more concerning the CN modeling framework. Within (P)CN, we view time and domains as fully abstract concepts, defined as follows: • Time is presented as a linearly ordered set (T, <) with a least element to, and a metric d on the set T . With this abstract representation of time, one can model discrete, continuous or event-based systems. • Simple and composite domains denote, respectively, simple data types (e.g. reals, integers, Boolean and characters) and structured data types ( e.g. arrays, vectors or objects). A simple domain is represented as a pair (A U { ± , 4 } , ^ ) where A is a set, J . ^ ^ A means undefined in A, and d& is a metric on A. Composite domains are obtained by the product of simple domains. With.this abstract notion, domains can be numerical or symbolic, discrete or continuous. We also view the behaviour of a dynamical system as a set of traces, where the trace of each variable is defined as follow: • Traces can intuitively be perceived as changes of values over time. Formally, a mapping v : T —> A from time T to range A is called a trace. An event trace is a trace with a Boolean range. An event in an event trace is a transition from 0 to 1 or from 1 to 0. It is important to note that for a deterministic system, the initial value of a trace completely defines the whole trace. Indeed, given the (deterministic) dynamics of a system, with the initial value of each variable, one can completely predict the value of each variable over the time history of the system. In the presence of uncertainty, however, this is not true. Given the model of the 15  Figure 1.6: The constraint net representing s = f(s) uncertain dynamics of a system and its initial values, one can only predict the possible trajectories, i.e., the set of all traces that the system can take. Therefore, one has to be extremely careful when designing systems exhibiting uncertainty since although some trajectories might be acceptable for a system, chances are that some undesired trajectory can be achieved with the current conditions. This is related to the notion of behaviour verification, which we will discuss in Chapter 6-8. In the C N framework, functions are referred to as transductions. Formally, these are defined as: • Transductions are causal mappings from inputs to outputs over time, either operating according to a certain reference time or activated by external events. Note that in PCN, the notion of causal mapping will be replaced by adapted mapping, its measure theoretic equivalent. However, both concepts have a similar intuitive meaning. The class of simple transductions contains transliterations and delays. A transliteration is a pointwise extension of a function. It can be seen as a transformational process without internal state (memory). A delay can be seen as a memory cell. A constraint net consists of a finite set of locations, a finite set of transductions and a finite set of connections. Formally, a constraint net is a triple CN — (Lc, Td, Cn), where Lc is a finite set of locations, Td is a finite set of labels of transductions, each with an output port and a set of input ports, Cn is a set of connections between locations. A location can be regarded as a wire, a memory cell or a variable. One of the advantages of C N is its graphical representation which allows one to represent a constraint net by a bipartite graph where locations are depicted by circles, transductions by boxes and connections by arcs. For example, the graph in Figure 1.6 represents the constraint net with a continuous time structure for the differential equation s = f(s) with initial value s(to) = soSemantically, a constraint net represents a set of equations, where locations are variables and 16  transductions are functions. The semantics of the constraint net, with each location denoting a trace, is the least solution of the set of equations.  Specification and Verification Along with the formal modeling framework of Constraint Nets, Zhang [Zha94] developed two specification languages to represent the behavior requirements of a system: Timed Linear Temporal Logic (TLTL) and Timed V'-automata. TLTL is a generalization of Linear Temporal Logic [MP91] where "linear" indicates linear orders and "timed" stands for metric distances between time points. Even though TLTL allows for formal specification requirements, there is no general procedure for verifying the behavior of a given system. However, TLTL has been proven to be very useful when used within the problem of control synthesis as we will show in next section. One of the most popular alternatives to temporal logics for expressing the behavior of a dynamical system is automata. In this framework, the behavior of a system can be seen as a language, and the specification can be represented as an automaton. Given such a representation, the verification step amounts to showing the inclusion of the behavior language within the language accepted by the automaton. Timed V-automata extend V-automata [MP87] by accepting timed traces. They provide a graphical representation, which is more intuitive and sometimes simpler than (temporal) logics. Furthermore, a formal verification method (via a set of sound and complete verification rules), based on model checking and stability analysis, was developed. It was shown that, given a constraint net model of a discrete time system, a set of state formulae can be deduced and then checked using an automatic or interactive theorem prover. It was also demonstrated that, if the system has a finite state space, then the verification rules can be used to deduce a fully automatic verification algorithm with polynomial time complexity in the size of the specification and the size of the system.  Constraint-Based Dynamical Systems It is argued that most dynamical systems are inherently constraint-based, where constraints may range from physical limitations or environmental restrictions to safety requirements. In the C N framework, constraint satisfaction is seen as a dynamical process that asymptotically approaches 17  the solution set of the given (possibly time-varying) constraints. A constraint solver is viewed as a constraint net whose behavior is a dynamical process that is asymptotically stable at the solution set of the constraints, i.e., whenever the system exits the solution set due to some disturbance, it will eventually return to a state which is part of the set satisfying the constraints. The behavior of such a constraint-based system can be specified using TLTL or timed V-automata. For example, using the specification language of TLTL, a persistence specification would be represented by D O C which e  means that for C , the e-neighborhood of the solution set of the set of constraints C will always (•) £  eventually (O) be reached. Many constraint methods can be implemented in the C N framework. Two types of problems are mentioned in [Zha94], global consistency and optimization. Typically, global consistency denotes the problem of finding a solution satisfying all the given constraints. Thus, global consistency corresponds to solving hard constraints. On the other hand, unconstrained optimization refers to the problem of minimizing an energy function of n arguments £ : IR —> K and corresponds to solving n  soft constraints. Finally, another type of constraint satisfaction problem is introduced: constrained optimization. This type of problem consists of solving (soft) constraints subject to the satisfaction of a set of hard constraints. Typically, there are two different types of constraint methods: discrete relaxation and differential optimization. The former can be represented as a state transition system while the latter can be represented by a state integration system. In [Zha94], a few constraint methods are introduced; namely the (discrete) projection method for global consistency; the (discrete) Newton's method along with the (continuous) gradient method for unconstrained optimization and finally, the (continuous) penalty method and the (continuous) Lagrange multiplier method for constrained  optimization.  Control Synthesis The problem of control synthesis is, given some requirements specification for the behavior of a dynamical system along with models of the environment and the body, to generate a controller inducing a behavior which satisfies the specification. With C N models of the environment and the body, Zhang and Mackworth [ZM95b] use TLTL for the requirement specifications, which are restricted to constraint-based specifications such as reachability, safety and persistence. They then  18  synthesize the controller using constraint methods such as those presented in the previous section. Basically, they view controllers as embedded real-time constraint solvers.  1.3.2  Modeling Probabilistic Systems  We are now ready to introduce our new paradigm, PCN, which extends C N to explicitly model and reason with uncertainty inherent in dynamical systems. However, let us first start by defining the more general class of non-deterministic systems. As opposed to deterministic systems, for which the behaviors exhibit no randomness and thus can be predicted perfectly, non-deterministic systems present inherent incomplete description which renders state transitions unpredictable. For a given system, when the nature of the underlying randomness of the transitions is unknown, and is not estimated by a probability distribution over states, we say that the system is non-deterministic. In general, the occurrence of non-determinism is due to intervention from the environment over which the system has no control or to the asynchronicity of concurrent processes. On the other hand, when the randomness of the state transitions of the systems is modeled, we talk about the notion of probabilistic or stochastic systems. Probabilistic systems are divided into two classes. We call a probabilistic system purely probabilistic if, for every state of the system, there is exactly one specified probability distribution over the next possible states. A system that exhibits both 2  probabilistic and non-deterministic behavior is referred to as a generalized probabilistic system. Let us now introduce the most commonly used frameworks for modeling systems with uncertainty.  Probabilistic Systems One of the most commonly used formalisms for modeling probabilistic systems is the finite-state  discrete-time Markov chain. A Markov chain has a specified initial state so , and for each state 3  included in the state space  S, there is an assignment of truth values to atomic propositions. The set of true atomic propositions Note that deterministic systems can also be seen as purely probabilistic models over the trivial one-point probability space. This means that for each state, the probability distribution is restricted to 1 for the realized state and 0 for all other states. The initial state of the system can either be given and unique or else a prior probability distribution over all "starting" states is given. 2  3  19  uniquely identifies each state. Conceptually, a Markov chain is the representation of a probabilistic system. At each time step, the system changes state following a probability distribution given by the transition probability function P : S x S —» [0,1]. Paths, which are also called execution sequences, arise from resolving the probabilistic choices. Formally, a path in a Markov chain M is an infinite sequence ir — s where  SQ  is the initial state of M, s» are states and P(si,  Si+i)  0  Si •• •  0 V i Finite paths can also be  >  extended to infinite ones by simply repeating the last state infinitely. The probability of a given finite path it, starting at 7r(0) in M is given by P (ir) = P(n(0), n(l))P(n(l), p  7r(2)) • • • , where  n(i) is the i-th state of the path n.  Generalized Probabilistic Systems The modeling of a probabilistic system requires the full specification of the transition probabilities. However, in many practical settings, estimating accurately the transition probabilities can be a complex task. One might want to avoid such a task and leave some of the transition probabilities unspecified, thus modeling those as non-deterministic transitions. Therefore, for many systems, a formalism allowing us to model behavior that is both probabilistic and non-deterministic is desirable. Many models that take into account probability and non-determinism have been recently presented in the computer science literature. An extension of the Markov chain model presented earlier is the Markov Decision Process (MDP). MDPs were introduced by Bellman [Bel57] and Howard [How60] and they have been the subject of much research in Decision Theoretic Planning (Al) [BDH99] and Operation Research. The Independent Choice Logic of [Poo97] has been introduced as a model for multiple agents under uncertainty. Inspired by probabilistic Horn abduction [Poo93], game theory [Mye91, Ord86], Markov Decision Processes (MDP) [Put94] and Bayesian Networks [Pea85, Pea88], it provides a natural and concise representation of agents under uncertainty. A logical representation is used to tackle the probabilistic issue. [CBT00] introduced DTGolog, a framework based on MDPs with the Golog programming language [LRL 97]. They present a framework +  that combines decision theoretic planning with agent programming. Several related models for generalized probabilistic systems include probabilistic finite-state programs [PZ86], concurrent Markov chains [Var85], Probabilistic Non-Deterministic Systems  20  (PNS) [BdA95], Timed Probabilistic Non-Deterministic Systems (TPNS) [dA97b] and Coloured Petri Nets [Jen81, Jen97]. TPNSs are similar to PNSs, with the distinction that TPNSs allow modeling of generalized probabilistic systems in which state transitions have different duration. Coloured Petri Nets (CP-nets or CPNs) is a modeling language developed for systems in which communication, synchronization and resource sharing play an important role. CP-nets combine the strengths of ordinary Petri nets with the strengths of a high-level programming language. Petri nets provide the primitives for process interaction, while the programming language provides the primitives for the definition of data types and the manipulations of data values. The syntax of Petri Nets (and its generalization Coloured Petri Nets) is very similar to that of PCN, i.e., a bipartite graph. However, the semantics of PCN is for maximum parallelism, while the semantics of Petri Nets is for concurrency. Moreover, Petri Nets are not as general as PCNs since timed Petri Nets are restricted to a single global clock while our framework allows for the modeling of hybrid systems with multiple clocks.  1.3.3  Requirement Specifications and Verification Techniques  The modeling of a probabilistic system focuses mainly on the structure and components of the system. However, no matter how fine grained the resulting model is, the overall behavior of the system cannot be fully specified. It is often very important to be able to impose restrictions on the behavior of the system. These restrictions represent global properties that should continuously hold in the system under study. For example, a requirements specification for a coffee delivery robot could be that coffee will be delivered within a bounded period of time after reception of the request. Requirements specifications restrict the behavior of a system by requiring (or forbidding) the system to be in certain states. These specifications become essential when building safe and reliable systems. Therefore, formal methods for expressing requirements specification and verifying that they hold are called for. There are two main approaches to the problems of modeling and verifying systems: the first one consists of a single language L, used for both the modeling and the specifications of the requirements of the system. The verification task then amounts to showing that the set of behaviors of the model LM is a subset of the behavior allowed by the specification Ls, i.e., LM Q LS C L . The 21  second approach uses two different languages: a modeling language M and a specification language S. For a given system A, the verification procedure then amounts to showing that MA entails SA, i.e., MA (= SA- Since the power of a specification language and the simplicity of the verification procedure are inversely related, we have to reach a compromise between the expressibility of the specification language and the applicability of the verification method. Using the same language for modeling and specification might render the verification procedure infeasible for complex dynamical systems. As was the case for the constraint net framework, we will, in this dissertation, use two different languages for modeling and specification. In this section, we present several types of formalism that have been used to specify probabilistic properties of systems along with algorithms used to verify those properties. We restrict our survey to methods which use different languages for modeling and specification.  Qualitative Verification and Temporal Logics Given a probabilistic system P, showing that a property represented by a temporal formula ip is fulfilled by all computations of the system, is referred to as qualitative verification. This amounts to showing that ip is satisfied with probability 1. A lot of work has been done on the subject of qualitative verification. In this section, we attempt to summarize the most relevant research from the literature. Three temporal logics evaluated on Markov chains were introduced by [LS82]. In this work, they present a linear time system which follows a linear history while permitting reference to untaken alternatives. [Pnu83] presented an alternative approach based on standard linear time logics. However, instead of specifying a new temporal logic, he introduced the notion of extreme fairness. This concept of fairness imposes fairness on probabilistic choices. [HS84] introduced a system which is based on branching time temporal logics interpreted on Markov chains. Probably the first one to raise the question of qualitative verification, [Var85] solved the problem on systems modeled with concurrent Markov Chains by automata theoretic methods which he extended to take into account the probabilistic nature of the problem. The complexity of the algorithm presented was showed by [CY88] to be doubly exponential in the size of the temporal property. They established the optimality of Vardi's algorithm by proving that the lower bound for this problem was doubly exponential.  22  [PZ93] extended previous work [PZ86] and introduced a qualitative verification method for probabilistic systems. The specifications are written using a restricted linear time temporal logic (RTL) which allows for all the operators of T L except for the until future operator. Unlike the algorithm of [CY88], the ones presented in [PZ86, PZ93] are of single-exponential complexity in the size of the specification. The reason for this diminution in the complexity is due to the use of RTL as a specification language. [PZ93] also present the notion of a-fairness, an improvement on the concept of extreme fairness of [Pnu83]. Real-time systems with continuous random delays and discrete probability and time have been modeled via the Real-time probabilistic systems of [ACD91]. This work introduced an algorithm for verifying qualitative specifications written in the real-time temporal logic (TCTL) of [ACD90]. Later, [ACD92] extended this work for continuous-time systems with specifications using the notion of timed automata introduced in [AD90].  Quantitative Verification and Temporal Logics In contrast with qualitative verification which attempts to prove that a certain property holds for every computation with probability one, quantitative verification refers to determining the probability with which a property is satisfied within a given system. In the last few years, there has been a significant amount of research done in the area of probabilistic model checking. Two of the most popular temporal logic approaches are linear-time logics and branching-time logics. The former considers time to be a linear sequence while the latter adopts a tree structure time, allowing some instances to have more than one successor. The choice between linear and branching models should be dictated by the type of properties one wishes to study. For 4  this work, we intend on modeling probabilistic systems, and since probability measures on system behaviors are similar in structure to the path quantifiers used in branching-time temporal logics, we will focus on these logics exclusively. In this section, we introduce a few temporal logics based on the branching-time temporal logics CTL and CTL* [CE81, BAPM83, EH85]. The underlying model of CTL is a tree of all possible computations. Probabilistic real time Computational Tree Logic (PCTL) [HJ94] was the earliest of many ex4  For a more in depth discussion of this issue, the reader is referred to [Lam80].  23  tensions made to CTL and CTL*. It considered systems modeled as discrete Markov chains and was developed to specify the probability of satisfying temporal formulae within a given number of chain transitions. The language allows formulae such as: O^^ip.  This formula expresses the property  that, with probability of at least 0.2, the formula ip will become true within 20 state transitions. The algorithm presented in this work has polynomial time complexity in the size of both the formula and the Markov chain. PCTL was later extended by [Han94] to take into account non-determinism and provide a more refined model of time and yielded an algorithm with time complexity exponential in the size of the system. Later, [ASB 95] introduced pCTL*, a probabilistic variant of CTL* where the operator P, used +  to express bounds on the probability of systems behaviors, is added. They also introduce the logic pCTL which is essentially identical to the logic PCTL of [HJ94]. This work considered two different types of system models: discrete Markov processes and generalized Markov processes. Furthermore, although they show, based on results arising from real closed field theory, that model checking with generalized Markov processes is elementary decidable, no practical verification algorithm is presented. A variant of pCTL* was later presented by [BdA95] which extended the logic to systems with non-determinism. They show that model checking for PCTL and PCTL* on PNSs can be done in time polynomial in the size of the PNS. However, in terms of the size of the formula, linear and doubly exponential times are respectively required for the PCTL and PCTL* model checking. Based on PCTL, the Probabilistic Branching Time logic (PBTL) was introduced by [BK97, BK98a, BK98b] and was intended for systems where fairness constraints are imposed. Syntactically, PBTL is almost identical to PCTL with the addition of universal and existential quantifiers over what they call adversaries .  Baier and Kwiatkowska gave a model checking algorithm for PNSs with  5  PBTL specifications. It is based on the algorithm from [BdA95] and has the same time complexity (polynomial in the size of the system and linear in the size of the formula). Recently, [dAKN 00] developed a symbolic model checker for PBTL using +  Multi-Terminal  Binary Decision Diagrams (MTBDD) [CFZ96] which are an extension of Bryant's Boolean De6  cision Diagram (BDD) [Bry86]. They adopt the Kronecker representation of [Pla85] which yields a very compact MTBDD encoding of the system. Their tool allows the model checking of purely In the literature, adversaries are also referred to strategies [BdA95] or schedulers [Var85, BK97]. These diagrams are also known as Algebraic Decision Diagrams (ADD) [BFG+93]  5 6  24  probabilistic systems of up to 10  states, and generalized probabilistic systems of up to 1 0  3 0  states.  It was shown that the model checking of qualitative properties could be done very rapidly. Furthermore, since P C T L is a subset of P B T L , their tool can also be used to verify properties expressed in that logic. In all the logics discussed so far, the representation of a single time unit is interpreted as an instantaneous transition in the model of the system. However, when modeling real-time systems, this may not be desirable, since it is possible for different events to take different amounts of time. A s mentioned above, T P N S were developed exactly to address that issue. Along with the T P N S model, [dA97a] introduced the logic pTL*. The motivation behind this work is to be able to study performance and reliability properties of probabilistic systems. p T L * extends C T L * with two new operators: P as describe earlier and D which is used to express bounds on the average time between events. The time complexity of the verification algorithm presented in this work has been showed to be polynomial in the size of the system and doubly exponential in the size of the specification formula.  Probabilistic Bisimulation Bisimulation is a very important concept in the field of concurrent systems [DEP02, Eda95, LS91]. Bisimulation is one of the most popular method for process equivalence as it enjoys a fixed-point characterization. With bisimulation, one obtains an equivalence relation which allows for the comparison of states of a system with each other. Desharnais et al. showed that bisimulation was sound and complete for P C T L * and C S L . That is, two states will satisfy the same set of P C T L * or C S L formula if and only if they are bisimilar [DGJP02, DP03]. This provides an interesting approach to the verification of properties satisfied by a model as one can use a simpler, yet bisimilar, model to prove the satisfaction of the {property.  25  Chapter 2  Measure-Theoretical and Topological Structure of Dynamics In this chapter, we present a measure-theoretical and topological approach to the structure of dynamical systems exhibiting uncertainty. We start with an introduction to probability, measure and topological theory as these concepts will play a central role in defining the structure of dynamics. The proofs of these results, along with any other formal result presented throughout this dissertation, are given in Appendix A. Within the field of topological theory which is broad and vast, we focus on two particular types of topology: partial order topology and metric topology. Based on these two types of topology and on measure-theoretic concepts, we formalize the key constituents of our model of probabilistic dynamical systems: time, domain and behaviour structures. We will refer to stochastic traces when discussing the (uncertain) evolution of systems over time. In a probabilistic setting, a stochastic trace denotes a set of traces which defines a system's behaviour. One of the most important concepts introduced in this chapter is one of (probabilistic) transduction, that we will define as a causal mapping from stochastic traces to stochastic traces. To conclude this chapter, we define abstract dynamics structures.  2.1  General Topology, Partial Order, Metric Space and Measure Theory  The Probabilistic Constraint Net framework has been developed to model probabilistic systems and processes which deterministic frameworks cannot adequately describe. With the PCN framework,  26  we want to account explicitly for the sources of uncertainty in the systems of interest. Historically, probability theory has been developed to assign probabilities to events of interest associated with the outcome of some experiment. Probability (measure) theory appears to be ideally suited to serve as a foundation for the PCN framework. Before introducing the PCN framework formally, we wish to provide an overview of some of the mathematical preliminaries in general topology theory and some of the elementary results of probability and measure theory that will be used throughout this dissertation. General topology allows us to reason about convergence, connectivity and continuity while probability and measure theory are tools that enable us to reason about integration in arbitrary measure spaces. For a more comprehensive introduction to the mathematical foundations introduced here, the reader is referred to [Gem67, Hen88, Vic89, MA86, War72, Roy88] while we suggest [Bil86, Bre68, Wil91, Rud66] for a more thorough training in measure and probability theory. Since all these notions are well known results, we quote them freely in this document. Moreover, some of the mathematical preliminaries introduced here are also components of the C N framework. Hence we reproduce some of the definitions introduced in Chapter 3 of [Zha94]. The fundamental concept of probability theory is a general space of outcomes, called a sample space, Q, which contains all possible outcomes of the experiment conducted. We will denote a single elementary outcome as u>, with u> £ Q. Now let us define A as a specific event of interest, a set of outcomes of the experiment: A C $7. We will assume that such experiments are of the form of a probability space (Cl, J , P) where Q is the sample space, J is a cr-algebra on Q and P is a 7  7  probability measure on (Q, T). We will define these notions shortly but let us first introduce the notion of topological space, a concept central to the C N formalism and thus to the PCN framework as well. Definition 2.1  (Topology and Topological space) Let X be a set and 0 be the empty set. A collec-  tion T of subsets of X is said to be a topology on X iff the following conditions are satisfied: • X G r and 0 • IfXi 6 r, X  2  e r. G T, then X\ n X  2  e r.  'Recall that in view of our Bayesian approach to probability theory, we view uncertainty as arising from the user's lack of knowledge or as a result of an overly abstracted model.  27  •IfXi£  r for alii G I (Ifinite),then UiXi G r.  (X, T) is called a topological space. When it is clear from the context, we will use X to denote topological space (X, r). The members of a topology r are said to be r-open subsets of X, or simply open if no ambiguity arises. A subset S of X is closed iff X  —  S is open. From this definition, it is easy to show that for  any topology on X, X and 0 are both open and closed. There are two extreme topologies on X. The coarsest (or smallest) topology on X is called  trivial if only X and 0 are open and the finest (or largest) topology on X is called discrete if r = V(X), where V(X) denotes the power set of X. We say that a topology T\ is &finertopology than T2 iff r i D T2. As mentioned earlier, topologies allows us to reason about convergence. Therefore, it should not be surprising that topologies can also be defined in terms of limit points. Let x G X and JV(a:) be a r-open subset of X containing x. N(x) is called a neighborhood of x w.r.t. r. A point x of X is a limit point of a subset S of X iff every neighborhood of x also contains a point of S distinct from x, i.e., VN(x), N(x) D S — {x} ^ 0.  Proposition 2.1 (1) A subset is closed iff it includes all its limit points. (2) A topology is trivial iff  every point x is a limit point of any subset with elements distinct from x. A topology is discrete if no point is a limit point of any subset. We provided results with regards to the convergence notion of topologies. Now we define connectivity and continuity on topological spaces. The notion of continuity will not only be useful for topological concepts, but will also be central to our approach with measure theory. A topological space is said to be separated if it is the union of two disjoint, non-empty open sets; it is otherwise connected.  Proposition 2.2 A topological space is connected iff the only sets that are both open and closed are the empty set and the total set. Now let us introduce the notion of continuous function. Let (X, r) and (X , r') be topological 1  spaces. A function / : Q x X —> X' is continuous iff for any r'-open subset S of X', f~ (S') — 1  {(to, x)\f(u, x) G S'} is r-open. Moreover, a function any r'-open subset S' of X', fj ^') 1  : X —• X' is pathwise continuous iff for  = {x\fu(x) G S'} is r-open. 28  1  Proposition 2.3 (1) Continuous functions are closed under functional composition. (2) A function fu : X —* X' is pathwise continuous, iff x G X is a limit point of S C X implies that f  (x) is a  w  point or a limit point of f (S) w  = {fu(x)\x G S}.  There exists smaller collections of subsets that can represent the open sets. These collections are referred to as a basis and a sub-basis of a topology. Definition 2.2 (Basis and Sub-basis) A subset B of a topology r is. said to be a basis for r iff each member ofr is the union of members ofB. A subset S of T is said to be a sub-basis for r iff the set B = {B\B  is the intersection offinitelymany members ofS}  is a basis for r.  It is also possible to derive new topologies that are based on known ones. Two important types of such derived topologies are called subspace and product topologies. Proposition 2.4 Let (X, T ) be a topological space, X' C X and r' = {W\W The collection T' is a topology on X'.  = X' f)U,U  G r}.  We call T' the subspace topology on X', and (X\ T') a  subspace of (X, r). Let {{Xi, Ti)}i i be a family of topological spaces and let x jXi be the product set of { X j } ; / . €  e  Let S = {xiVi\Vi — Xi for all but one i G I, and Vi G r, for all i G topology on  We call r the product  xjXi iff iS is a sub-basis for T . We call {xjXi, r) the product space of {(Xi,  Tj)}j /. e  If Xi = X with the same topology for all i G I, XjXi is denoted by X . 1  Proposition 2.5 Let {Xi} j i€  be a family of topological spaces and  J be an arbitrary index set.  Then ( x / X , ) = X / X / . 7  Definition 2.3 (Hausdorff Topology (T^ space)) A topological space (X, r) is said to be Hausdorff (T ) if given distinct x, y G X, there exist disjoint open sets U,V 2  G r (that is, UDV  = 0)  such that x G U and y G V. The trivial topology is non-Hausdorff and the discrete topology is Hausdorff.  Next, we will introduce the notions of partial order topology and metric topology. These two important types of topologies are central to the foundation of our framework. We will highlight the fact that partial order topologies in general are non-Hausdorff and metric topologies are Hausdorff.  29  2.1.1  Partial order  The results and notions introduced in this section are analogous to those presented in Section 3.1.2 of [Zha94]. The application of a partial order relation to a given set leads to a partially ordered set, which we call, for simplicity, a partial order. Formally, the notion of partial order is defined as follows: Definition 2.4 (Partial order) Let Abe a set. A binary relation < A  Q  A x A is called a partial  order relation iff <A is reflexive, anti-symmetric and transitive. (A, <A) is called a partial order; it is called a linear order or total order iff in addition, Vai, 02 S A, either ai <A 0,2 ° a>2 < A r  For any partial order relation  a  i-  we define <A as the strict relation of <A, i.e., a\ <A a% iff  «i <A 0,2 and ai ^ a2. We will simply use A to denote partial order (A, <A) if no ambiguity arises. Now let us introduce the notions of partial orders related to subsets and set products. Definition 2.5 (Sub-partial order) Let (A, <A) be a partial order and A' C A. A partial order  relation <A* Q A' x A' is called the sub-partial order relation on A' iff a.\ <A' 02 whenever &i <A 0,2- (A', <A<) is called a sub-partial order of (A, <A)Definition 2.6 (Product partial order) Let {Ai}  j / £  be a set of partial orders and A = x i rAi. e  A partial order relation <A C A x A is called the product partial order relation on A iff a < A 0! whenever <A a\ for all i e I. (A, <A) is called the product partial order of {(Ai, <Ai)}ieit  Let us now introduce the notion of least and greatest element of a partial order. Note that not all partial orders have such elements. Definition 2.7 (Least (Greatest) element) Let Abe a partial order. An element ±A (~1~A) £ A is  a least (greatest) element in A iff it satisfies the following axiom: ±A < A  A  (TA > A 4 Va £ A .  It follows from the antisymmetry of <A that least (greatest) elements, if they exist, are unique. Definition 2.8 (Flat partial order) A flat partial order, written A, is a set A augmented with a new  element J_A» viz., A = A U {-LA} such that a <^ a' implies a — a' or a = J_A-  30  Hence, any set A can be extended to a flat partial order by augmenting a least element A-A£ A. The element ±A is the least element of A and ±A means undefined in A. With this augmentation, we obtain the property that any partial function to A can be extended into a total function to A, i.e., f(a) =A-A if / is not defined at a. In this dissertation, we will only consider total functions unless explicitly stated. A subset of a partial order may have a least upper bound and/or a greatest lower bound. Definition 2.9 (Least upper (Greatest lower) bound) Let A be a partial order, D  C  A and a & A.  Then a is an upper (lower) bound ofD iff d <A a(d >A a) for every d G D. Moreover, a is a least upper bound (lub) (greatest lower bound (glb)J of D iff 1. a is an upper (lower) bound ofD and 2. if d'is an upper (lower) bound of D then a < A d! (a > A d!). Similarly to the least (greatest) element, it follows from the antisymmetry of < A that the existence of least upper bound (greatest lower bound), guarantees its uniqueness. We use \J D {f\ A  A  D)  to denote the least upper (greatest lower) bound of D in A, when it exists, and we will drop the subscript A if it is clear from context. To adhere to usual mathematical conventions, to denote V and f\, we will use "sup", "inf" when A is the set of real numbers and "max", "min" when D is finite. One important kind of subset of a partial order is directed subset. Definition 2.10 (Directed subset) Let Abe a partial order and D  and for all d\,d  2  C  A. D is directed iff D ^ 0  £ D, the set {di, d } has an upper bound in D. 2  A complete partial order is a type of partial order that will prove very useful throughout this dissertation. We thus present the formal definition along with two propositions related to complete partial orders. Definition 2.11 (Complete partial order (cpo)) A partial order A is complete iff:  1. it contains a least element, denoted L A , and 2. every directed subset of A has a least upper bound in A.  31  Proposition 2.6  Aflatpartial order is a cpo.  Proposition 2.7  The product of epos is a cpo. Let {Ai}i j be a set of epos and A = xjA{. The e  least element of A is ±A with  ( J - A ) J =-l-Aj, V i G  I. Let D be a directed subset of A. The least  upper bound of D is \J D with (\j D)i = \f . Dj, Vi G I, where Di is the projection of D onto A  A  A  its ith component, i.e., Di — TliD. A topology can be defined from a partial order. Definition 2.12 (Partial order topology)  Let Abe a partial order. A subset S of A is open iff (1)  S is upward closed, i.e., a G S implies that W >A a, a' G S, and (2) S is inaccessible from any  directed subset D of A, i.e., if\J D £ S, then 3a G D, such that a G S. This collection of open A  sets on A forms the partial order topology of A. A partial order (A, <A) is non-trivial iff there exist two elements a, a' in A such that a <A a'. Proposition 2.8  The partial order topology of a non-trivial partial order is non-Hausdorff.  Note that partial order topologies have different properties than their more familiar Hausdorff counterparts. For example, every open set in the partial order topology over K U {—oo, +00} with the usual < order is of the form {x G R\x > c} for some c £ i Thus, (3, +00) is an open set in this topology, but (3,4) is not. Recall that a function is continuous iff pre-images of open subsets are open subsets. Thus, the function Xx.y/x is not continuous in the partial order topology over the reals.  2  Let us now introduce the notion of lattice which will be essential when proving the main results on the semantics of our framework in Chapter 3. Definition 2.13 Lattice A lattice ^  is defined as a tuple (A, <), formed by a non-empty set S and  a binary relation <. The relation < induces a partial order in A. Moreover, it is assumed that fo any two elements a,b G A, there is a least upper bound and a greatest lower bound. The lattice  is called complete if every subset B of A has a least upper bound and a greatest  lower bound. For notions and facts concerning lattices, the reader should consult [Bir67] \x.expr(x) is a lambda expression of a function / , equivalent to Va;, f(x) = expr(x).  2  32  The following two propositions declare the properties of continuous and pathwise continuous functions in partial order topologies. Proposition 2.9 Any continuous (or pathwise continuous) function is monotonic, i.e., if f : fl x  A —* A' (fu : A —> A') is continuous (pathwise continuous), then (u\,a\) < n A X  a ) implies f(tJi,ai) 2  <A' / ( w 2 , a 2 ) (fu(ai) <A>  (^2,^2)  (ai <A  / U ^ ) ) -  Proposition 2.10 Let A and A be two epos. Then f : fl x A —> A' is continuous ifffor every 1  directed subset D C (fl x A), 1. f(D) — {f(d)\d G Z?} w directed and 2- / ( V Q A £ ) = V A ' / W X  T/ie rame result applies to pathwise continuous functions. 2.1.2  Metric space  Metric topology is the most direct generalization of the topology used for real numbers in analysis. A metric space S is a topological space; we call this kind of topology a metric topology. Let us now formally introduce the notions of metric, metric space and metric topology. Definition 2.14 (Metric and Metric Space) Let X be a set and R  +  be the set of non-negative real  numbers. A real values function d : X x X —> R is called a metric or sometimes a distance +  function on X iff • d(x,y) = d(y,x). • d(x,y) < d(x,z) +d(z,y). • d(x, y) = 0 iffx = y. (X, d) is called a metric space. Intuitively, a metric space is a set X with a global distance function (the metric d) that, for every two points x, y G X, gives the distance between them as a nonnegative real number d(x, y).  33  Definition 2.15 (Metric topology) The metric topology of a metric space is a topology with the set  of spherical neighborhoods as a sub-basis, where the spherical e-neighborhood ofx is {x'\d(x', x) < e}, denoted N (x), for (X, d) a metric space, x G X and e a positive real number. £  Proposition 2.11 Metric topologies are Hausdorff.  In a metric space, U is open if for every x in U we may find an e > 0 such that D (x) is also e  contained in U. Another important concept used in analysis is measure. However, before introducing the notion of measure formally, we will present the concepts of algebra and o-algebra, on which the definition of measure is dependent. Definition 2.16 (Algebra) Let fi be an abstract set. A collection T of subsets of fi is called an  algebra on fi if (i) tie  J, 7  (ii) F € F =>• F :— fi\F G T, where \ denotes set difference. c  (iii) F,G £ F =3> F U G £ F Note that from (i), (ii) we obtain that 0 G F and from (ii), (iii) we obtain that: F,G eF  ^ Ff]G  £F?  Thus, an algebra on fi is simply a family of subsets of fi closed under finitely many set operations  (\,u,n). Definition 2.17 (a-algebra) A collection F of subsets o / f i is called a a-algebra on fi if T is an  algebra on fi such that if F G F, n G N then n  [JF eF. n  n 3  We get this result since F U G 6 T. Hence, the complement of that set (which is F n G) must also be C  C  in T  34  Once again, it can be shown that D F n  e F. Thus, a cr-algebra on fi is a family of subsets of fi  n  closed under any countable collection of set operations. Let us now define the notion of filtration, which arises when looking at sequences of increasing cr-algebra. Definition 2.18 A family M. = {Mt}t>a of a-algebra M. c F is a filtration on (fi,.F) if{M-t} t  is increasing, i.e., 0 < s < t =>• Ai  C Ait-  s  It is natural to wonder what is the difference between topology and cr-algebra as both definitions appear very similar. In fact, the definitions of topology and cr-algebra are different. Topology does not require closure under complement operation. The most important difference is that open sets are defined based on topology, rather than a-algebra while measure is defined based on cr-algebra, rather than topology. Note however, that set theory is applicable in both topology and cr-algebra. Now let us introduce the notion of set functions along with some important properties. Definition 2.19 (Additive and Countably Additive Set Functions) Let Cl be a set, let Eo be an algebra on fi, and let 7 be a non-negative set function defined as: 7 : F^  [0, 00].  Then 7 is said to be additive «/7(0) = 0 and for F,G £ F, we have F n G=  0 => -y(F U G) = 7(F) + 7(G).  The mapping 70 is furthermore called countably additive 1/7 is additive and whenever (F : n G N ) n  is a sequence of disjoint sets in F with the union F = UF in F , then 4  n  (F) = J2^(Fn)n  1  Definition 2.20 (Measurable space) A pair (fi, F), wherefiis a set and F is a u-algebra on fi, is called a measurable space. Definition 2.21 (Measure space) Let  (fi,  F) be a measurable space, with F being a o-algebra on  fi. The mapping 7: ^ [ 0 , o o ] 4  We are not requiring that T be a cr-algebra, thus this has to be assumed. 35  is called a measure on (fl, J-) 1/7 is countably additive. The triple (fl, J-,7) is called a measure space. Definition 2.22 (Borel a-algebra) Let (X, r) be a topological space. B(X), the Borel a-algebra  on X is the smallest a-algebra generated by the family of open subsets ofX which contains r. The members ofB(X) are called the Borel sets ofX. Example 2.1 Let us define B :— B(R).  The a-algebra B is perhaps the most important of all  a-algebras. Every subset ofM. used in practical applications is an element ofB.  •  Now that we have introduced the notion of measure and measure space, we are ready to define the important notion of probability measure. Definition 2.23 (Probability measure) The measure P is called a probability measure if  P(fl) = 1, and the triple (fl, T, P) is referred to as a probability space. Therefore, a probability measure is a real-valued function defined on the u-algebra J- which assigns a value between 0 and 1 to each set A which is a member of T (A e T).  2.1.3  Random Variables  Equipped with the above notions of probability, measure and topological theory, we are now able to formally construct a random variable (which will obviously be of great importance for the PCN framework), but first let us introduce the key notion of measurable functions. Definition 2.24 (Measurable function) A function f : fli —» fl of two measurable spaces (fli,Ti) 2  and (fl , ^2) is called measurable if f~ (J- ) l  2  2  Q  tffli  and fl are topological spaces, we call 2  f : fli —> fl Borel-measurable if it is measurable with respect to the Borel a-algebra offl\ and 2  fl . 2  An important result, which will prove useful when proving properties of PCNs, arises from Borel sets and continuous functions: 36  Proposition 2.12 Let (X, B) be a measurable space. Every continuous mapping of X is Borel  measurable. Definition 2.25 (Random variable) Let (Cl, F) be a measurable space. A function X : (Cl, F) —>  M. is a random variable if for every subset A C Cl of the form A={u:  X(u) < £ } , £ e K  is an element of the o -algebra F. By definition of a random variable X, all sets of the form A  =  {LJ:  X(LO)  <  £}  have well-defined probabilities. The probabilities are well-defined since A C Cl and A € f , and as defined above, P assigns probabilities to all such sets A. Therefore, the existence of such a probability function is guaranteed. Definition 2.26 (Probability Distribution Function) A real-valued function associated with a ran-  dom variable X and defined by  F (0  = P({w  X  :  X(u) < ft)  is called the (cumulative) probability distribution function. We note that probability distribution functions are by definition monotonic non-decreasing. The probability distribution function is a basic entity associated with any random variable that allows us to generate probabilities of sets of interest. By definition of random variables, we are assured of its existence. However, we are not assured of the existence of its derivative everywhere, but if it does exist, it is often easier to use and more revealing in terms of graphical interpretation. If a scalar-valued function f (-) exists such that x  F (0 = f x  f (p)d x  P  holds for all values of £, then this function is the probability density function of X. We are not guaranteed of the existence of this function, but if F (-) is absolutely continuous, then it exists. X  Furthermore, one can show that the probability that X(u) = x lies in any set A is then P({u:X(u)eA})= 37  [ JA  f (Od£. x  Now let us review what can be obtained through a random variable mapping X. We saw that X maps Cl into R such that each irreducible set in Cl maps into a value in K . Thus, the sets of interest in R will be elements in the Borel a-algebra TB associated with R. For all sets i 4 c i and A G TB, we can define probabilities through P ( A ) = X4 fx(€)d£, where P (-) is the probability function X  x  (Borel measure) associated with R . Therefore, we now have a new probability space, (R, TB-, P I ) , generated by the mapping X from the original probability space:  (Cl,T,P) ^ x  (R,^ ,Px) B  Thus, one can quite often neglect the original probability space and describe a problem conveniently in terms of the new probability space (R, TB, P I ) .  Functions of Random Variables In the previous section we introduced the concept of random variable as a measurable function from Cl to R . When modeling probabilistic dynamical systems, many inputs are in fact random variables and it is important to understand the effects of such inputs on the resulting outputs of the system. Therefore, we believe that the notion of function of random variables deserves to be introduced in some detail. Let assume that X is a vector random variable that maps the sample space Cl into the ndimensional Euclidean space R . Now consider a continuous mapping #(•) from R n  thus generating a vector y G R  m  n  into R , m  from a vector x G R™. This mapping can be out of a more  general class of functions than the continuous functions, referred to as the Baire functions (Borel measurable functions), composed of continuous functions and limits of continuous functions. Now let us define the m-dimensional function cp as the composite mapping 0 :=  Then  4> is itself a random variable. It can be shown that every Baire function of a random variable is also a random variable. Recall from the previous section that X generates a new probability space (R™, TB, P I ) from the original probability space (Cl, T, P ) . If 9 is a measurable function on R , then for every set n  of interest A in the range space R " \ the inverse image in R , i e R " : 9(x) G A, is an event for n  which probability has been defined through P . If we were to view ( R , TB, P ) as the underlying n  x  x  probability space, then this just defines 9(-) itself as a random variable mapping from the sample 38  space K " into the space E . Thus we would get m  (fi, F,  P)  X  X  ]  (R , F , n  B  PX)  (R , ^ s , m  P*)  Then we can directly map the original probability space into the new probability space:  (fi,^,P)  (R ,F ,P^)  rt^™  m  1  B  Therefore, it is obvious to see that the variable 0 has a distribution induced by the distribution of X.  F*(0 = P(w : 0(w) < 0  =  -Px(x  : 9(x) < £)  Finishing up this section, we define the concept of limits. For uncertain dynamical systems, we very often have a (finite or infinite) sequence X\, X2, • • •, X of random variables and are interested n  in their asymptotic behaviour, that is in the existence of a random variable X which is the limit of the X in some sense. There are several different ways in which such a convergence can be defined. n  Broadly speaking these fall in two classes, one in which the realizations (which will be defined as stochastic traces shortly) of X  n  are required to be close in some way to those of X and another one  in which only their probability distribution need to be close. In this body of work, we will consider the latter class of convergence, and more specifically convergence in distribution. Let us first formally define the notion of convergence in distribution. Definition 2.27 (Convergence in Distribution)  An infinite sequence of random variables X\,X2, • • •, X ,... n  is said to convergence in distri-  bution to a random variable X if Hm ^ Fx„ n  00  (x) = Fx{x), at all continuity points of Fx,  where Fx is the probability distribution function of the random variable X. This type of convergence is also known as convergence in law. Given any sample space fi, linear order L and topological space X, v : fi x L —> X is called a stochastic linear set of values. A limit of v is defined as a generalization of the convergence in distribution of a sequence. 39  Definition 2.28 (Limit in distribution)) Let X be a topological space, fl a sample space, and  v : fl x L —> X be a stochastic linear set of values. A random variable v* : fl — • > X is called a limit in distribution ofv, written F —> F *, ij^^m;_ P(i;(Z) < a:) = P(v* < x) for all continuity v  v  >0O  point x £ X of FxIf L has a greatest element IQ, then F —> F y y Therefore, the concept of limits in distribution v  v  0  is also a generalization of the "final" value in distribution. We will use l i m ^ o o F ^ to denote the v  limit in distribution of v if it is unique. For Hausdorff topologies, the following proposition shows that limits in distribution are in fact unique. Proposition 2.13 If X is of a Hausdorff topology and v : fl x L  — >  X is a stochastic linear set of  values, then F —> F * and F —» F * imply F * = F *. v  v  v  v  v  v  Note that fully deterministic linear set of values, that for which \ fl\ = 1, are simply special cases of the general stochastic linear set values introduced above. In this case, the limit in distribution simplifies to the usual notion of limit of a sequence (the weight of the limiting distribution is concentrated at the value of that limit). Moreover, we can show that for Hausdorff topologies, these limits are unique and possess the point-wiseness property [Zha94]. 2.1.4  White Noise, Brownian Motion and Stochastic Integrals  For continuous time systems, it has become the norm for dynamical systems to be modeled by a set of differential equations of the general form dX/dt = b(t,X ). However, since we are interested t  in explicitely modeling the uncertainty inherent in these systems, we need to find a suitable mathematical interpretation of the noise that will be included in models with continuous time structures. Consider the following differential equation augmented with an uncertainty term denoted by W: t  ^ - = b(t,X )+g(t,X )-W t  t  t  (2.1)  It seems reasonable to assume that any stochastic process W that represents the uncertainty term in t  Equation 2.1 will have the following properties: 1. Wt is independent of W whenever t ^ s. s  40  2. E(W ) = 0 for all t. t  3. The joint distribution of {W +t, • • •, Wt +t} is independent of the value t. This is equivalent tl  k  to stating that the distribution {Wt} is stationary. The assumptions (1), (2) and (3) suggest that W should have stationary independent incret  ments with mean 0. However, it can be shown that there does not exist any reasonable stochastic process which satisfy (1) and (3) in continuous time. Such a process cannot have continuous paths [Oks98]. In fact, Kallianpur showed that if we require E(W^) = 1, then W cannot even be meat  surable with respect the cr-algebra B x J [Kal80]. 7  Although it is possible to represent Wt as a white noise process which is constructed as a probability measure on the space <S' of tempered distribution on [0, oo) [Hid80, Adl81, Roz82], it is more common to avoid this type of construction and instead rewrite Equation 2.1 in a form that allows for the replacement of W with a proper stochastic process. Based on this, we can show that the t  only process with continuous paths is Brownian motion B [Kni81]. The Brownian motion process t  is also referred to as the Wiener process. Using the usual integration notation, we can rewrite Equation 2.1 to obtain an equation of the form (2.2) where Bt(ui) is a 1-dimensional Brownian motion starting at the origin and where it is assumed that the stochastic process X  t  = X (uj) of Equation 2.1 is the solution to Equation 2.2 where an t  appropriate meaning for the last integral is provided. Brownian motion is independent of each of its time realizations which can let us think of Gaussian white noise as the formal derivative of Brownian Motion. It is well known that E(Bf) = t and E(Bl) = 0 so that E(B  - B)  2  t+s  t  = t + 5 - 2t + t = S. We can see that B s - B is roughly t+  t  of the order of v/r5 and thus we do not expect the sample paths of a Brownian motion to be almost surely differentiable anywhere [Bre68]. Actually, this reflects the fact that the path variations of B  t  are too big to enable us to define the rightmost integral of Equation 2.2 in the Riemann-Stieltjes sense. In particular, the total variation of the path of a Brownian motion is infinite. It can be shown that unlike the Riemann-Stieltjes integral for deterministic functions, when calculating the value of the rightmost integral of Equation 2.2, the choice of the interval end points 41  affect the final result. The key difference between the results of the Ito integral and the result in a Riemann-Stieltjes integral from real analysis is that the integrand is not evaluated on any point t' 6 [U, U+i] but precisely at the left endpoint tj of the interval. The value of a stochastic integral changes with the choice of the point t' where the integrand is evaluated. Choosing the mid point (ti + i j i ) / 2 leads to the Stratonovich integral, and in general the result is different from the Ito +  integral. Example 2.2 Comparing ltd and Riemann-Stieltjes Let us now compare the results of the Riemann-Stieltjes and ltd interpretations in calculating f  T Q  BtdBt. When considering the integral as a the Riemann-Stieltjes integral the result is simply  1/251.. However, the results under the ltd interpretation, is l/2Bj. — 1/2T while the result for the Stratonovich interpretation is the same as the Riemann-Stieltjes, i.e., 1/2B^. • Based on these reflections, a natural question that one might ask would be: Which interpretation of "JQ g(s, X )dB " s  makes Equation 2.2 the correct mathematical model for Equation 2.1?  s  Although there are some cases for which the Stratonovich interpretation is more reasonable, the specific feature of the Ito model of not looking into the future, as apparent from the selection of the end points of its calculation, is a powerful reason to select the Ito paradigm when in the presence of real-time dynamical systems. For a similar, more in depth discussion about why Ito is more appropriate for biological systems, the reader is referred to [Tur77]. Moreover, as Ito integrals are Martingales [Oks98], this leads to an important computational advantage over the Stratonovich interpretation as Martingales are the stochastic analog of conservations laws: expectation is "conserved", i.e., for a filtration Tt, E ( X i | J ) — Xt, V£. Hence, in this dissertation, we will adopt the r  t +  4  Ito interpretation for its convenience and computational advantage. Furthermore, it can be showed that the solution of Equation 2.2 obtained via the Stratonovich interpretation is identical to the solution obtained from the modified ho equation X = X + J* b{s, X )ds + i J g'(s, X )g(s, X )ds + J* g(s, X )dB t  0  s  s  s  s  s  (2.3)  where g' is the derivative of g(t, x) with respect to x [Str66]. Hence, due to this explicit connection between the two types of integrals, the choice of paradigm is not as critical as it may appear since one can always jump back and forth between the Ito and Stratonovich interpretation with a simple modification. 42  Let us now enumerate some important conditions of stochastic process. These conditions will prove essential when discussing the existence and uniqueness of stochastic differential equations (SDEs) in § 4.3.2. We first define the notion of cr-algebra generated by a random variable and the concept of adaptiveness. Definition 2.29 Let Ft = Ft denote the a-algebra generated by the random variables B (-), s  s < t, where Bt(u>) is the n-dimensional Brownian motion. Hence, Ft is the smallest a-algebra that contains all the sets ofthe form {w,  F\,--- ,B (uj) € Fk},withtj <tandFj  j B (w) €  tk  tl  C l "  being Borel sets, j < k = 1,2 .... Intuitively, Ft can be seen as denoting the history of B up to time t. Hence, a random function s  r(u) is Ft-measurable if and only if r is equivalent to the pointwise limit of sums of functions of the form gi(B )g2{Bt ) tl  2  • • • 9k(Bt ), where the functions g are bounded continuous functions and  tj < t for j < k, k — 1, 2  k  For example, the function r\(uj) — B /i{<jj) is .Ft-measurable while t  r2(u>) — B3t{u) is clearly not. Definition 2.30 Assume {Ft}t>a to be an increasing family of a-algebra of subsets ofCl. We call  -adapted, a process f(t,u>) : [0, oo) x Cl —> M if, for each t > 0, the function u> —> f(t,to) is n  Ft-measurable. Hence, it is easy to show that ri(w) = 5 / (w) is J^-adapted, unlike r (oi>) = Bzt(oj)t  2.2  4  2  Time Structures  As we are modeling dynamical systems, a model of time and its evolution is necessary. In fact, a clear notion of the concept of time is central to understanding dynamics. As it was done within the C N framework, we formalize time using an abstract structure that captures its most important properties. In general, a time structure can be considered as a totally ordered set with an initial start time, an associated metric for "the distance between any two time points" and a measure for "the duration of an interval of time." Formally, we define the concept of time structure as follows. Definition 2.31 (Time structure) A time structure is a triple (T, d, p) where • T is a linearly ordered set (T, <) with 0 as the least element; 43  • (T, d) forms a metric space with das a metric satisfying: for all to < t\ < t , 2  d{t ,t ) = d(t ,*i) + d{ti,t ), 0  0  2  2  {t\m(t) < r} has a greatest element and {t\m(t) > r} has a least element for allO < r < sup{m(t)\t G T} where m(t) — d(0, t); • (T, fj, p) forms a measure space with a as the Borel set of topological space (T, d) and p as a Borel measure satisfying p([ti,t )) < d(t\,t ) for allt\ < t where \ti,t ) = 2  2  2  2  <t <  t } andp([t t )) = /i([0,t )) - M([0,*I)). 2  u 2  2  To abridge the notation, we will simply use T to refer to the time structure (T, d, p) when no ambiguity arises. In general, we have p{\t\,t )) = d(t\,t ). However, when T is an abstraction 2  2  of another time structure, it is possible that 3ti, t ,p([ti, t )) < d(t\,t ). Discussions on time 2  2  2  abstraction can be found in Chapter 6, Behavior Analysis of [Zha94]. We will refer to a time structure T as being infinite iff T has no greatest element and p(T) = oo. Moreover, the time structure T is continuous iff its metric space is connected while it is discrete iff its metric topology is discrete. Example 2.3  Consider the set of natural numbers N and the set of nonnegative real numbers R  +  along with the metric d(ti,t ) — \t\ — t \ and the measure yu([0, t)) = t. N and M respectively +  2  2  define discrete and continuous time structures. The set {1—  |n G N} with the metric and measure  defined above also defines a discrete time structure. However, the sets {1 —  € N} U {1},  {0} U {^r|n G N} and [0,4] U [5,7] with the metric d and the measure p form time structures neither discrete nor continuous. • Note that the set of rational numbers Q with the metric d and the measure p does not form a time structure. This can be proved using the fact that the set Q of rationals lacks the least upper bound property stating that if a set S has the property that every nonempty subset of S which has an upper bound also has a least upper bound. This property is summarized in the following proposition. Proposition 2.14  Equivalent properties of time structures  1. For any time structure (T,d,p), ifTczT bound in T. 44  has an upper bound in T, T has a least upper  2. The following properties for a time structure are equivalent: (a) (T, d, p) is discrete. (b) Let (ti, t ) = {t\ti < t < t }. For all t, ift is not the least element ofT, then 3t' < t, 2  2  denoted pre(t), such that (t , t) — 0, and for all t, ift is not the greatest element ofT, 1  then 3t' > t, denoted suc(t), such that (t, t') = 0. (c) (T, d, p) is well-founded, i.e., Vt e T, [0, t) is finite. 3. The following properties for a time structure are equivalent: (a) (T,d, p) is continuous. (b) (T, d, p) is dense, i.e., for all ti < t , there exists to such that ti < to < t . 2  2  Now let us discuss the relationship between two different time structures. A time structure (T, d, p) can be related to another time structure {T , d , p ), where (%, < ) is a total order with r  r  r  r  0 as the least element, by a reference time mapping h : T —> T satisfying the following properties: r  r  • the order among time points is preserved: t < t' implies h(t) < h(t'), r  • the least element is preserved: h(0) — 0 , r  • the distance between two time points is preserved: d(ti, t ) = d (h(ti), h(t )), and 2  r  2  • the measure on any finite time interval is preserved: p([0, t)) — p ([0 , h(t))). r  r  In such a case, we will call T the reference time of T , and T the sample time of T . For r  r  example, if h : N —> R is defined as h(n) = n, K is a reference time of N. For any time structure +  +  T , a reference time of T is as "dense" as T . Furthermore, it can easily be shown that the reference relation is in fact transitive [Zha94]. Now that we have a formal definition of the concept of time, we wish to formalize the notion of domains, stochastic traces and events in a similar fashion. The definitions associated with the notion of domain are analogous to those introduced in §3.3 of [Zha94]. Since these notions are of importance to a clear understanding of the upcoming sections, we reproduce some results from that section and adapt some to the stochastic case. 45  2.3  Domain Structures  As with time, we formalize domains as abstract structures so that discrete and continuous domains are defined uniformly. A domain can be either simple or composite. Simple domains denote simple data types, such as reals, integers, Booleans and characters; composite domains denote structured data types, such as arrays, vectors, strings, objects, structures and records. Definition 2.32 (Simple domain) A simple domain is a pair (AU  where A is a set,  { J - A } , <^A)  A means undefined in A, and dA is a metric on A. Let A = A  U {J-A}-  For simplicity, we will use A to refer to simple domain {A, dA) when no  ambiguity arises. For example, let K be the set of real numbers, M is a simple domain with a connected metric space; let B = {0,1}, B is a simple domain with a discrete topology on B . Any simple domain A is associated with a partial order relation < . A  (A, < ) is a flat partial A  order with ±A as the least element. In addition, A is associated with a derived metric topology T = TA U {A} where TA is the metric topology on A derived from the metric dAProposition 2.15  {-LA}  is not T-open. The only neighborhood of-LA is A.  A simple domain (A, dA) can also be represented as a triple (A, < , r) where < A  A  is the partial  order relation and r is the derived metric topology. A domain is defined recursively based on simple domains. Definition 2.33 (Domain) (A, <A,  T),  with <A as the partial order relation and r as the derived  metric topology, is a domain iff: • it is a simple domain; or • it is a composite domain, i.e., it is the product of a family of domains {(Ai, <A , i~i)}i£i such t  that (A, <A) is the product partial order of the family of partial orders {(Ai, <Ai)}izi and (A, T) is the product space of the family of topological spaces {{Ai,  Tj)}j /. e  Note that there is no restriction on the index set I, which can be arbitrary (finite or infinite, countable or uncountable). For simplicity, we will use A to refer to domain (A, <A,T) when no ambiguity arises. For example, let n be a natural number, then M " is a composite domain with n components; 46  let N be the set of natural numbers, then N —> B (or equivalently, B ) is a composite domain with infinitely many components. Given a simple domain A, a value a G A is well-defined iff a  Given a composite domain  xjAi, a value a G X / A , is well-defined iff Oj is well-defined for all i £ J . A value in a domain is undefined iff it is the least element of the domain. Intuitively, for any domain, its partial order topology characterizes the information (or definedness) hierarchies of data and its derived metric topology characterizes the limit properties of data. Proposition 2.16  For any domain, its partial order topology isfinerthan its derived metric topol-  ogy, and both are non-Hausdorff. A signature is a syntactical structure of a multi-sorted set of data with associated functions. Definition 2.34 (Signature) Let E = (5, F) be a signature where S is a set of sorts and F is a set  of function symbols. F is equipped with a mapping type: F —> S* x S where S* denotes the set of  allfinitetuples ofS. For any f G F, type(f) is the type of f. We use f : s* —> s to denote f G F withtype(f) = (s*,s). Example 2.4 Two Basic Signatures  The signature of an algebra on the Naturals can be denoted  by EN = ( N , {0, +, —,x}). This signature has only one sort, N, with 4 different function symbols.  The Boolean algebra can be described as: E(, = ({b}, {0, - i , A , V}) with 0 :—> b, -> : b —> b  A : b, b —> b, and V : b, b —> b. Ef, has one sort with a constant 0 (nullary function), a u function - i , and two binary functions A and V. • A domain structure of some signature is denned as follows. Definition 2.35 (E-domain structure)  is a pair ({A } s, {f }feF) A  s  s€  Let E = (S, F) be a signature. A E-domain structure A  where for each s G S, A is a domain of sort s, and for each s  f : s* —• s G F with s* : I —> S and s G S, f  A  : XiA * s  —•  A is a function denoted by f, which s  is continuous in the partial order topology. To be continuous on a domain in its partial order topology is not a real restriction on a function. Strict functions are continuous functions in partial order topologies. A function is strict w.r.t. an argument iff its output is undefined whenever the value of that argument is undefined. A function is strict iff it is strict w.r.t. all of its arguments. 47  Given any partial or total function / : fl x / Ai —» A, a continuous function / : fl x / A4 —> A can be defined as: f(u, a)  if ( w , o ) e f ! x / Ai and f(ui, a) is defined otherwise.  We call / a strict extension of function / . We will also use / to denote its strict extension if no ambiguity arises. For example, let E = ({r}, {0, +, •}) with 0 :—> r, + : r, r —> r and • : r, r —• r. r  Then ({R}, {0, +, •}) is a S -domain structure, where + and • are strict extensions of addition and r  multiplication on R, respectively. However, not every extension of a function that is continuous should also be strict. For example, ({B}, {0, -1, A , V}) is a X^-domain structure where -1, A and V are negation, conjunction and disjunction, respectively. Function V : IB x IB —> B is continuous but not strict, since V is an "or" logic satisfying 1 V x = 1 for all x 6 B, thus, IV  -LBT^-LB-  The following propositions characterize the general properties of continuous functions on simple domains. Proposition 2.17  (1) Function f : fl x A —> A' is continuous in the partial order topology iff f is  strict or constant. (2) If f : fl x A —> A' is continuous in the derived metric topology, then f is continuous in the partial order topology. (3) Function f : fl x A —> A' is continuous in the derived metric topology iff f is continuous in the partial order topology and the restriction  offonflxA  and A' is continuous in the metric topology, namely, for any open subset S of A', f~ (S) D (fl x A) 1  is open. The properties of continuous functions in partial order topologies can be generalized to composite domains. A function / : fl xj Ai —> A is continuous w.r.t. an argument j, iff function \aj.f(uj, a, aj) is continuous for all (u, a) S fl / - { j } Ai. x  Proposition 2.18  Let I be afiniteindex set. (1) Function f : fl x / Af —* A is continuous in the  partial order topology iff f is continuous w.r.t. alii £ I. (2) If f : fl x / Ai —> A is continuous in the derived metric topology, then f is continuous in the partial order topology. (3) Function f : fl x j Ai —> A is continuous in the derived metric topology iff f is continuous in the partial order topology and the restriction of f on fl Xj Ai and A is continuous in the product metric topology, namely, for any open subset S of A, f~ (S) fl (fl Xj Ai) is open. 1  48  A function is well-defined iff its output is well-defined whenever its input is well-defined. Both well-definedness and strictness are closed under functional composition, and a function can be both well-defined and strict. For example, a widely used conditional function, cond : A x A x A' x A' —> A', is defined as follows: (2.4)  else if x = y otherwise.  Function cond is continuous in the partial order topology; it is continuous in the derived metric topology if A is of a discrete topology. Furthermore, it is well-defined and strict w.r.t. arguments x and y.  2.4  Stochastic Traces and Events  In this section, we define the concept of stochastic trace, an extension of the concept of trace that was introduced in the C N framework. Stochastic traces will be central in representing the dynamical behaviour of the systems modeled within the PCN framework. A stochastic trace intuitively denotes the (random) changes of values over time. Formally, a stochastic trace is a mapping v : fl x T —> A from sample space fl and time domain T to value domain A. For a given UJ e Cl, the function Vu : T —> A is simply called a trace. In the literature, a trace is often referred to as a sample function, a realization, a trajectory or a path of the underlying stochastic process. We will use v to denote both the stochastic trace v or one of its realization trace  when it is clear from the context  and no ambiguity arises. A stochastic trace v is well-defined iff v(u, t) is well-defined for all (UJ, t) e O x T . A stochastic trace v is undefined iff v(u, t) is undefined for any (UJ, t) £ fl x T . For example, denote a Brownian motion process by B (uj) and T = R  +  t  and A — M. Then have that v = Xui,t.B (cj) is a wellt  defined stochastic trace. For a fixed LJ in  fi,  = Xt.B (u>) represents a path of the Brownian t  motion process. On the other hand, v\ = Xt. cos(£) and v-i = Xt.e~ are well-defined deterministic l  traces, i.e., stochastic traces for which  = 1.  Due to the fact that physical systems encompass uncertainty, one is often more interested in the distribution of the system rather than in one specific trace. Hence, rather than merely looking at one 49  given execution trace, one may pay attention to the distribution of traces of the system. One important feature of a trace is that it provides complete information about the current execution of the system of interest at every time point. In the presence of uncertainty, the limiting value of a specific execution trace v is of little interest since the measure of that trace is typically zero. w  The distribution of a stochastic trace, on the other hand, provides complete information about the probability of the state of the system at every finite time point. Although trace distribution values at infinite time points are not represented explicitly, they can be derived when limits (in distributions) are introduced. The limiting distribution of a stochastic trace can provide useful information when assessing the behaviour of the system in the long run. For example, consider the stochastic trace associated to the system denoted by / : fl x M —> R , where +  +  f(uj, t) — 1 + B (u>)e~ , with B (u>) a Brownian motion process. For each value of t, one can easily t  t  t  show that / follows a Gaussian distribution with mean 1 and variance te~ (Ff — J\f(l,te~ )). 2t  2t  The limiting distribution is hence Zim _ A/'(l, te~ ) = JV(1,0), which indicates that in the long 2t  t  >00  run, the system will converge to value 1 and will notfluctuateaway from it, despite being influenced by a Brownian motion with increasing variance. Let A be a domain, L a linear order and i > : Q x L — > A b e a stochastic linear set of values. The distribution of a random variable v* : fl —> A is a limit in distribution of v, written F  v  —> F * , iff v  P(v* < a) = Zzm;_ P(u(?) < a) for any a S A. >00  Limits in distribution of v may not be unique. However, the set of limits in distribution of v has the following properties.  Proposition 2.19 Let v : fl x L —> A be a stochastic linear set of values. Then (1)  F  v  ->  (2) F  v  —> F * and F  ±  ,  and  F  A  v  v  —> F * imply that either F * — F * or one of F * v  v  v  v  and F * is Fj_ . v  A  Now equipped with the concept of limit in distribution, we can complete a stochastic trace with its distribution at limit time points. Given a time structure T , let T ° ° be the set of downward closed intervals, i.e., for any T e T°°, (1) T / I and (2) t e T implies that for all t' < t, t' e T. A stochastic trace v : fl x T — > A can be extended to its completion v°° : fl x T°° —> A as v°°(T) = l i m v | where v\ denotes the restriction of v onto T. If T has a greatest element to, T  then  F y o o ^  =  F( y v to  T  A  trace completion provides distributions at infinite as well as atfinitetime  points. Note that T e T ° ° , for any stochastic trace v : fl x T — > A, v°°(T) = limv can be 50  considered as the "final" value. For simplicity, we will use v to refer to both v and its completion v°° when no ambiguity arises. Let us introduce notation that will prove helpful for the reminder of this dissertation. Let T<j = {t'\t' < t}. Then T  <t  element of T , <t  G T°° whenever t > 0. We usepre(t) to denote both T  and the greatest  <t  if it exists. Let T< _ = {t'\t' < t,d(t,t') > r} for r > 0. Then T < _ G T°° t  T  t  r  whenever m(t) > r. Moreover, it can be shown that for any time structure T , T< - has a greatest t  T  element whenever m(t) > r [Zha94]. To denote the greatest element of T < _ when m(f) > r, t  T  we will use t — r. We now define the notion of stochastic trace space, formed from the set of all possible stochastic traces. Definition 2.36 (Stochastic Trace Space) Given a time structure T and a domain (A, <A,T),  the stochastic trace space is a triple (A^ ', XT  of all functions from fi x T to A), < nxr A  <A^^T,T)  where A  f i x 7  " is the product set (the set  is the product partial order relation constructed from  the partial order relation <A, and V is the product topology constructed from the derived metric topology r. For a fixed w £ i l , the triple ( A  a , x T  , <  , T) simplifies to what we will call a trace space. Once  AUIXT  again, to abridge the notation, we will refer to the trace space (A , QxT  < nxr,T) A  as simply  A, T  when no ambiguity arises. A stochastic trace space is essentially a composite domain. Therefore limits in the distribution of a linear set of stochastic traces can be defined accordingly. The reader is referred to §3.4 of [Zha94] for a formal description of the results in the deterministic cases. The extension to the stochastic case is straightforward. We now present the definitions of non-intermittent and right-continuous stochastic traces, two special types of stochastic traces. Definition 2.37 (Non-intermittent stochastic trace)  A trace  : T —> A is path-wise non-intermittent iff for a fixed UJ and for any T G T°°, v(T) =-LA  implies that VT" D T, v(T') =_LA- A trace v  u  : T —> X / A j is path-wise non-intermittent iff Vi is  path-wise non-intermittent for all i G / . A stochastic trace v : Q x T —> A is non-intermittent ifv is path-wise non-intermittent for every UJ in Q.  51  *—• •  •  •  •—•—*~ t  Figure 2.1: An event trace: each dot depicts a time point Definition 2.38 (Right-continuous stochastic trace)  A right-continuous stochastic trace is a special type of trace defined as follows. A trace is right-continuous at to iffVt > to,t —> to implies v (t) u  : T —> A  —» i> (£o)/ ui is right-continuous iff it v  w  is right-continuous at all t £ T. A stochastic trace is right-continuous if its execution traces v  w  are path-wise right-continuous for all u> in fl. A discrete-time stochastic trace is always rightcontinuous according to this definition. A stochastic event trace is a non-intermittent and right-continuous stochastic trace whose domain is B . For each ui £ fl, an event trace  : T —> B with  ^ A i . ± B generates a structure  (%u> eu, Me > from (T, d, p) where d  u  • T  eu  • 4  U  • Vi  C T is defined as % = {0} U {t > 0|e (*) #-L , e„,(i) ^ e^^re^))}, u  =  w  B  \T x% >  d  eu  e T , Me e u  u  w  ([0,i)) = M([0,i)),  and/z^ (Tej = p(T) for T =  {i|e (i) ^ ± w  B  } -  Proposition 2.20 For any time structure T and any event trace e , (T , d , p ) w  eui  eui  ew  is a discrete  sample time structure ofT. For any event-based time, each transition point of the event trace defines a time point (Figure 2.1). Similarly to stochastic traces and stochastic trace spaces, the set of all possible stochastic event traces on a reference time structure, associated with a partial order relation and a derived metric topology, forms a stochastic event space.  52  Definition 2.39 (Stochastic event space) A stochastic event space is a triple (£^ ,  < nxT,T'}  xT  where T is a time structure, £ < !)xr £  f i x 7  " c B^* " 7  £  is the set of all stochastic event traces on fl x T,  is the sub partial order relation of C H X T , and V is the subspace topology ofT that is the —S7xT  derived metric topology ofM Similarly to stochastic trace spaces, for fixed ui in fl, we will call the triple  (£ ' ' , u  x  r  <  £  u  x r ,  T'), an  event space £ . r  2.5 Transductions Within the PCN paradigm, transductions will dictate the evolution, and hence the behaviour, of the modeled systems. Intuitively, transductions are mathematical models of general transformational processes. In PCN, transductions are similar to functions of random variables. For a given input, the output is unpredictable other than it is known to obey the probability distribution function resulting from the application of the transduction to the random variable. The concept of transduction is at the core of the ability of PCN to model the inherent uncertainty in the systems under study. We will characterize two classes of transduction: primitive transductions and event-driven transductions. These classes will be constructed from the functional composition of three types of basic transductions: transliterations, delays and generators. Generators will be used to represent random variables which will introduce uncertainty in the model. Finally, we will introduce the notion of event-driven transductions. Event-driven transductions will prove essential to construct models of temporally hybrid systems, i.e., systems encompassing different time structures. Note that deterministic transductions will be seen as a simple case of a transduction: one with no random variable as input.  2.5.1  Transductions: general concepts  Formally, a transduction is a mapping from input stochastic traces to output stochastic traces that satisfies the causal relationship between its inputs and outputs, i.e., the output value at any time depends only on inputs up to and including that time. The causal relationship stipulates that the evolution of the system cannot be dictated by the future state of the system, but only by past and present values. Formally, causality can be defined as follows  53  Definition 2.40 (Causality via Jt-adaptedness) Assume {Ft}t>o to be an increasing family of  a-algebra of subsets of A  n x T  . A mapping F(v)(u>,t) : A  Q x T  —>is Acausal ' if F(v)(u,t) is ,clxT  Tt-adapted. A causal mapping on stochastic trace spaces is called a transduction. For instance, a probabilistic state automaton (also known as a Markov chain) with an initial state defines a transduction (composed of two basic transductions: a generator and a unit delay) on a discrete time structure; a temporal stochastic integration with a given initial value is a typical transduction on a continuous time structure. Just as nullary functions represent constants, nullary transductions represent stochastic traces. Transductions are closed under functional composition. 2.5.2  Primitive transductions  Primitive transductions are denned on a generic time structure T . As mentioned above, primitive transductions are functional compositions of three types of basic transduction: generators, transliterations and delays. First, let us introduce the notion of generators. A generator will be the most basic component of our framework to introduce uncertainty in the model. Generators will in fact represent random variables that will be incorporated in transliterations and ultimately primitive transductions. Definition 2.41 (Generator) Let A be a domain, fl be a sample space and T a time structure.  Moreover, let F \A denote the (potentially conditional) cumulative distribution function for the X  random variable X. A generator ^(VQ)  : flxTxA-^Aisa  basic transduction defined as  where rand(Fx\A, •) is a random number generator associated with F \ A X  We allow the distribution function F \ A to X  D e  conditioned on t and values of the systems to pro-  duce a general model of uncertainty. This enables the user to model systems where the uncertainty component is non-stationary and conditioned on the state of the system. Also note that in this dissertation, we are not interested in the simulation of random variables per se, but rather in the analysis of the resulting models. Hence, we will assume that we are given, for each generator included in the model, appropriate random number generators. For more details on this widely studied field, we 54  refer the reader to the the following books and articles chosen from the plethora of work done on random number generators [BFS87, Dev86, Gen98, HLOO, Knu98, LKOO, L'E94, L'E98]. Definition 2.42 (Transliteration) A transliteration  is a pointwise extension of a function. Formally,  let f : fl x A —> A' be a function and T be a time structure. The pointwise extension of f onto T is a mapping fr : 4 J  n x T  —> A'  satisfying fr(v) = Aw, t.f(v(u>, t)).  nxT  By this definition, (/ o g)r — fr o g . r  We will also use / to denote transliteration fr if no  ambiguity arises. Intuitively, a transliteration is a transformational process without memory or internal state, such as a combinational circuit. For example, consider the function / : f i x R - > I defined as f(x) = 0.5x + Y(u>), where Y(-) is a stationary random variable uniformly distributed over the set {1,2}. The pointwise extension fr of / is a simple transliteration producing an output resulting in the sum of the input value x times 0.5 with a random value included in [1,2]. We will come back to this seemingly simple transliteration as it will serve as basis for a running example throughout this section. We will also use this example to illustrate the semantics of the PCN framework. Note that in the absence of any random variable within the transliteration, the transformational process is simply a deterministic function of the input. An important class of transliteration is that of asynchronous event control [Sut89b]. For example, let © : Bx B —> B be a function defined as x®y = (->x) Ay V x A (->?/), i.e., the "exclusive or". The transliteration ©, as introduced in [Zha94], functions as the basic "or" logic in asynchronous event control (Figure 2.2). For a more detailed description of deterministic event logics, the reader is referred to §5 of [Zha94]. Now let us present the last type of basic transductions: delays. There are two types of delay: unit delays and transport delays. For a given trace, a unit delay  5 (ui, T  u ) acts as a unit memory for data in domain A, given a 0  discrete time structure. We will use S(VQ) to denote unit delay 5^(LO, VO) if no ambiguity arises. Definition 2.43 (Unit delay)  Let Abe a domain,  time structure. A unit delay 8J(UI,VQ) : A  n  x  T  55  VQ  a well-defined value in A, and T a discrete  —> A  n  x  T  is a transduction defined as  el  e2  el ore2  Figure 2.2: Event logic for "or" where VQ is called the initial output value of the unit delay. However, in the presence of non-discrete time structures, unit delays may not be meaningful. Hence we need a transduction that is suitable for more general time structures. Definition 2.44 (Transport delay) Let A be a domain, VQ a well-defined value in A, T a time structure and r > 0. A transport delay A^(T)(U>, VO) : A^  xT  —• A  n  x  T  is a transduction defined  as A$(T)(LU,V )(V) = \t. I I v(u>,t — T) otherwise W  0  where VQ is called the initial output value of the transport delay and r is called the time delay. We will use A(T)(VO) to denote transport delay AJ-(T)(U,VO)  if no ambiguity arises. Transport  delays are essential for modeling sequential behaviors in dynamical systems. Let us now introduce an simple example of a primitive transduction. Let F denote the transduction represented in Figure 2.3 where f(u>, x) = 0.5x + Y(u>), and Y  w  : fi —> {1, 2} is the generator  following Fy, a distribution with uniform probability over the set {1, 2}. Note that the unit delay 6(0) is introduced to eliminate an algebraic loop, Hence, the output of this transduction would be a random sequence of values where the value at time t + 1 would be half of the value at time t 56  T =N A = R  +  l-y=Unilbrm({1.2}) f(x.y) = 0.5x + y  0—f(x,y)  -0  5(0)  Figure 2.3: Simple Transduction with uncertainty added to 1 or 2 with 50% probability. A possible execution trace resulting from this transduction on T = N is {0,1,2.5,3.25, 2.625,...} with measure 0.0625. 2.5.3  Event-driven transductions  As denned above, a primitive transduction maps stochastic traces to stochastic traces with the same time structure. However, in general, a hybrid system consists of components of different time structures. Therefore, it is important to introduce a mechanism for handling such situations. In this section, we introduce event-driven transductions, which will allow us to model components with various time structures. In order to properly introduce the notion of event-driven transductions, we need to define the concept of sample and extension traces. Let T be a reference time of T with a reference time r  mapping h. The sample stochastic trace of v : Cl x T —> A onto T is a stochastic trace v : r  Cl x T — > A satisfying v = XUJ, t.v(u, h(t)). The extension stochastic trace of v : Cl x T —> A onto T is a stochastic trace v : Cl x T —> A r  r  satisfying viu)^- ^)) 1  if Eli <G T,p ([O ,t )) r  otherwise  57  r  r  < p([0,t)) or p ([O ,t )) < p{T) r  r  r  where h' ^) 1  = {t\h(t) < t } G T°°. r  r  Both sampling and extension can be seen as "transformational processes on traces, hence they are transductions. Sampling is a type of transduction whose output is a sample trace of its input. Extending is a type of transduction whose output is an extension trace of its input. Proposition 2.21 Sampling and extending are continuous transductions  An event-driven transduction is a primitive transduction augmented with an extra input which is an event trace; it operates at each event point and the output value holds between two events. This additional event trace input of an event-driven transduction is called the clock of the transduction. Intuitively, an event-driven transduction works as follows. First, the input trace with the reference time T is sampled onto the sample time T generated by the event trace e. Then, the primitive e  transduction is performed on T . Finally, the output trace is extended from T back to T . e  e  Definition 2.45 (Event-driven transduction) Let T be a time structure and Fq : A  a primitive transduction. Let £  n x T  n x T  —» A  , i } x T  be the set of all stochastic event traces on time structure T. The  event-driven transduction of F is a mapping F  r  :£  f i x 7  " x A^  —> A '  xr  n x T  satisfying:  F% {v) otherwise. We will use F° to denote event-driven transduction F if no ambiguity arises. T  2.6  Dynamics Structures  With preliminaries established, we define an abstract structure of dynamics. Definition 2.46 (E-dynamics structure) Let E = (S, F) be a signature. Given a Tj-domain struc-  ture A and a time structure T, a E-dynamics structure V(T, A) is pair (V, F) such that • V = {A^ } s xT  sG  U£  n  x  T  where A^  xT  is a stochastic trace space of sort s and £  f i x T  is the  stochastic event space; • F = Fq U F%- where Fq is the set of basic transductions, including the set of transliterations {fq}feF,  the set of unit delays {Sq (v )} s,v eA ' s  s  58  se  s  S  e  m  set of transport delays  {A^  s  (T)(V )} £S,T>O,V £A , S  S  S  s  and the set of generators  {^^ }SGS> S  F%-  is the set of event-  driven transductions derived from the set of basic transductions, i.e., {F°\F £ FT}. We will now complete this chapter by enumerating the various properties of dynamics structures. The following propositions establish the fact that the partial order of a trace space and the partial order of an event space are epos. Proposition 2.22  The partial order of a domain is a cpo.  Proposition 2.23  The partial order of a stochastic trace space is a cpo.  Proposition 2.24  The partial order of a stochastic event space is a cpo.  The following propositions characterize the continuity of basic transductions in partial order topologies. Proposition 2.25  A transliteration f-j- : ^ 4  n x T  — >  A'QxT  o n  y  an  mple space fl and time struc-  sa  ture T is continuous if f : A—> A' is continuous. Proposition 2.26 A unit delay on any discrete time structure is continuous. Proposition 2.27  A transport delay is continuous.  The following proposition characterizes the continuity of event-driven transductions. Proposition 2.28 An event-driven transduction  F°  is continuous if its primitive transduction  F  on  any discrete time structure is continuous. The following theorem concludes these properties. Theorem 2.1 (S-dynamics structure) Let A be a H-domain structure and T a time structure.  The H-dynamics structure V(T, A) — (V, F) satisfies (1) V is a multi-sorted set of epos and (2) transliterations, transport delays and event-driven transductions in T are continuous in the partial order topology. If, in addition, T is discrete, all transductions in F are continuous in the partial order topology.  59  Transductions are functions. The well-definedness and strictness of a transduction is the welldefinedness and strictness of the function, respectively. The following propositions characterize well-defined and/or strict transductions in dynamics structures. Proposition 2.29  A transliteration f-r is well-defined iff function f is well-defined; fr is strict w.r.t.  an argument iff f is strict w.r.t. the argument. Proposition 2.30  Any delay is not strict. A unit delay on any discrete time structure is well-defined.  A transport delay is well-defined. Proposition 2.31  An event-driven transduction F° is well-defined iffF on any discrete time struc-  ture is well-defined; F° is strict w.r.t. its event input, and F° is strict w.r.t. one of the other arguments iffF is strict w.r.t. the argument. Event traces are non-intermittent and right-continuous. We call a transduction non-intermittent iff its output is non-intermittent whenever its input is non-intermittent. We call a transduction rightcontinuous iff its output is right-continuous whenever its input is right-continuous. The following propositions characterize non-intermittent and/or right-continuous transductions in dynamics structures. Proposition 2.32  A transliteration fj- is right-continuous if f is continuous in the derived metric  topology; fq- with f : X / A j —> A is non-intermittent if f is strict, well-defined and continuous in the derived metric topology. Proposition 2.33  A delay is non-intermittent. A transport delay is right-continuous.  Proposition 2.34  An event-driven transduction is right-continuous. An event-driven transduction  F° is non-intermittent if F is non-intermittent. For example, the "event or" transduction © (Figure 2.2) is well-defined and strict; it is also right-continuous and non-intermittent. "Event or" is a typical event synchronizer. In Chapter 5, Modeling in Constraint Nets, we will define other event synchronizers that are all non-intermittent and right-continuous. We have presented a topological structure of dynamics by formalizing time, domains and traces in topological spaces and by characterizing primitive and event-driven transductions. With such  60  a topological structure, continuous/discrete time and domains can be represented uniformly, and hybrid dynamical systems can be studied in a unitary model.  61  Chapter 3  Probabilistic Constraint Nets By definition, hybrid dynamical systems can have multiple sorts associated to diverse data types. Moreover, the various components of such systems can rely on different clocks, whether synchronized or event-based. The PCN framework developed in this thesis is designed to model stochastic systems as situated agents, which in general cannot be adequately described by existing deterministic frameworks. Furthermore, it allows for a complete hybrid modeling approach, where one can model time and domains as either discrete, continuous or both as well as incorporate a model for the uncertainty in the system. The flexibility of our framework is a great asset as it allows a designer to model a complex system under the umbrella of a single modeling language. As our framework extends the C N framework, we refer to the brief survey of the C N framework presented in § 1.3.1. For a thorough introduction, however, we direct the reader to the original work of [ZM95a].  3.1  Probabilistic Constraint Nets  The CN modeling tool was built on a topological view of time and domain structures along with notions of traces, events and transductions. As it became obvious from Chapter 2, we retained these concepts and extended them with a measure-theoretic approach and a notion of transduction encompassing random locations obtained via generators. These definitions represent the foundation for the notion of time and domain in the Probabilistic Constraint Net framework.  62  3.1.1  Syntax of Probabilistic Constraint Nets  Similarly to a constraint net, a probabilistic constraint net consists of a finite set of locations, a finite set of transductions and a finite set of connections. However, in order to be able to handle the uncertainty in the systems that we model, we add an essential component: the generator. A generator acts like a random number generator, following a given probability distribution and inducing a random location as its output. Thus, in practice, generators can be represented as discrete (e.g. Poisson, uniform) or continuous (Gaussian, exponential) probability distributions although we will use a general (and formal) measure theoretic definition. Definition 3.1 (Probabilistic Constraint Nets) A probabilistic constraint net is a tuple PCN  =  (Lc, Td, Cn), where Lc is afiniteset of locations, each associated with a sort; Td is afiniteset of labels of transductions (either deterministic or probabilistic), each with an output port and a set of input ports, and each port is associated with a sort; Cn is a set of connections between locations and ports of the same sort, with the restrictions that (I) no location is isolated, (2) there is at most one output port connected to each location, (3) each port of a transduction connects to a unique location. Intuitively, each location is of fixed sort; a location's value typically changes over time. A location can be regarded as a wire, a channel, a variable, or a memory cell. A output location of a generator will be viewed as a random variable. Each transduction is a causal mapping from inputs to output over time, operating according to a certain reference time or activated by external events. Note that probabilistic transductions are built of at least one basic transduction called a generator. Every generator is associated with a given probability distribution, either discrete or continuous, thus the sort of the output of a probabilistic transduction is the sort of its probability distribution. Connections relate locations with ports of transductions. A clock is a special kind of location connected to the event ports of event-driven transductions. A location I is called an output location of a PCN iff I connects to the output port of a transduction in Td; otherwise, since isolated locations are not allowed it is an input location. We will use the notation I (PCN) and O(PCN) to denote the set of input and output locations of a probabilistic constraint net PCN. A probabilistic constraint net is open if there exists at least one input location, 63  otherwise it is said to be closed. Another feature of our framework is its graphical representation. A PCN can be represented by a bipartite graph where locations are depicted by circles, transductions by boxes, generators by double boxes and connections by arcs. To differentiate them from deterministic locations, we depict random locations with double circles. It should be noted that, as mentioned above, a generator induces a random variable as its output location since the output port of a generator will follow a probability distribution. The transductions with this (random) location as input will then behave like a deterministic transduction with probabilistic inputs, thus generating probabilistic outputs which will follow a mixture probability distribution, the nature of which will be dependent on the transduction and on the distribution of the input locations. Therefore, even though we will talk about output locations of generators as being random locations, in the presence of feedback and at least one generator, all locations will be random locations. However, we chose to differentiate between locations and random locations as it provides a visual and intuitive way of assessing where uncertainty initially enters the system. Most commonly used families of probability distributions are parameterized, i.e., one can fully specify a probability distribution by giving values to the parameters of the family. By allowing input locations to be connected to generators (which are basic transductions), we also provide a very general framework that allows the designer to include any probability distribution in a model, whether it is one with static parameters or a more complex conditional distribution depending on time or on the current state of the system. The ability of generators to be dependent on certain locations of the model also greatly simplifies the design task when modeling a complex system for which the various uncertain inputs are not fully known. Indeed, specifying the parameters of a probability distribution is often hard and counter-intuitive. Therefore, a designer could set the parameters of the distribution to some default location value, and then, as the system evolves, learn the values of the parameters of the distribution, thus updating their values as a better estimate is being learned. For example, to model sensor noise with a PCN generator following a Gaussian probability distribution on the discrete time structure T = N , one would simply need to connect the inputs of the generator to the locations holding the static values of the mean fi and the variance a  2  to generate samples from the Gaussian distribution at every time point in T (see Figure 3.1).  64  Figure 3.1: Gaussian probability distribution as a Generator and random location.  PitLp) i  Figure 3.2: The probabilistic constraint net representing a Markov model. To exemplify the graphical syntax of PCN further, we present in Figure 3.2 an open probabilistic constraint net on a discrete time structure where / is a transliteration, 6 is a unit delay with initial value so and Pr(i ) is a generator following the given discrete probability distribution p. The p  random location i is, at each time step, a realization (sample) from the distribution p. This PCN is p  thus a representation of a probabilistic state automaton: s(0) = so, s(n +1) — f(i(n), i (n),s(n)). p  This state automaton can be showed to be equivalent to a Markov model with state space I x S. As an example of continuous time system modeling, Figure 3.3 displays a closed probabilistic constraint net representing the correct mathematical interpretation of a generic autonomous SDE (also known as an ltd process):  X = X + f f(X )ds + f g{X )dB t  t0  s  J to  s  s  (3.1)  Jto  Note that the second integral J in this equation is the ltd Stochastic integral which differs from the usual Riemann integral.  65  Figure 3.3: PCN model of an Ito process. The Modeling of Subsystems Complex physical systems may be composed of a set of subsystems which by interacting together in a hierarchical fashion produce the behavior of the global system studied. Based on the definitions of subnets and modules for CN, we introduce the notions of subnets and modules within the PCN framework. Definition 3.2 (Subnet) A probabilistic constraint net PCN\  PCN  2  = (Lc ,Td ,Cn ),  I(PCNi) C  2  2  2  written PCN  X  C PCN  2  = (Lc\,Td\,Cn\)  iff Lc C Lc , Td x  2  x  C Td , Cn  is a subnet of  2  x  C Cn and 2  I(PCN ). 2  Definition 3.3 (Module) A module is a triple (PCN, 1,0),  where PCN is a probabilistic con-  straint net, I C I (PCN) and O C O(PCN) are subsets of the input and output locations of PCN. We say that I U O defines the interface of the module. When it is clear by the context, we will use the notation PCN (I, O) to denote the module (PCN, I, O). Similarly to the general definition of a PCN, a module PCN (I, O) is closed if / = 0; it is otherwise open. Graphically, a module will be represented by a box with rounded corners. Moreover, locations in I (PCN) — I and O(PCN) — O are respectively called hidden inputs and hidden outputs and are used to model non-determinism in a system. Three basic operations that can be applied to obtain a new module from existing ones were introduced in [Zha94]; namely union, coalescence and hiding. We extend these operators for the PCN framework. The union operation is used to obtain a new module created by two modules side by side. Formally, let PCN  X  = (Lci,Td  1:  C m ) and PCN  2  66  = (Lc ,Td , Cn ) be two probabilistic con2  2  2  straint nets, with L e i n Lc  2  PCN (I ,0 ), 2  2  2  = 0 and Td n T d  written PO/V"i (7i, d )  1  = 0 \ then the union of PCNi(Ii,0\)  2  || PCN (I ,0 ), 2  2  2  is a new module PCN(I,0)  and where  PCA'' = (Lc, Td, Cn) is a probabilistic constraint net with Lc = Lei U Lc , Td = Td\ U Td and 2  2  C n = C n i U Cn , IUO defines its interface with I — h U I and O = 0\ U 0 . 2  2  2  The coalescence operator combines two locations in the interface of a module into one, with the restriction that at least one of these two locations is an input location. Formally, let P C AT = (Lc, Td, Cn) be a probabilistic constraint net, I e I and V e I U O be of the same sort, the coalescence of PCN (I, O) for I and I', denoted PCN (I, 0)/(l, I'), is a new module PCN'(I', O') with P C AT' = (Lc[l'/l],Td, Cn[l'/l}), I' = I-  {1} and O' = O, where  denotes that i in  X q is replaced by v. Finally, the hiding operation, as its name suggests, deletes a location from the interface by turning it into a hidden location. Formally, let PCN = (Lc, Td, Cn) be a probabilistic constraint net and I e I U O, the hiding of PCN(I, O) for I, denoted PCN(I, 0)\l, PCN'(I', O') with P C W = PCN, I' = I-  is a new module  {1} and O' = O - {I}.  In addition to these three basic operations, three combined operations have been developed in [Zha94]: cascade connections, parallel connections and feedback connections. Intuitively, the cascade connection joins two modules in series, the parallel connection does it in parallel and the feedback connection connects an output of a module to an input of its own. Since the extension of these operations to the PCN framework is straightforward, we simply illustrate them in Figure 3.4. The reader is referred to the original work on constraint nets for more details on these operations [Zha94]. The introduction of modules in our framework provides many beneficial effects. First of all, modules allow for the hierarchical composition of simpler structures for complex systems. This facilitates the design task as one can use existing modules as building blocks for modeling more complex systems. For example, consider the probabilistic state automaton of Figure 3.2 along with a deterministic transliteration labeled Output. By selecting either {i, s} or {i, s'} as the interface for this probabilistic constraint net, we obtain a probabilistic state automaton module PSA. By cascading the module PSA to the transliteration Output, we can construct the input/output probabilistic automaton IOPA showed in Figure 3.5 In addition to providing the ability to model systems hier'Note that Td is the set of transduction labels, which can be different for the same transduction.  67  PCNl PCNl Parallel Connection  ©  PCN2  PCN2  Oa^ PCN  ;°  Feedback Connection  Or?  ©  Figure 3.4: Combined Operations: cascade, parallel and feedback.  PSA  -9^- state  OUTPUT  IOPA  Figure 3.5: P C N model of an input/output probabilistic automaton (state denotes either s' or s). archically, modules provide an efficient and flexible way of constructing various different systems with the same set of basic components. For example, let us revisit the input/output probabilistic 68  automaton IOPA of Figure 3.5. Formally, a input/output probabilistic automaton is defined as a tuple (I, S, O, f , f ) where I , S and (!) represent the finite set of input values, states and output s  0  values respectively. f and f are known as the state transition and output functions. Note that the s  0  state transition and output functions need not be deterministic and can be uncertain, following a given probability distribution. The output function relates a value to the state of the system. For example, it can be seen as a state observation arising from a sensor reading. Deterministic output functions are not very interesting as they only provide a relabeling of the current state of the system. On the other hand, probabilistic output functions can be used to reason about the current state of the system when the actual state is unknown. For our example, if we select {i, s'} as the interface of the probabilistic state automaton PSA then IOPA is a Hidden Markov model [BP66, BE67, BS68, BPSW70] with f = / and f = g o f. s  Q  If, on the other hand, we select {i, s} as the interface of PSA, then we get that IOPA has a deterministic output mapping f  0  — g and hence is a Markov model. Note also that for the former  choice of interface, IOPA is equivalent to a probabilistic Mealy machine [Mea55] while the latter choice of interface leads to a probabilistic Moore machine [Moo56] with a deterministic output function. Finally, another reason to introduce modules in our framework is the ability to capture internal structure of a system via hidden outputs and to introduce nondeterminism with hidden inputs. With the addition of nondeterminism to our probabilistic modeling framework, we are now able to model completely general systems with any type of uncertainty, whether it be probabilistic, stochastic or nondeterministic.  3.1.2  Semantics of Probabilistic Constraint Nets  We have briefly introduced the syntax of the probabilistic constraint nets model, which has the useful properties of being graphical and modular. However, the syntax does not provide a meaning for the model. Indeed, there are multiple models with similar syntax to probabilistic constraint nets (Petri Nets [Pet81] and their generalization Coloured Petri Nets [Jen81] for example) that have completely different interpretations. Therefore, it is necessary to have a formal semantics of probabilistic constraint nets in order to correctly interpret models of complex physical systems. The fixpoint theory of partial order has been used as a semantical model for programming lan69  p(x=l) = 0.75 p(x=2) = 0.25  ©  p(Output=6) = 0.3 p(Output=7) = 0.1 p(Output=ll) = 0.45 p(Output=12) = 0.15  E(Output) = E(X+Y) = E(X) + E(Y) =9.25 P(y=5) = 0.4 p(y=10) = 0.6  Figure 3.6: Simple PCN for a probabilistic sum. guages and models [Hen88]: in this case, a program (or a model) defines a function / and its semantics are defined to be the least solution of a; = f(x), or the least fixpoint of / . A similar approach was developed to provide a fixpoint semantics for the Constraint Net model [ZM95a]. However, even though our framework is similar to that of Constraint Nets, the semantics of PCN differ significantly from that of CN. This is due to the fact that we have now introduced uncertainty in the set of equations induced by the PCN model. Hence, a probabilistic constraint net is a set of equations with locations serving as variables. Some of the variables (locations) in the equations, those that are outputs of generators, are in fact random variables, obeying some probability distribution, which in turn affect the value of the transductions for which they are inputs. Transductions play the role of functions and the connections between locations and transductions generates a set of equations. Obviously, the semantics of a PCN should be a solution to this set of equations containing random variables. Figure 3.6 demonstrates the effect of random locations on the transductions. Transduction Add is a very simple transliteration representing the sum of two (probabilistic) inputs X and Y. It is easy to notice that the output value for this transliteration also follows a probability distribution. In this case, there are 4 possible values which each have different likelihood of occurrence. One should note that although the distribution of a random variable is helpful in reasoning about its behaviour, one can reason about statistics such as the expected value, that is, one can redefine the notion of behavior in terms of average behavior for the system. In our simple example, we can see that the average output value of the system is 9.25. Since the equations in a PCN model do not converge to a fixpoint but rather to a stationary distribution, the fixpoint theory of partial order cannot be utilized directly to provide a denotational  70  0  20  60  ,.40  time  80  0  20  ..40  time  (a) = -X (X -l)(X -2);  t  ' '  80  (b)  (a)ODE:X l g U r C  60  t  t  X  t  = -1 and X  0  = 1.5  0  (b)SDE:X = - X ( * t - l ) ( * t - 2 ) + JVt; X = - 2 t  t  0  semantics for PCN. In fact, in the presence of uncertainty in the system, the least solution of an equation with random variables is a Markov stochastic process. To further illustrate the difference between the semantics of a deterministic system (CN) and one encompassing uncertainty (PCN), let us compare two dynamical systems with nominal component X =  -X (X -l){X -2).  T  T  t  t  The first one is deterministic and has two distinct stable attractors (equilibria), at 2 and at 0, as 2  shown in Figure 3.7(a). The behaviour of this system is fully determined by its initial value and it reaches one of the two stable fixpoints based on this initial value. The second system, which cannot be modeled with a constraint net, is stochastically affected by a simple Brownian motion process. A sample path for this system, for an initial value of XQ = —2, is shown in Figure 3.7(b). For this specific realization, the system is initially attracted toward the closest equilibrium which is at X = 0. The system then fluctuates around this attractor, reacting under the influence of the Brownian motion component and, around time t = 12, a large enough noise disturbance pushes the system over the value of 1, causing the system to be attracted toward the other equilibrium, at X = 2. Another spike of noise flips the system back to the lower equilibrium at t = 35 and so on. This example shows the effect of uncertainty on the system and its behaviour. In this case, there is no fixpoint for this realization nor for the full system. For a set of sample paths with There are in fact three different equilibria, at 0, 1 and 2 respectively. However, the equilibrium at 1 is unstable. Any shift in value will cause the system to move away from this unstable equilibrium and move towards one of the other two stable equilibria. 2  71  Distribution ot dx - -x(x-1)(x-2)dt +dBt 0.4  0.35  0.3  0.25  £  0.2  0.15  0.1  0.05  0 -1  -0.5  0  0.5  1  X  Figure 3.8: Density of dX = -X(X  1.5  2  2.5  3  - 1)(X - 2)dt + dB . t  non-zero measure, the system will keep moving back and forth between the two stable equilibria as it is affected by the noise introduced by the Brownian motion component of the equation. However, the system will reach a stationary distribution. That is, in the long run, the probability distribution of the system will remain unchanged, independent of time. The corresponding density function for this distribution is shown in Figure 3.8. One can clearly observe that the system is symmetrically distributed with higher weight around the two stable equilibria located at X = 0 and X = 2. One should note that if the effect of the Brownian noise is diminished, the peaks at X = 0 and X — 2 rise or fall (depending on the starting value) as the noise is less likely to cause a jump large enough to cause the other equilibrium to become the main attractor. Letting the effect of the noise converge to zero would lead to the deterministic case as presented in Figure 3.7a), that is, the stationary distribution would be degenerate everywhere except at the equilibrium corresponding to the initial value of the system. Hence a deterministic system is in fact a simple case of the more general stochastic system. We define the semantics for the Probabilistic Constraint Net model to be the least fixpoint of the distribution of the solution to the set of equations of the PCN model. These semantics are, as it was mentioned in the previous paragraph, applicable to any system, whether it be stochastic or deterministic.  72  3.1.3  Fixpoint in distribution of partial orders  A fixpoint in the distribution of a function / can be considered as a solution of the equation x = f(x), where /(•) is an stochastic function. The least fixpoint is the least element in the fixpoint set. Definition 3.4 (Fixpoint in distribution and Least fixpoint) Let f : Cl x A —> A be a function on  a sample space Cl and a partial order A. A function g : Cl x A —> A is a fixpoint in distribution of f iff the distribution of g is a stationary distribution for f. It is the least fixpoint in distribution off  iff, in addition, F < F i for every other function g' which is a fixpoint in distribution off. g  g  Least fixpoints in distribution, if they exist, are unique. The least fixpoint in distribution of / will be denoted by p.Fj. Based on the above definition, we can state our first fixpoint in distribution theorem as follows. Theorem 3.1 (Fixpoint Theorem I)  Let Abe a cpo and assume that either A is also a total order  or that the set of distributions over A is a cpo and the function over distributions is continuous Then, every continuous function f : Cl x A —> A or pathwise continuous function f^-.A^A  (for  afixedLU £ Cl) has a leastfixpointin distribution. We now present our second fixpoint in distribution theorem which is applicable to a function of two arguments. Theorem 3.2 (Fixpoint Theorem II) Let  A and A' be two epos and assume that either A, A' are  also total orders or that the set of distributions over A' is a cpo and the function over distributions is  continuous. If f : Cl x A x A' —> A' is a continuous function, then there exists a unique continuo function p.f : Cl x A —> A', such that for all a £ A, the distribution of(p.f)(a) is the least fixpoint in distribution ofXoj, x.f (a, x). UJ  The distribution of the continuous function p.f : Cl x A —> A' is called the leastfixpointin  distribution of function / : Cl x A x A' —> A' or the least solution of the equation y — f(x,y). Continuous and pathwise continuous functions can also be extended. Proposition 3.1  Let I C J be an index set. If f : Cl x (xjAi) —> A is a continuous or pathwise  continuous function, then the extension off, f : Clx (xjAj) —> A satisfying f'(u), a) = f(oj, a\j), is a continuous or pathwise continuous function. 73  Formally, a set of equations can also be written as o = f(u, i, o) where i is a tuple of input variables and o is a tuple of output variables. Based on our previous results, if / is continuous, then its least fixpoint in distribution is a continuous function, denoted p.f.  3.1.4  Semantics of Probabilistic Constraint Nets  In this section, we define the fixpoint in distribution semantics of probabilistic constraint nets. Let E = (S, F) be a signature and c G 5 be a special sort for clocks. A probabilistic constraint net with signature E is a tuple PCN-^ = (Lc, Td, Cn) where • each location I G Lc is associated with a sort s G S, the sort of location I is written as s/; • each transduction F G Td is a basic transduction or an event-driven transduction, the sorts of the input and output ports of F are as follows: 1. if F is a transliteration of a function / : s* —> s G F, the sort of the output port is s and the sort of the input port i is s*(i); 2. if F is a unit delay 5 or a transport delay A , the sort of both input and output ports is s  s  s; 3. if F is an event-driven transduction, the sort of the event input port is c, the sorts of the other ports are the same as its primitive transduction; Let V(T, A) = (V, F) be a E-dynamics structure. PCN-£ on (V, F) denotes a set of equations {o =  F (X)} Z (PCN)> 0  0(  0  s u c  h that for any output location o G O(PCN),  • F is a continuous or pathwise continuous transduction in F whose output port connects to o, 0  • x is the tuple of input locations of F , i.e., the input port i of F connects to location x(i). Q  Q  The semantics of a probabilistic constraint net is defined as follows. Definition 3.5 (Semantics) The semantics of a probabilistic constraint net PCN on a dynamics structure (V, F), denoted [ P C A ] , is the least stationary distribution of the set of equations {o = r  Fo(x)}oeO(PCN)> given that F is a continuous or pathwise continuous transduction in J for all 7  0  o G O(PCN); it is a continuous or pathwise continuous transduction from the input trace space to the output trace space, i.e., [ P C W ] : Xi(pcN)A^*  r  74  — >  Xo(PCN)A^* . T  Realization of f(x) = 0.5x +y 5  • |  :  0 J  fbi  1  201  301  401  501  time  Figure 3.9: Sample path of the system in Figure 2.3 from page 57. Given any set of output locations O, the restriction of \PCN] onto O, denoted [PC/V] \Q : —> XoAj , is called the semantics of PCN for O. For example, the probabilistic  i(PCN)Af  x  i  o  constraint net in Figure 2.3 on page 57 denotes equations x' = f(x,ui) = 0.5x + y(u>) and x — S(0)(x) with Fy — Uniform({l,2})  and Q = {u>i,u }- Given a discrete time structure N , 2  a domain 1 — {1,2} for inputs and a domain O = M for output, the semantics for x is F : j  n  x  N  I  n  x  N s u c  h that F(w)(0) = 0 and F(v)(n) = f(F(v)(n - l),v{n - 1)) where the  limiting distribution for F is stationary. Let us show the derivation of the semantics of this model (see Figure 2.3). In Figure 3.9, we plot a realization trace of the system, while in Figure 3.10 we can see the empirical distribution of the system after 10000 time steps. The least fixpoint distribution follows a uniform distribution over the range [2,4]. The evolution of the distributions is presented in Figure 3.11. One can see that the system's distribution starts as uniform over the range {1,1} and the distribution gradually increases to reach a stationary distribution which follows a uniform distribution over [2,4].  3.1.5  Semantics of Modules  In the previous section, we have formally defined the semantics of a probabilistic constraint net as a stationary distribution of a transduction. Now let us introduce the semantics of a PCN module. Formally, we define the semantics of a module as a set of transductions.  75  Empirical Distribution 1400  2125  2.375  2.625  2.875 3.125 3.375  3.625  3.875  f(x,y)=0.5x + y  Figure 3.10: Empirical Distribution of Example 2.3.  Definition 3.6 (Semantics of P C N modules) Given that the semantics of a probabilistic constraint netPCN is\PCN\ is \PCN{I,0)\ x  : x A^.  -> x A^ ,  xT  0[PCN)  = {F : x M "  -» x A^ }  x T  where F (i) = [PCN]\ (u,i)  r  u  l(PCN)-lA^*  the semantics of a module PCN{I, O)  xr  I{PCN)  0  u  ueU  0  and U C  is the set of well-defined hidden input traces.  r  Proposition 3.2 Here are some properties that can be inferred from module operators. The semantics of a composite module can be derived from the semantics of the components with which it was constructed: • Cascade connection: IfPCN(I,0) [PCN(I,0)]  = PCN (I ,0 ) 2  = {F oF \F 2  1  1  • Parallel connection: IfPCN(fO) [PCN(I,0)]  = {(F F )\Fi lt  2  2  o PCN {I ,0\),  2  x  G [PCN (I ,0 )lF l  l  1  x  G  2  = PCN {I Ox)  2  G {PCN (h,0 )lF 76  1  2  2  e  then  \PCN {I ,0 )\}. 2  + PCN (I ,0 ),  u  l  x  2  2  2  then \PCN {I ,0 )\} 2  2  2  Probability Distributions of F"(x)  F (x) n  4. Unrtorm(2,4)  3-J ,50%  /tzf  2.  1-1  ] n  oo  2  100%  Figure 3.11: Evolution of the distributions of f(x). where (F , F > , (i) = F i ( t | ) and ( F F ) x  2  0l  7l  l 5  • Feedback connection: IfPCN'{I',0') [PCN'(I',0')]  2  | 0 2  (i) = F (i ). 2  | / 2  = F(PCN(I,0)), then = {p.F\F e  [PCN(I,0)}}  where p.F is the the leastfixpointof F. • Union: IfPCN(fO)  = PCN {h,Ox)  [PCN(I,0)j  x  \\ PCN {I ,0 ), 2  = IPCN^O^J  2  2  x  then {PCN {I ,0 )\. 2  2  2  We say that a probabilistic constraint net PCN is well-defined iff its semantics, transduction [PCN}, is well-defined. For example, consider again the probabilistic constraint net in Figure 3.2. Given a discrete time structure (e.g., T — N), a well-defined function / and a proper probability distribution Pi, the P C N is well-defined. Similarly, we will say that a module is well-defined iff all the transductions in its semantics are well-defined. Moreover, if a probabilistic constraint net is well-defined, then, by definition, all its modules are well-defined. One important properties of modules is that their well-definedness is closed under the following module operations. 77  Proposition  3.3 If CN\(I\,Oi)  and CN2(h,02) are well-defined modules, then the following  resulting modules: • Union Connection: PCNi{h,Oi)  \\  • Cascade Connection: PCN\(I\,0\) • Parallel Connection: PCN^IuOi)  PCN {h,0 ), 2  o  2  PCN (I ,0 ), 2  2  + PCN (h,  2  0 ),  2  2  are well-defined modules. Note, however, that well-definedness is not closed under the feedback operation. The following proposition denotes the relationship between the well-definedness of a probabilistic constraint net model and the strictness of the transductions in the model. Proposition  3.4 Let A and A be two epos and assume that either A and A are also total orders 1  1  or that the set of distributions over A' is a cpo and the function over distributions is continuous. f : fl x A x A' —> A' is a strict continuous function w.r.t. its third argument (a' G A'), then the leastfixpointof f, or the least solution of the equation o — f(u>, i, o), is undefined. To illustrate this property, consider the PCN model of Figure 2.3. For the purpose of this example, let +, • : R x R —> R represent the strict extensions of the addition and multiplication operators,+ and •, respectively. Obviously, one can derive the equivalent transliterations associated with these two operators: +, • : R  xR  —> R . The least solution of x = 0.5x + y on the dynam-  ics structure V(T, R) is undefined, even though g, with F = Uniform([2,4]), is a well-defined g  random variable which is the least fixpoint in distribution. In general, and as illustrated in the example above, the presence of an algebraic loop precludes probabilistic constraint nets from being well-defined. We now define formally the notion of algebraic loop with the PCN framework and discuss its implications. Definition 3.7 (Algebraic loop) A location I is strictly dependent on a location I' in the probabilis-  tic constraint net PCN, denoted by I <—  iff: (1) there is a transduction F in PCN such that I i  the output location of F, I' is an input location of F, and F is strict w.r.t. the input port (indicatin  an input argument) that connects with V; or (2) 31" : / <— I", I" <— I'. We say that P algebraic loop on a location I iff I is dependent on itself, i.e., I <— I. 78  Proposition 3.5 (Adapted from Proposition 4.2.10 of [Zha94]) A module PCN (I, O) is not well-  defined if there is an output location I G O such that PCN has an algebraic loop on I. It is very common that an output location is also an input location of the same transduction. In fact, this is exactly the effect of the feedback connection. Hence, we need a mechanism to break algebraic loop and hence obtain a well-defined net. A common strategy to break an algebraic loop is simply to insert a delay since all real components have non-zero delays. For example, by inserting a unit delay 5(0) to the equation x = 0.5a; + y, we have z = 0.5x + y, x — 6(Q)(z). This is a welldefined probabilistic constraint net which has the well-defined semantics equivalent to a uniform distribution over the range [2,4].  3.1.6  Family of Probabilistic Constraint Nets  As mentioned in the previous section, systems are often parameterized, yielding a family of systems whose behaviours may differ significantly from each other due to their parameter value. In this section, we introduce the notion of parameterized probabilistic constraint nets and present the notion of limiting semantics of such PCN models. Let us first introduce the notion of a static parameter. Formally, a static parameter is a variable in a transduction whose value does not change over time; hence a static parameter could be an input location of a transduction only if this location's value does not change over the life of the system. Typical examples of static parameters of robotic systems include physical constants such as gravitational force, mass, friction coefficient as well as elements specific to the system of interest like initial state, time delay and parameters of probability distributions such as mean (p) and variance (a ) in the case of a Gaussian distribution. Note that for a static parameter the value is kept constant 2  throughout the life of the agent. Each value thus represents a different agent, taken from the family of agents generated by the space of the parameter. We now formally define a parameterized probabilistic constraint net. Let PCN be a probabilistic constraint net and P be a set of static parameters in PCN. We use PCN  ¥  and PCN (I,0) V  to  denote a parameterized net and a parameterized module, respectively. Each static parameter p G P is associated with a set of values D . The cross-product xp_D is called the parameter space of the p  p  system. The semantics of a parameterized net PCN^ is defined as follows.  79  Definition 3.8 (Semantics of parameterized PCNs) The semantics of a parameterized probabilistic constraint net PCN , P  denoted {PCN^J, is a mapping from the parameter space to the set of  — >  transductions of the system, i.e., [PC/V ] : Xf>D —> (X ^ N) ^ 1P  A  p  that for any static parameter tuple v 6 x D , r  I  XT  PC  Xo(PCN)A^ )  such  xT  \PCN ] (v) = [PCN[v/F]} where PCN[v/F] P  p  denotes that each p G P in PCN is replaced by its corresponding value v(p). denoted [PCN (I, O)], is now also a  The semantics of a parameterized module PCN (I,0), p  function of the parameters: {PC N (1,0)}(v) = p  P  [PCN(I,0)[v/P]].  The main reason for introducing the notion of parameterized nets is two-fold. First, this enables a system designer to model and analyze a system under variations of its static parameters. Parameters can have a significant impact on the behaviour of a system. For example, a certain subset of the parameter space might cause the system to be unstable while the rest of the parameter space renders the system stable. For example, consider a parameterized version of our example of Figure 3.2. Let A; be a gain parameter with  — R, and z = kx + y,x — 5(0) (z), with y be time independent,  uncorrelated output from a uniform distribution over {1,2}, be a probabilistic constraint net on dynamics structure £>(N, R). As seen before, the semantics of this system is a sequence of random variables 0, 2 + y\, kyi + J/2,  k yi + + 2 / 3 , • • • where yi denotes the realization from a uniform random variable over (1,2} 2  at time instant i. An analysis similar to the deterministic case can be performed to reason about the effect of parameter k on the system's behaviour. For general values of k we have l i m ^ o o z(n) = 2~2i=o k Vn-i- ^ is l  e a s  y to show that this series converges for |fc| < 1 and diverges for any other  value, i.e., \k\ > 1. Moreover, we have that E(z ) = E ( £ " n  = 0  fc^n-O = E(y)  2/i's are independent and identically distributed. In the limit, for E(y)lim ^ n  oc  J^Lo  k%  = T=k>  w  i  t  h  ^"in- oo-^(n) = >  £r=o  k i s i n c  e the  < 1, we get lim ^ E(z ) n  00  =  n  Uniform({2,4]) as discussed earlier.  Second, by introducing parameterization of PCN models, we allow for the notion of limiting semantics to be defined. Let P be a set of parameters, x p / J be the parameter space, and < o p  a partial order relation. If ( x Z ? , P  p  is a linear order, and PCN  P  < fJ > Xp  p  net whose semantics is a mapping {PCN ] r  PCN  V  Xr  : x^D —» x A^ , xT  p  Lc  p  be  is a closed parameterized the limiting semantics of  w.r.t. the parameter set P, written \PCN*\, is defined as the limit of the linear set of traces  [PCN*], i.e., [PCN*] = lim[PCW ]. lp  An important parameter used with limiting semantics is the infinitesimal, denoted by e. Let e 80  be a parameter over the real range D — (0,1) C IR associated with the partial order relation <£> e  defined as ei <£> e iff e £  2  the parameterized net PCN  2  £  <R  t\.  e  is a linear order. Then, the limiting semantics of  (D ,<D ) €  t  is lim^oI-PCN }. Limiting semantics will prove essential to reason e  about the semantics of stochastic temporal integration, which we introduce in the following section.  3.1.7  Stochastic temporal integration  One of the most important transductions on continuous time structures is temporal integration. Several issues are raised when considering continuous uncertainty present in a system. Brownian motion is the only stochastic process with continuous paths that has desirable properties for modeling uncertainty. But how does one integrate over Brownian motion processes? Can it be done in a similar way to common Riemann integration following, for example, the forward Euler method? Furthermore, a desired property of temporal integration within the PCN framework, either deterministic or stochastic, is to be defined on any time or domain structure. To ensure the full abstraction of temporal integration, we define temporal integration on vector spaces (whether numerical or symbolic) and provide the semantics of probabilistic constraint nets with temporal integration using limiting semantics as introduced above. In this section, we will focus mainly on the stochastic integral, namely the Ito type. There are two main types of stochastic integral, namely Ito and Stratonovich integrals. Both interpretations have their specific uses in mathematical modeling, depending on the nature of the system being modeled. In subsequent sections however, we will refer to the Ito version of the integral as a simple transformation converts Ito to Stratonovich. For a thorough treatment of numerical simulation of SDE, the reader is referred to [HigOl]. Note also that a deterministic temporal integral arises as a special case of the stochastic version. For a complete discussion of deterministic temporal integration, the reader should consult §4.2.5 of [Zha94]. First, let us briefly introduce building blocks of temporal integration in PCN, i.e., the notion of vector space. For a thorough discussion on vector spaces, the reader is referred to [War72] or any introductory vector algebra and topology textbook. Formally, a vector space is a set X associated with the functions sum and product: + : I x X - » I and • : R x X —• X and with Ox € X satisfying the following conditions:  x + y = y + x, (x + y) + z = x + (y + z),  81  a(x + y) = ax + ay, (a + j3)x = ax + fix, a((3x) = (af3)x, x + Ox = x, Ox = Ox, lx — x. As it is conventionally the case, we denote the sum of all elements in {xi}i^i with S / X j . Furthermore, a topological vector space is defined as a vector space with a topology such that the two operators + and • are continuous functions. We will assume that + and • are strict extensions. Let U be a vector space with functions + : U xU —> U and • : K x U —> U continuous in metric topology. Stochastic temporal integration J(SQ, B ) : U u  a Brownian motion trace  :W  —> U with an initial state SQ £ U and  can be defined as follows.  }xr  First, we define stochastic temporal integration on a discrete time structure. For a discrete time structure T , we have that for all t > 0, pre(t) denotes the previous time point. Moreover, assume that we have a B that is a discretized Brownian motion trace over T . Stochastic temporal w  integration is defined as follows:  f  n8o,B )(u)=Xt. u  {  f so  if t = 0  ( V ><t{B {t') - B (pre{t'))) • u(pre(t')) otherwise.  J  0<t  u  u  We can also represent J{SQ, B ) as the least solution of the equation u  s = S(s )(s) + dBu • 6(0)(u) 0  with  {  ift = 0  0  Bu(t') - Bu{pre(t')) otherwise. This equation for stochastic integration can be represented by a probabilistic constraint net over discrete time structures. Let us now extend temporal integration to an arbitrary time structure. For any arbitrary time structure T , stochastic temporal integration is defined using an infinitesimal event trace. More specifically, let T be a discrete sample time of the arbitrary time structure e  T . T is generated by an event trace e with e = A(e)(0)(->e) for an infinitesimal parameter e. We e  define int B {u, SOT  u  s) = 6(so)(s) + dB • 6(0)(u) for a Brownian motion trace B u  u  on T . Stochas-  tic temporal integration J (so, BJ) can be computed by a PCN module PCN(u, s) where PCN represents the following two equations: s = int° (e,u,s,Bu),  e = A(e)(0)(->e)  So  82  where e > 0 denotes an infinitesimal. This definition can be considered as derived by the Euler-Maruyama method [Mar55], the stochastic equivalent to the forward Euler method. Its convergence has been proved in [GS04]. It represents the most-studied, best-understood and simplest-to-implement numerical method. Despite its popularity, the Euler-Maruyama method suffers from lack of numerical stability, low convergence order, incorrect stationary laws and some problems with the geometrical invariance properties. However, in this dissertation we are interested in semantics, rather than numerical simulation of stochastic differential differential equations. The reader is referred to [KP99] for an in depth introduction to the numerical solution and simulation of stochastic differential equations. As an example, let us investigate the limiting semantics of the classical stochastic integral Jo B (s)dB(s) for a given Brownian motion B (s). For this example, we have U — R , T = R . +  UJ  w  This equation can be modeled by a closed probabilistic constraint net represented by three equations: s  e = A(e)(0)(-.e), .u = Bu.  = int° {e,u,s,Bu), So  The solution for e is:  {  0  if I - I is even  1 otherwise.  It can be shown that the exact solution of this SDE is [Oks98, Mao97]  J^B {s)dB{s) = \{Bl{t)-t)  (3.2)  w  Now let us demonstrate that the limiting semantics of this stochastic integral leads to this exact solution. The solution for s is the least solution of s = int° (e, B ,s,B ). u  Let s° = Ai. _LK be the least  u  element. Then following the proof of the Fixpoint Theorem we get  s  1  =  int {e,B ,s°,BJ) o  S0  u  Ai.  so  if i < e  -LR  otherwise,  so s  2  =  m^^e.Soj^ ,^) 1  = Ai.  if i < e  s + 5 ( 0 ) ( 5 a ; ( e ) -B (0)) 0  w  -LR 83  u  ife<i<2e  otherwise,  so  ift<e  so + B (0)(B (e) u  s  3  =  int°(e,B ,s ,B )  = \t.{ s + B^'B^e)  1  u  u  - B (Q))  u  -  0  if e < t < 2e  U  B (0))+ u  B (e)(B (2e) - B (e))  if 2e < t < 3e  J_K  otherwise,  UJ  UJ  u  if t < e  so  if e < t < 2e =  int Je,B ,s ,B ) 0  = \t. { so + B (0)(B (e) - B (0))  k  a  u  u  u  w  u  iffee< i < (fc + l)e otherwise. -(B 'ke)-B ((k-l)e)) ] Let s = V-R+ { }- Then s — A i . s ' - ' J ( 7 J ) is the least solution of the equation s = int°(e, B , s, 2  u  sfc  u  +1  u  With a simple manipulation we get that  s  k+1  = \ ^B (t)  2  u  - B (0) ~ u  The term Ej=o(- ^(( + ) ) ~ B^ke)) B  fc  1  e  I>"((  fc + )) X  £  «( »  B  ke  2  j  (3.3)  in Equation 3.3 can be shown to have expected value t  2  and variance of 0(et). Hence, for small et we expect this random variable to be close to the constant t. Therefore, the limiting semantics of the net for s is s* = Xt. lim _»o s(t) — ^(B (t) — t) , which 2  e  corresponds to the exact solution of s — j  l Q  B^dB^.  Note that limiting semantics only applies to a closed parameterized net and is not composite. For a probabilistic constraint net with multiple temporal integrators, we will use a single infinitesimal for all the integrator transductions. Furthermore, note that an Ito equation might not have a unique solution on the whole interval [0, t}. For example, Girsanov [Gir62] has shown that the one-dimensional Ito equation x(t)=  t\x(s)\ d (s) a  B  Jto  84  has infinitely many solutions when 0 < a < 1/2. In the next chapter, we will address the issue of existence and uniqueness of the solution of a stochastic differential equation and discuss the conditions under which a probabilistic constraint net produces the "correct" solution. Analogously to the approach in the CN framework, we can define three variations of stochastic temporal integration: (1) stochastic temporal integration with bounds, (2) stochastic temporal integration with reset, and (3) stochastic integration against another trace on domain B L As the extension of these variations of temporal integration is straightforward and their names are selfexplanatory, we refer the interested reader to [Zha94] for a detailed description of these alternate temporal integrals which can prove very useful when modeling complex dynamical systems.  85  Chapter 4  Modeling in PCN As denned in the previous chapter, a probabilistic dynamical system is built on a dynamics structure £>(T, A) where T and A denotes a time structure and a domain structure, respectively. The time and domain structures are abstracted so that they can be either continuous or discrete or hybrid. In Table 4.1, we present examples of the most commonly used models for probabilistic/stochastic dynamical systems in each of the four basic situations that can occur with discrete/continuous time and domain. In this chapter, we will discuss the issues related to modeling various types of systems with the PCN framework. In the next chapter, we will demonstrate how each of the models enumerated in Table 4.1 is subsumed by the PCN framework and show how to translate them into an equivalent PCN.  4.1  Events  We are interested in modeling the larger class of probabilistic dynamical systems encompassing components of more than one basic type. These systems are referred to as hybrid systems. We have Time Discrete Continuous  Domain Discrete  Continuous  Markov chains Dynamics Bayesian Networks Continuous Time Markov Chains  Stochastic Difference Equations Stochastic Processes Stochastic Differential Equations  Table 4.1: Generic Types of probabilistic/stochastic models  86  developed Probabilistic Constraint Nets (PCN) as a formal model for probabilistic hybrid dynamical systems. Within the PCN paradigm, a probabilistic hybrid dynamical system consists of modules with different time structures, with its domain structure multi-sorted and with a set of probabilistic generators, as basic transductions, which allows for the modeling of the uncertain components of these modules. As mentioned in the previous chapter, event-driven modules constitute an important part of our framework as they allow us to model systems with modules that are associated with different clocks. Hence, we can unify, within the same model, modules with different sample time structures generated by event traces. There are two ways in which an event trace can be generated: either with a fixed sampling rate, or by an event generator that reacts to changes in its inputs. Moreover, we can also combine multiple event traces, yielding new event traces. Typically, event traces are combined using event logic which allow various asynchronous components within a given set of modules to be coordinated. Common logical interactions are "event or", "event and", and "event select". With event logic modules, asynchronous components can be coordinated. As event control logic, and event generators and synchronizers in PCN are analogous to those introduced in CN, we will simply summarize the most important concepts here. For more details, the reader is referred to the original work on the Constraint Net framework [Zha94].  4.1.1  Event generators  Events generators are one way in which events traces can be produced in the PCN model. Formally, an event generator is a transduction whose output is an event trace. Consider for example the transport delay e = A(£ )(0)(->e). Recall that "(0)" indicated the initial output value of the delay is s  0. This transduction is an event generator whose output is an event trace with a fixed sampling rate. In Figure 4.1, we show two examples of basic event generator modules which were implemented as follows: • Module NotEqual(i,o) (Figure 4.1(a)) is composed of a unit delay and a transliteration notEqual where notEqual : B x B —> B is defined as notEqual(x, y) = cond(x, y, 0,1). • Module F(i,o) (Figure 4.1(b)) is composed of a unit delay and a transliteration / where / : B x B —> B is denned as f(x, y) = cond(x, 0, y, -iy).  87  ©  ^~ not Equal  \  ©  ^ 0  O^5(0)H NotEqual i  F (b)  (a)  Figure 4.1: Two basic modules for event logics: NotEqual and F. where cond is the conditional function defined in Equation 2.4. Although transliterations notEqual and / are applied to a different second input y, one can easily deduce that both these transliterations act as an "exclusive or", ©. An important property of event generators is that any cascade connection to an event generator is also an event generator. For example, an event generator that generates an event whenever its input changes from 0 to 1, usually referred to as a rising transition, is a cascade connection of NotEqual to F, i.e., F o NotEqual.  Obviously, event generators can also include uncertainty by simply encompassing generators in the module. We now introduce an event generator that will prove very useful for modeling stochastic systems with a continuous time structure. Let us consider an event generator process in which events occur randomly in time. The phrase events occur randomly in time is generic and could represent, for example: • The times when a piece of radioactive material emits particles • The times when customers arrive at a service station • The times when requests arrive at a server computer • The times when accidents occur at a particular intersection Consider the following counting process: {X(t) | t € R  +  U {0}}. Such a process is called a  Poisson process (named after Simeon Poisson) [Poi37] with parameter A if  88  1. The probability that at least one Poisson arrival occurs in a time period of duration r is P(T) =  XT +  O{T),  where o(r) denotes a term that goes to zero faster than kr as r goes to zero (for any constant k). Mathematically, lim ^ ^pT  0  — 0.  2. Let N(t) be the total number of Poisson arrivals occurring in the interval [0, t]. We assume thatiV(O) = 0. For the interval {ti, t }, the number of Poisson-type arrivals [N(t ) — N(ti)] 2  2  for t > t\ > 0 is dependent only on (t — t\) and not on t\ or N(ti). 2  2  3. If 0 < ti < t < £3 < £4 < . •., the numbers of arrivals occurring in disjoint time intervals 2  [N(t ) — N(ti)], [AT(£4) 2  — Nfa)],...  are mutually independent random variables.  4. The probability that two or more Poisson arrivals occur in a time interval of length r is 0(T). The basic underlying assumption of a Poisson process is that the behavior of the process after an arrival should be independent of the behavior before the arrival and probabilistically be like the original process. This property is often referred to as regeneration [Fel68]. In particular, the general regeneration assumption means that the times between arrivals, known as inter-arrival times, must be independent, identically distributed random variables. Furthermore, regeneration should occur at a fixed time t. In particular, if the first arrival has not occurred by time t, then the time remaining until the arrival occurs has the same distribution as the first arrival time itself. This is known as the memoryless property and can be stated in terms of a generic inter-arrival time X as follows: P(X >t + s\X > s) = P(X > t), Vs, t > 0. One can also easily show that the only probability distribution which possesses these properties is the exponential distribution. Based on these results, one can easily show that the occurrence time of events follows the iteration: log(U(0,l)) tn+l  — tn  ^  ( -iJ 4  where U(0,1) is a random variable following a Uniform distribution with parameter (0,1) and A is the rate of the exponential distribution. Hence, an event generator following a Poisson process will yield events at uncertain times ti, i = 0, • • • , following Equation 4.1. 89  4.1.2  Event synchronizers  A possible way of handling events is to modify existing event traces. For this purpose, event synchronizers were developed. Event synchronizers are transductions that map event traces to new event traces. For example, event or as presented in Figure 2.2 is an event synchronizer that generates an event if and only if no two events happen at the same time. Other event synchronizers that can be modeled in PCN include event and, eventfilter,event select and any event logic elements described in [Sut89a], such as Switch, Event-Controlled Storage Element (ECSE), Toggle, Arbiter. Let us now briefly review the types of uncertainty arising in physical systems, and thus the types that an efficient framework should be able to model. We will then proceed to a discussion of the types of computation that are possible within the PCN framework.  4.2  Types of Uncertainties  When choosing a model for an uncertain system, it is important to capture the essential features of the real system and the uncertainties in that system so that the trajectories of the model mimic the behavior of the real system. There are many different types of models for uncertainty and the model to be used depends not only on the type of uncertainty expected within the system of interest but also depends upon what kind of analysis one wishes to perform on the model. Indeed, a particular uncertainty model might reflect the physical system's uncertainty very closely but might also be too complex to analyze, thus rendering the model useless. For this reason, it is often necessary to enlarge the class of uncertainties to ensure that we get a tractable model. For example, in the problem of optimal control, one might want to ensure that the model of the uncertainty is tractable so that methods for control synthesis can be applied. Unfortunately, this can lead to conservative control system designs. In this thesis, we will focus mainly on modeling uncertainty via stochastic processes such as Markov chain for discrete time systems and Wiener processes for continuous systems. Note that stochastic modeling of uncertainty has been widely studied in the recent years and many alternative models have been suggested. For example, the notion of stochastic uncertain systems which relies on integral quadratic constraints has been developed to handle various types of uncertainty and is especially suited for analysis related to stochastic stability and robust control of systems [PUSOO,  90  Figure 4.2: A sequential module Ugr98].  4.3  Computation in Probabilistic Constraint Nets  No computational model is suitable for every type of computation. A given model is developed to handle a certain type of computation and (hopefully) provides advantages over other models for this type of computation. For example, analog circuits are used to represent parallel and continuous computations while Turing machines and Markov chains are used for sequential computations, the latter when in the presence of uncertainty. The PCN framework is no different in that it is inherently designed for parallel computation of uncertain dynamical systems with its main advantage over existing models being its abstraction of time and domain. However, PCN also allows for sequential computation to be modeled. In this section, we will introduce the reader to the notion of sequential computation by means of the PCN framework and will conclude with an introduction to continuous computation in PCN.  4.3.1  Sequential and Analog computation  It is possible, in PCN, to represent sequential computation using events to coordinate the order of computation. In fact, just as for the C N framework, we model sequential computation as a module with an event input indicating the start of a computation and an event output indicating the end of the computation (see Figure 4.2 taken from [Zha94]). The time duration between the start and the end of the computation is variable, depending on the input data. We call such a module a sequential module.  1  ' A basic transliteration / , described as a pointwise extension of a function in previous chapters, can be seen as a a sequential module with End = Start and DatajDut = f(Data_In), i.e., computations happen instantly within a transliteration.  91  An important result is that given a set of basic functions and their sequential modules, the set of functions closed under functional composition, recursive schemes and minimization operations can be computed by sequential modules. In fact, this set is large enough to include all the computable functions given a small set of basic functions [Zha94]. However, many functions that are not easy to model in sequential computation are easy to compute as traces which, conveniently, are at the core of the PCN framework. Example 4.1 As an example of modeling uncertain continuous time systems with a Brownian motion, consider the following function on continuous time structures: Aw, i . C e ^  _  5'  j 2  )  t +  '  t B w  (*).  This  function (or stochastic process) is the solution of a probabilistic constraint net x = f (C)(Xx) + f (C, B )(px) representing the geometric Brownian motion [Mao97] u  dX = XX dt + fiXtdWt. t  t  (4.2)  This SDE is of great importance in the field of economics andfinanceas the well-known BlackScholes partial differential equation can be derivedfromit. Let us now raise the following important question: Given a set of stochastic differential equations modeled in probabilistic constraint nets over a continuous time structure, what is the relationship between the semantics of the probabilistic constraint nets and the solutions of the stochastic differential equations? This question is equivalent to asking if the PCN has a well-defined semantics, and if the set of stochastic differential equations has a unique solution. With a positive answer to both of these questions, we have that a trace, obtained as a solution of a set of stochastic differential equations, can be computed as the limiting semantics of the PCN representing the set of SDEs. 4.3.2  Stochastic Taylor expansion  Let us now present an introduction to stochastic Taylor expansions. Stochastic Taylor expansions will be at the core of the limiting semantics for PCN on continuous time structure. Similarly to the case of ordinary differential equations and Taylor expansions, stochastic Taylor expansions are at the core of the numerical integration schemes for stochastic differential equations. First, let us define  92  as the (i-dimensional gradient in the x-direction and C (U, V) as the family of continuous functions k  from U to V with continuous derivatives up to order k. We also need to introduce the famous Ito formula [It651 ] - m d - E E 9t(t, )gl(t, )  ci  £0  <  +  m  x  )  ,y  > ,  x  +  X  02  —  X  j = l i,k=l  K  and C? =< gi(t, x), V >d where j = 1,2,..., m. x  Moreover, before formally introducing the notion of stochastic Taylor expansion to define the limiting semantics of a probabilistic constraint net of stochastic differential equations, we have to formulate what is meant by multiple indices, hierarchical sets, remainder sets, coefficient functions and multiple integrals in the Ito sense. This will allow us to present the results in a clearer and more compact fashion. Definition 4.1 A multiple index has the form a = (a 1,0:2, • • • •«;(«)) where 1(a) £ N is called the length of the multiple index a, and n(a) is the total number of zero entries of a. The symbol v denotes the empty multiple index with l(v) — 0. The operations a— = (a\,... ,a;( )-i) and a  —a = («2) • • • i i(a)) a  a  r  e  called right- and left-subtraction, respectively (in particular, ( « i ) - =  — (a\) = v). The set of all multiple indices is defined to be: M,  k m  = {« = (cti,a ,.. 2  • , a ) ) : a £ {k, k + 1,... ,m},« = 1, 2 , . . . ,l(a), with 1(a) £ N} . i(a  t  A hierarchical set Q c M.o, is any multiple indices a £ A^o.m such that v £ Q and a £ Q m  implies —a £ Q. The hierarchical set Qk denotes the set of all multiple indices a £ Mo m with t  length smaller than k £ N, i.e., Qk = [a £ M o , : K ) ^ a  m  The set R(Q) = {a £ Mo  Q '• OL— £ Q} is called the remainder set R(Q) of the hierarchical  iTn  set Q. A multiple Ito integral I ,s,t[V(-, •)} is defined to be a  W [ " ( v ) l  = < J  -  '  —  W  ^  '  J* V(u, X )dWu u  Mm 0l  >  1  (4.3,  otherwise  (a)  for afixeda £  W  {"} and for a given process V(t,X ) where V £ C°'°([0,T] x W  A multiple Ito coefficient V  t  a  £ C°'°([0,T] x  R ,R )  93  d  k  for a given mapping V = V(t,x) £  l(a)M«)f[o T] x M. , R ) is defined to be d  c  k  f  f C ^V _(t,x)  if 1(a) > 0  l  a  V (t,x)=\ a  (4.4)  otherwise  I V(t,x)  Now that the notational background has been introduced, we are able to state a general form of the Ito-Taylor expansion. Stochastic Taylor expansions for Ito processes have been introduced and studied originally by Wagner and Platen [WP78]. An Ito-Taylor expansion for the standard Ito SDE 2  3  m  dX =  f(t, X )dt  t  where / , g : [0, T] x R j  d  +  t  £9  j  (t,  X )dWi, t  -> R are the drift and the diffusion parts, and where { W / : 0 <  t < T} represent m mutually independent Wiener processes on the complete probability space (fl, T, (Ft)te[a,T\, P), is of the form:  V(t,X ) = t  ^ri [V (s,X )}+ aiStt  a  I*, ,t[V (;-)]  s  a  a€Q  a  (4.5)  a€R(Q)  for a given mapping V = V(t, x) : [0, T] x R —• R which is smooth enough. For completeness, d  k  we restate Theorem 5.1 of [KP99].  Theorem 4.1 Wagner-Platen Expansion Let p and r be two Tt-adapted stopping times with to<p<T<T<oo  (a.s.). Assume V :  [0, T] x R -> R . Take any hierarchical set Q G M , - Then, each ltd SDE with coefficients f, g d  k  j  0  m  possesses a Ito-Taylor expansion of Equation 4.5 with respect to the hierarchical set Q, provide that all derivatives ofV, f, gi (related to Q) exist. Based on these notions we can present the following result.  Proposition 4.1 Given a probabilistic constraint net made of the stochastic differential equations Xk = fk(x) +  gk(x)N , k = 1,..., n with x (t ) t  functions, and given that all fk and 2 3  k  g^  0  e R and  fk, gk  : R " —> R as partial or total  are sufficiently smooth at  This expansion is also called Wagner-Platen expansion for its creators. Similar results hold for Stratonovich SDEs  94  x(to),  the limiting semantics of  the probabilistic constraint net, based on the Euler-Maruyama method, is well-defined over T = [to,ti]for some t\ > to. In particular, the results of Theorem 4.1 apply for the smooth function V(t, x) = x. The reader should note that if fk and g are polynomial functions, then both fk and k  gk are smooth over R and hence this results holds. n  Example  4.2 Recall the Geometric Brownian motion denoted by Equation 4.2, and introduced  briefly in Example 4.1. Let us calculate the stochastic Taylor expansion of this process. Applying the Ito-Taylor expansion of Equation 4.5 to the SDE 4.2 we get  x  t  (I + E  =x  t0  = X  A V - /a,t , ) n(Q  ( l + X(t - to) + p(W - W ) + E a g ^ A K W . d ) }  t0  _  A - C V ^ W )  A E A V N W  t  ^  [A-^(i-t )]  — -*-t 2-,i,j=0 Y  +  0  O  0  i\  0  i  n(a)  J  j\  0  4  ( 4  |MWt-W« )l ' n  = X exp ((A - £ )(i - t ) + p(W to  (a)  0  t0  '  6 )  W^))  t  where the coefficient functions are V(t, x) = x, V (t, x) — X ^ ^( )- ( ) n  a  n  a  a  x  with n(a) as the total  number of zeros of a G MQ,\, V as the empty index, and where I ,s,t without the argument [•] is a  understood to be I ,s,t\f\-  a  a  So far, we have provided a meaning for the rightmost integral of Equation 2.2, but did not mention whether we can obtain existence and uniqueness theorems for such equations. Moreover, one might be interested in knowing what kind of properties these solutions have, if any. We reproduce  here the well known existence and uniqueness theorem for stochastic differential equations (SDEs) from §5.2 of [Oks98]. For simplicity we present the one-dimensional case but the results can be generalized to more complex SDEs. We will omit the proof here and refer the interested reader to the original work. Theorem 4.2 Existence and uniqueness theorem for SDEs  LetT > 0 and b(-, •) : [0,T] x R  n  -> W , g(-, •) : [0,T] x R " -> M " 1  x m  be measurable  functions satisfying \b(t,x)\ + \g(t,x)\ < C ( l + |x|);  x G R , t G [0,T] n  (4.7)  for some constant C, and such that \b(t,x)-b(t,y)\ + \g{t,x)-g(t,y)\<D\x-y\; 95  x, y G R", t G [0,T]  (4.8)  for some constant D. Let Z be a random variable which is independent of the o-algebra generated by B (-), s > 0 and such that E[\Z\ ] < oo. 2  s  Then, the stochastic differential equation dX = b(t, X )dt + g(t, X )dB t  t  t  0<t<T,X  u  0  = Z  (4.9)  has a unique t-continuous solution Xt(u)) with the property that Xt(to) is adapted to the filtration J-1 generated by Z and B (-); s < t and s  (4.10) Jo where Equation 4.7 is a linear-polynomial boundness condition on f and g while Equation 4.8 is the well-known Lipschitz condition. The Lipschitz Condition 4.8 guarantees that / and g do not change faster with change in x than does the function x itself. This implies in particular the continuity of f(t, •) and g(t, •) for all t e [to,T\. It is easy to show that if / and g are linear functions, i.e., f[x) = Ax, then both the Lipschitz and linear-polynomial boundness conditions are satisfied. Hence, linear SDEs always have a unique solution. Another important question is whether the solution X of Equation 4.9 is a stationary Markov t  process. A necessary and sufficient condition for stationarity is that Xt must be homogeneous along with a series of more complex analytical conditions. We will omit the details here but the interested reader should consult [Kha69], p.97 for more details. In this case, the limiting semantics lead to a stochastic process with a stationary distribution for all t e [to, T]. To summarize, in this chapter we have demonstrated the modeling power of PCN by showing it can model sequential as well as analog computation. Moreover, we introduced the notion of event generators and event synchronizers, central concepts that allow us to handle system with multiple clocks. Finally, we provide a formal limiting semantics for analog computation based on the well known results of SDEs and stochastic Taylor expansions.  96  Chapter 5  Models Subsumed by PCN In this chapter we look at the various probabilistic modeling frameworks that are special cases of the PCN framework. Table 4.1 of Chapter 4 highlights the classification of the models based on the time and domain structure. For each of those four cases, we will show how these commonly used models can be equivalently represented by the PCN framework, hence demonstrating the flexibility of our framework.  5.1  Discrete time and discrete domain systems  The simplest case of a PCN model is one with discrete time structure along with a discrete domain structure: henceforth referred to as a DD-PCN. Note that a PCN model, even if restricted to these discrete-time/discrete-domain constraints, is still powerful and allows the modeling of a large class of systems. Discrete asynchronosity (multiple discrete clocks), as well as deterministic, non-deterministic or probabilistic behaviors are all properties that can be modeled within the DDPCN framework. Furthermore, we will show that widely used models such as Markov Chains and Markov Decision Processes (in a later chapter) are in fact instantiations of the DD-PCN class of models.  5.1.1  DTMC to PCN conversion  The most commonly used discrete time/domain model for stochastic systems is the Discrete Time Markov Chains model (DTMC). Showing that every possible DTMC can be represented as a DDPCN model is a trivial exercise. Indeed, given a discrete time Markov chain Jvi represented as 97  6(S ) 0  Figure 5.1: Representation of a DTMC as a P C N the tuple (S,so,V), where <S, So  ana  " V represent the finite set of states (assume \S\ = n), the  initial state and the probability transition respectively, the equivalent DD-PCN M  pcn  is simply  ({5}, 6(So), V , C ). As it can be seen, the set of locations contains one and only one location, vcn  pcn  S, with domain {1,2,... ,n}, where each value of the location represents the encoding of each state in the DTMC. M n has only one deterministic transduction, a unit delay S(So), and one generator pC  following the probability distribution V. The unit delay S(So) is not only essential in avoiding an algebraic loop but it also allows us to model the Markovian property of the Markov chain within M cn- The unit delay guarantees that the next state of the DD-PCN can only depend on the value P  of location S (state S depends only on state S -i). t  t  Note that the P C N framework also allows  the modeling of Markov chains of Markovian order greater than one by simply using longer time delays: S ,n> 1, n G N. Furthermore, observe that the initial state So is only used to set the value n  of S in the unit delay <5(S ) of the DD-PCN 0  The only generator in M  pcn  M . pcn  is equivalent to the probability transition matrix of the Markov  chain M. That is, given the state value St, the generator represents a probability distribution of the next possible state S +\. t  The set C  pcn  contains three connections: 1) connecting V  pcn  to its output location S; 2) con-  necting location S to the unit delay 5(SQ); and 3) connecting the output of the unit delay to the generator V . Figure 5.1 shows the graphical representation of a DTMC as a DD-PCN. pcn  5.1.2  DD-PCN equivalence to DTMC  We can show that any DD-PCN with the following characteristics can be converted into a DTMC: 98  1. all locations have finite domains 2. all delays are bounded 3. all transductions have finite memory The transformation of a DD-PCN into a DTMC simply consists of obtaining a state space from the set of locations of the DD-PCN (along with their respective domains) and generating a set of transition probabilities from the set of generators of the DD-PCN. We can show that a naive approach to transforming a DD-PCN into a DTMC will most likely yield a state space much larger than in fact is needed. An unnecessarily large state space would result in a transition probability matrix which is extremely sparse. We will present the intuition behind reducing the state space to a smaller size, while still keeping the correspondence between the DD-PCN and the DTMC. Within the DD-PCN class, each model is represented by a finite set of locations Lc (each location with a finite domain), a finite set of deterministic transductions Td, a finite set of generators Q, and a set of connections Cn, linking locations with transductions and generators. To keep things interesting, we will assume that the set of generators is non-empty. Indeed, in the absence of probabilities, the system is purely deterministic and the current state of the system fully determines its next state, hence removing the need for a probabilistic analysis altogether. Formally, in order to see if a DD-PCN is equivalent to a DTMC, one needs to obtain, from the components of the DD-PCN model, a state space S, and a probability transition matrix which amounts to the transition probability per state s: P(s, s'). The initial state of the DTMC will be dictated by the initial values of the delays within the DD-PCN. The naive approach to generating the state space is to make it the cross-product of the domains of all locations. For instance, given a simple DD-PCN XYZ and with respective domains D  x  — {0,1,2}, D  y  with three locations X, Y and Z,  = {1,2}, D  z  = {1,2,3,4}, one would get a  state space consisting of 24 states, each state equivalent to an element in D  x  x D x D . The first y  z  requirement on the structure of the DD-PCN, the one stipulating that the domain of each location be finite is essential to guarantee that the resulting cross-product (state space) is in fact finite. Once one has a state space to work with, the only thing missing is the transition probability matrix. Since we are given the probabilistic transductions for each location, it becomes easy to extract 99  P(X=0) = 0.15 P(X=1) = 0.60  1 y 1 —v) —  VJ.T  P(Y=2) = 0.6  Figure 5.2: Simple DD-PCN with three locations the transition probability for the induced state space. In order to maintain Markovian dynamics, we will assume that the DD-PCN of interest only possesses unit delays and transliterations (primitive transductions without memory nor internal state). This assumption is fairly restrictive and will soon be relaxed (while retaining the Markov property), once the intuition behind the concept of generating the transition matrix has been presented. By using the information provided by the generators of the DD-PCN, we can obtain the transition probabilities quite easily. Let us demonstrate this by revisiting the DD-PCN XYZ  introduced  above. This DD-PCN is shown in Figure 5.2. In this system, two random variables (X and Y) are summed and the result is called Z. This is a very simple system as neither probabilistic transduction depends on previous time steps, and it is made of only one deterministic transduction. Real probabilistic systems will often be composed of hundreds or even thousands of components, with the presence of unit and transport delays. However, the intuition gained from looking at the DD-PCN XYZ will apply to those as well. In order to obtain the transition probabilities, we first need the state space of the DTMC associated with this DD-PCN. As shown above, we simply compute the cross-product of the domains of locations X, Y and Z. This gives us a state space of 24 states . Now to compute the transition 1  probabilities, we need to take into account the fact that we are in the presence of a deterministic transduction, namely the Add transliteration, which simply returns the sum of its two inputs. This 'Notice that some states obtained from the cross-product operation are in fact impossible. For example, the state {X = 1, Y = 2, Z = 1} is not feasible. We will discuss this issue later  100  transduction has the effect that it renders location Z fully dependent on locations X and Y. Indeed, given the values of X and Y, the value of location Z is fully determined (with probability 1). It becomes obvious that the probability that a state {x, y, z} will transition to another state {x , y', z'} ;  is equivalent to the probability that at the next time step, X = x' and Y = y', multiplied by the indicator function I (z' z  = x' + y') (which yields 1 for z' = x' + y' and 0 otherwise). Hence  the transition probability is simply the product of I with specific probabilities obtained from the z  probabilistic transductions of XYZ. It should be noted that most PCN models contain both types of transductions (probabilistic and deterministic). An interesting situation arises in the presence of deterministic transductions. In this case, there is full dependence between two locations which makes some transitions impossible. For example, within the DD-PCN XYZ,  we already mentioned that location Z is dependent on  both locations X and Y. Furthermore, as briefly discussed above, this dependence creates some constraints on the transitions of the systems. However, for this example as is the case for most DD-PCN models, we have more than one choice for a possible state space: in fact, for this example, we have three distinct choices. One could consider a state space consisting of all three locations (as presented above). One can also consider X and Y as defining the state space or alternatively consider only location Z. Although the first choice appears to be the intuitive one, as it provides an easy way of generating a state space, it comes with the price of having to handle impossible transitions. However, depending on the interests of the system designer, it might be valuable to only consider a subset of locations as being worthy of defining the state space. Assume for example that for the XYZ  system, the designer is in fact only interested in the value of location Z, and knowing the  specific values of the two random variables X and Y are irrelevant to her.  2  In this situation, it  would make sense for her to build a state space based only on location Z, hence reducing the state space from 24 states to merely 4 states, without any loss of information (at least what the designer considers relevant information). The transition probabilities for this new state space are easily obtained from the product of transition probabilities of locations X and Y, under the assumption This is only the case if location X and Y are both independent of any previous values. In the case where transduction P would have a delayed value of X as input, then the state space would require X to be present, as the transition probability for Z would be conditioned on the value of X. Hence not all locations can be omitted from the state space, even if they are not considered important by the system's designer. 2  x  101  Observed Variables  Figure 5.3: Bayesian Network representation of the temporal dependence that X and Y are independent at time t. In our example, it is easy to see that locations X and Y are in fact independent since none of the transductions depend on past values. However, even if both probabilistic transductions depended on the values of X and Y at the previous time step, X and Y would still be independent at time t, given their values at time t — 1. To show this, we can simply represent the temporal behaviour of the DD-PCN as a (Dynamic) Bayesian Network (Figure 5.3) and see that since the values of Xt-i and Yt-\ have been observed (evidently since we are now at time t), then the values of X and Y must be independent. Hence, to t  t  obtain the distribution of Z, we can proceed using simple probabilistic rules. So far, we have assumed that DD-PCNs are only composed of unit delays and transliterations. This was done in order to ensure that the transition probabilities possess the Markov property. However, it is possible to maintain the Markov property while greatly relaxing the assumptions on the components of the DD-PCN model. This relaxation is specified by the second and third characteristics of DD-PCN specified at the beginning of this sub-section. Obviously, in the presence of unbounded delays, the Markov property would be totally lost, since the system would depend on an unbounded number of previous time steps. The same reasoning holds for transductions with infinite memory as this would assume that the transitions of the  102  p(x=01 x=0) = 0.15 p(x=0 I x= 1) = 0.48 p(x=l I x=0) = 0.60 p(x=l I x=l) = 0.10 p(x=2 I x=0) = 0.25 p(x=2lx=l) = 0.42  p(y=l |y=0) = 0.4 p(y=l I y=l) = 0.77 p(y=2 I y=0) = 0.6 p(y=2 I y= 1) = 0.23  Figure 5.4: Non-Unit delays for the X Y Z DD-PCN system not only depend on the current state but also on the whole history up to the current time. One may wonder why we only restrict the DD-PCN to bounded delays and finite memory transductions. Intuitively, a delay of 2 time steps appears not to respect the Markov property: the system depends on the current state but also on the previous state, and similarly with transductions that are not solely transliterations. However, there is a way around that problem. By including part of the history in the state space, one can convert a seemingly non-Markovian DD-PCN into a Markovian one. For instance, going back to our previous example of the DD-PCN with three locations, let us assume this time that the generators driving X and Y are dependent on X -2 and Y -i respectively. t  t  This system, although very similar to the initial XYZ system, is represented in Figure 5.4 for sake of clarity. This new situation implies the presence of delays of three units for X and two units for Y. By using a state space obtained from the cross-product of D xD xD xD xD , x  x  x  y  y  containing information for X at three different times {X , X -\, X - } t  t  t  2  one now gets a state and for Y at two different  times {Yt, V j - i } . The probability of transition from a given state St = {Xt, Xt-i,X -2, t  Y , Yt-i} t  to another state St+i = {Xt+i, X , Xt-i,Yt+i,Yt} is simply the product of the two probabilities t  P (Xt+i\X -2), x  t  Py(Y i\Yt-\, which can easily be obtained from the generators for X and Y and t+  a set of indicator functions ensure that Xt is now equal to Xt-\ once the transition has occurred.  103  Figure 5.5: Representation of a DTMP as a PCN  5.2 5.2.1  Discrete time and continuous domain systems DTMP to PCN conversion  Similarly to a DTMC, the PCN equivalent to a Discrete Time Markov Process (DTMP) is very 3  simple. The only difference if that instead of a probability transition matrix, we have a probability measure over sets of states. The representation of such a DTMP is shown in Figure 5.5 where the domain of location S is continuous rather than discrete as it was the case for the DTMC conversion. Notice that the generator is now defined on a set of states A, given the present state s', which demonstrates the fact that the location's domain is continuous.  5.2.2  DC-PCN equivalence to DTMP  The class of discrete time, continuous domain PCNs, called DC-PCN, can be shown, in a similar way as with DD-PCN, to be equivalent to the class of discrete time Markov processes. We omit the details here as the extension to continuous domains, although more complex due to the presence of measures and dense domains, is straightforward. The term Markov chains is used for discrete domains while Markov processes is the accepted terminology for continuous state spaces 3  104  5.3  Continuous time and discrete domain systems  In this section we will discuss the relationship between our framework and the paradigm most commonly used to model continuous time systems over discrete domains: Continuous Time Markov Chains (CTMC). However, first, let us discuss the meaning of probabilities when in the presence of a continuous time structure. For continuous domains where the uncertainty is modeled as a stochastic process such as a Gaussian process or Brownian motion, the combination of the time continuum and probabilities is meaningful and easily understood since uncertainty evolves continuously with time. However, if one is studying systems with non-continuous domain structures, then one needs to clearly understand the meaning of probabilistic transitions. What does it mean for a system to evolve uncertainly, continuously over time, over a discrete set of values? Some desired properties of such probabilistic systems include: 1. The numbers of transitions in non-overlapping time intervals should be independent for all intervals. The occurrence of a specific transition should in fact be independent of when the previous one occurred. 2. The probability of two or more transitions occurring in a sufficiently small interval h should be o(h). A way to handle such systems while guaranteeing that the above properties are satisfied is to have event-driven transductions, where events trigger a probabilistic transduction, i.e., a transduction containing a generator. Formally, we can show that the event generator should be modeled as a Poisson process, with the event being the arrival time. Recall that we introduced the Poisson process event generator in § 4.1.1 as represented by Equation 4.1. For example, consider the simple probabilistic system XYZ  of Figure 5.2, but now with a  continuous time structure. The probabilistic transductions for X and Y are the same as before. However, since time is continuous, we have to ensure that we have a finite number of transitions within any finite interval [ i i , ^ ] - Without this constraint, the system would be ill-behaved since locations would continuously transition from one value to another. We therefore model the transition times for each probabilistic transduction X and Y, as a Poisson process. Note that both X and Y can have different event generators, which in turn means that 105  only a subset of the variables would in fact be transitioning at one given time. Note also that in general the event generator can be dependent on locations of the model. This allows for the rate of transition to vary according to a subset of the variables constituting the state space of the system. Hence a system can stay longer on average in one state while it might stay for a very short period of time in another state. A PCN of such a probabilistic model can be shown to be equivalent to a CTMC. Let us first define formally the notion of CTMCs, and then present how to obtain them from a PCN model and vice-versa. Formally, a continuous time Markov chain is a tuple M = {S, SQ, R) where <S is a finite set of states, so is the initial state, and R : S x <S —> R  +  is known as the rate or intensity matrix.  Note that this general definition does not preclude the system from having self-loops. Self-loops are represented by R(s, s) > 0 and denote the fact that a transition can occur while the system remains in the same state. The inclusion of self-loops in the model might appear to diverge from the classical definition of C T M C . However, while they do not impair the expressibility of the paradigm, self4  loops allow the usual interpretation of linear-time operators from temporal logic. We will expand on this notion in a later chapter when we introduce the probabilistic verification of PCN models. Moreover, for simplicity, we will assume that the initial state of the system, so, is known with probability one. This restriction can easily be lifted and the following discussion generalizes to the case of a distribution over the initial state. Intuitively, there exists a possible transition from state s to s ' of M. iff we have R(s, s') > 0. The delay in transition between two states s and s ' is governed by an exponential distribution with rate R(s, s'). Hence, the probability that a transition s —> s ' happens within t time units is denoted by the expression 1 — -^(s.«')-*. When the system can transition from a state s to multiple states e  with different rates, i.e., R(s, s') > 0 for more than one s' e S, a race condition ensues and the race is won by the minimum delay among the exponential distributions. Hence, E(s) —  J2s'&S ^ ( i ' ) s  s  denotes the total rate at which any transition can occur at state s and leads to the conclusion that the probability of leaving state s within t time units is 1 — e~ ^ . With this, we can infer that the E  4  probability of moving from state s to another state s' is determined by the probability that the delay in evolving from s to s' is completed before the delays of the other possible transitions. Therefore, we obtain P(s, s') = R(s, s')/E(s). Although many other bodies of work allow the modeling of self-loops in their definition of CTMC, among other, [BDH99] 4  106  Figure 5.6: PCN equivalent of a CTMC  5.3.1  CMTC to PCN conversion  Similarly to the previously discussed Markovian models, the conversion of a CTMC to a PCN is very straightforward. Given an n state CTMC M =  (S,SQ,R),  we can obtain an equivalent  PCN Mpcn = ({S}, {Rate, P}, C) where S is the sole location of the system with domain(S') = {srj,si, • • • ,s -i} encoding the n states of the system; Rate is the event generator following an n  exponential distribution with state dependent rate E(s) that triggers an event when the race condition has been completed; P is the generator following the distribution P(s, s') — R(s, s')/E(s) for all s in domain(S) which causes the system to transition probabilistically to a new state s' when: 1) the system is in state s, and 2) an event signifying the completion of the race condition has occurred. This general situation, which applies to any C T M C with discrete state space, is represented in Figure 5.6.  5.3.2  PCN to CTMC  Let us now look at what class of PCN can in fact be mapped into a CTMC. This conversion is more complex, since for a general PCN, each location (one variable out of the set of variables constituting the state space of the system) can have a different rate of delay. When there is more than just one exponentially distributed event generator, only a subset of the locations can transition at one given time, hence causing a localized transition of the system. To keep things as general as possible, let us assume for the remainder of this section that each PCN location is obeying its own  107  exponentially distributed event generator, resulting in transitions in only one location for each event. Indeed, since time is continuous, we will assume that there cannot be multiple events arising from the exponentially distributed generators of the model. This assumption will become important when we combine the rate matrices of all the locations of the system. The case where multiple locations are related to the same event generator can easily be dealt with based on similar reasoning. Without loss of generality, we can assume that, when building the state space of the equivalent CTMC, we only consider the locations that are outputs from a generator (i.e., locations equivalent to random variables). Any location which is an output of a deterministic transduction can be ignored as its value is merely a deterministic relabeling of the input values and hence does not affect the state space if ignored. Moreover, we assume that each generator gi G Q is event-driven and the event generators follow a Poisson process with an exponential distributions with rate A j . Assume that we are given a general PCN model of the form M  = (Lc, Td, Cn) where Lc  PCN  is a finite set of output locations, Td is a finite set of transduction labels (with Q C Td being a finite set of generators) and Cn is the set of connections between transductions, generators and locations. As mentioned above, we will only consider the set L G Lc of locations that are outputs g  of generators and we assume that \L \ = n. Therefore, it is easy to show that the state space of g  the equivalent CTMC M, denoted by SM, will consist of the cross product of all the locations in L , i.e., XiL ,i g  9i  = 1,• • • ,n. We then need to show that there exist a rate matrix R(-, •), which is  equivalent to the dynamics of the PCN model. Based on these assumptions on M. , VCN  we have each location of interest modeled in a way  similar to the single location of the PCN shown in Figure 5.6. We can thus extract, for each location L  gi  G L ,i g  = 1, • • • , n, the following information:  1. The event generator Rate(L ) provides the conditional total rate at which any outgoing tranSj  sition from location L  gi  is taken. Hence, in the CTMC framework, we have that Rate(L ) = 9i  E(I(RaXe(L )), Rate(L )) is the total transition rate for the L gi  9i  9i  component of the state space  SM, where /(•) denotes the inputs of the transduction or generator. 2. From the distribution of the generator P L . , one can obtain the probability of moving from 9  the current state of the system to another value (possibly the same value since self-loops are allowed by design),  108  Therefore, knowing the relationship between the total rate E(-), the rate matrix R(-,-) and the probability of transition P(-,cdot), namely P(s,s') = R(s,s')/E(s), we can easily obtain the conditional rate matrix, R(-, •), for each L  gi  e L. g  To complete the conversion, we need to combine the individual conditional rate matrices to obtain a rate matrix that would allow us to see the system as a single process. The method used to calculate the global rates is similar in essence to the amalgamation operator developed in the context of continuous time Bayesian Networks [NSK02], with the exception of a small modification needed to handle self-loops. To obtain the whole system's rate matrix, we first note that any transition which involves a change in more than one location will have an intensity of zero as no two locations can transition simultaneously. Then, similarly to the amalgamation operator, for a non-diagonal element of the rate matrix, one simply uses the intensity of the rate matrix for the associated location. Diagonal elements, which are equivalent to a self-loop need to be treated separately. Since any single transition from a state value to that same state value causes a self-loop, the transition rate corresponds to the minimum delay in which the locations constituting the state space will transition. Due to the fact that the transition delays are exponentially distributed, the minimum delay is hence obtained by summing all the rates for self-loops in each location. In this manner, we obtain the rate matrix for the system as a whole. Since we already built the state space as the cross-product of the domains of the PCN locations, this completes the conversion of a PCN into a standard CTMC.  5.4  Continuous time, continuous domain  In the previous chapter we discussed the notion of analog computation within PCN. Analog computation is modeled via stochastic differential equations (SDEs) which is a general model for continuous time and continuous domain systems. The general class of SDEs includes Markov processes and diffusion processes which, as mentioned earlier, can be modeled within the PCN framework. The wide area of SDE has been extensively studied over the years. For more details on the field, the reader is referred to the following books on SDEs and their applications [Oks98, Arn74, Mao97].  109  5.5  Other models of interest  Markov Decision Processes (MDPs) and their noisy observation counterpart Partially Observable Markov Decision Processes (POMDPs) are models that have received growing attention recently, especially in the field of artifical intelligence [Put94, BDH99]. Similarly to the other Markovian models presented in this chapter, these paradigms are also special cases of the PCN framework. However, since our main interest in MDPs and POMDPs lies in their use in of obtaining optimal policies from the models, we will delay their introduction until the chapter on control synthesis of PCN models.  110  Chapter 6  Introduction to Behavioural Verification The online satisfaction of the constraints imposed on the dynamics of the system by the model ensures that the system's behaviour will behave according to the stochastic process that corresponds to its solution. However, such constraint satisfaction does not guarantee that the behaviour of the system will satisfy global behavioural constraints. For example, consider the dynamics of an elevator system as described in Appendix B. Equation B . l represents the constraints on the dynamics, essentially stating that the elevator must move obeying Newtonian laws of motion. Although this equation can represent very accurately the local behaviour of the system, it does not preclude the elevator from stopping halfway between two floors nor does it guarantee that a service request will always be successful in a timely manner. None of these requirements are explicitly specified in the model of the dynamics. In fact, such restrictions are global constraints on the behaviour of the system and cannot easily be represented within the PCN modeling framework. However, such global behavioural constraints are absolutely necessary when designing a dynamical system that will be used in practice. Hence, we need to define an appropriate requirements specification language that would allow the designer to specify global behavioural constraints. Moreover, we need to develop a formal method to verify the behaviour of a system with respect to a given requirements specification, thus guaranteeing that the system will in fact behave as it is designed to. In Chapter 7 and 8 respectively, we develop average-timed \'-automata, and the Probabilistic Arbitrary-time Timed Temporal Logic (PATTL), two requirements specification languages for which  111  we provide formal verification methods.  6.1  Average-Timed V-Automaton  A popular method for representing behavioural constraints of systems is automata. This method is also well suited for the PCN framework as we can view traces as a generalization of infinite sequences. A desired property of the systems (hence the traces) can be specified by an automaton. That is, a trace of a system would satisfy the behavioural constraints iff the associated automaton accepts the trace. Manna and Pnueli [MP87] first proposed V-automata and applied it to the specification and verification of concurrent programs. An extension to V-automata, timed V-automata, was proposed in [Zha94] and applied in the context of behaviour verification of dynamical hybrid systems. In the next chapter, we begin by briefly introducing the notion of V-automata, adopted from the definition given in [MP87] and of timed-V-automata as introduced by Zhang and Mackworth [ZM96]. Second, we extend discrete timed-V-automata to discrete average-timed V-automata, by augmenting the automaton states with average time bounds. With this addition, we can reason about the average temporal behaviour of systems. Finally, we generalize our definitions to accept traces of arbitrary time structures.  6.2  Our Probabilistic Temporal Logic: PATTL  An alternative to automata for specifying behavioural constraints is temporal logic. Probabilistic logics provide a simple yet powerful specification methodology for (dynamical) systems with temporal behaviour. We develop the Probabilistic Arbitrary-time Timed Logic (PATTL) where Timed refers to the notion of temporal evolution of the systems and Arbitrary-time denotes the ability to handle systems with arbitrary clocks. This logic is intended to specify quantitative probabilistic behavioural constraints applied to a PCN model of a system. For example, consider a coffee-delivery robot which upon request from his master, must fetch and deliver coffee. The robot's master might want to reason about the overall quality level of his robotic butler. This level could be defined by the waiting time to receive a cup of coffee once the robot has been summoned. Furthermore, the master might be interested in maintaining a certain basic quality level at all times but might agree, as a cost  112  saving strategy, to accept that the probability of the quality level dropping below the basic level for a short period of time be less than 10%. In industrial applications, this requirement is often referred to as quality of service and is a very important requirement for many applications. It is essential to be equipped with a specification language that is powerful enough to allow us to explicitly state such requirements but also offers computational efficiency so that one can verify that the desired requirements for real-world practical applications are indeed satisfied. The PATTL logic is based on a well-known logic and extends an already existing logic to account for arbitrary time and domains. Specifically, PATTL is a probabilistic temporal logic arising from the Computational Tree Logic (CTL) [Eme90, EC82], a branching time logic, on which some of the most used probabilistic logics are based. While C T L is not expressive enough to specify quantitative properties of uncertainty, it has a natural correspondence with the computation of concurrent and non-deterministic programs. The fact that CTL is a branching time logic rather then a linear time logic makes it a good candidate when investigating probabilistic temporal logics. Indeed, the underlying structure of time being tree-like provides an ideal framework to see each branch as a different probabilistic (or non-deterministic) successor of the current node. In our probabilistic approach to systems, instead of seeing time as having a tree structure, we view the branching property of such logic as a representation of each possible event u> taken from the event space fl. We will elaborate on this notion later in subsequent chapters. However, for a more in depth (and classical) comparison of linear time logic and branching time logic, the reader is referred to [Eme90]. Recently, with the increasing number of practical applications believed to exhibit inherent uncertainty, the need to specify quantitative, rather than qualitative, probabilistic properties has lead to the development of full probabilistic logic, most of them directly extending the CTL framework. We will base PATTL on the Probabilistic Computational Tree Logic (PCTL) of [HJ94], which we generalize to obtain a logic for arbitrary time and domain structures to allow formulae to be interpreted over Probabilistic Constraint Nets.  6.3  Behavioural Verification  In the previous chapters, we presented the syntax and semantics of Probabilistic Constraint Nets, a framework for modeling probabilistic hybrid systems. We showed that the PCN framework models a large class of systems and subsumes some of the most commonly used modeling framework such 113  Coffee_Delivery  Figure 6.1: Robot Delivery V-Automaton Specification as Markov processes and (PO)MDR We argued that the PCN framework is a general and representationally very powerful modeling framework which can be of great use for a system modeler. In the following chapters, we introduce the notion of average-timed V-automata and extend the logics CTL and PCTL to define the PATTL temporal logic. Both average-timed V-automata and PATTL constitute specification languages which we will use to specify behavioural constraints on systems. At this time, the difference between system modeling and the specification of behavioural constraints might still be unclear. Although they might appear similar, these two notions are inherently very different. The modeling task focuses mainly on the dynamics of the systems and how different components interact together. Essentially, it imposes local constraints on the systems dynamics. On the other hand, the specifications of a system impose global constraints on its behaviours. For example, the dynamics of a mobile robot can be modeled by differential equations following basic laws of physics such as the relation between velocity and acceleration (v = at or s — \/2at ). 2  These laws represent the constraints on the dynamics. However, although these represent well the local behaviour of the system, it does not preclude the robot from hitting people as it is roaming around nor does it guarantee that a delivery robot will always be successful when attempting to deliver coffee to its "master's" office. Such restrictions are global constraints on the behaviours of the system and cannot easily be represented within the PCN modeling framework. They can, however, be compactly expressed with a PATTL or a V-automaton specification. For example, the coffee delivery requirement can be represented in PATTL as "•ODeliver_Coffee", meaning that the robot will always (•) eventually (O) be successful at delivering coffee. The V-automaton specification is represented in Figure 6.1.  114  Once one is equipped with a model of the dynamics of a system (via the PCN framework in our case although many other modeling languages have emerged over the years) and with a requirements specification of the global behaviour of the system (either in V-automaton or PATTL), a key question is to ask whether the behaviour of the system satisfies these requirements. This is called behavioural verification. In the next two chapters, we will present behavioural verification procedures when armed specifically with average-timed V-automaton and PATTL specifications.  115  Chapter 7  Behavioural Verification with Average Timed V-Automaton In this chapter, we augment the notion of V-automata behavioural verification to average-timed Vautomata for stochastic dynamical systems. We first discuss the relation between V-automaton and stochastic systems. The notion of behavioural verification in this context is not as straightforward as with deterministic systems. We will then provide a simple introduction to timed V-automaton for discrete time structure and proceed to augment it to average-timed V-automaton for a similar time structure by specifying a set of on average constraints on automata states. Finally we generalize discrete average-timed V-automaton to average-timed V-automaton whose time structure can be arbitrary.  7.1  V-Automata  V-automata are non-deterministic finite state automata which can be used to specify requirements for concurrent programs [MP87] or time traces from deterministic dynamical systems [Zha94]. We present the definitions surrounding V-automaton along with the classical notion of acceptance of traces (the definitions are reproduced from Section 10.1 of [Zha94]). We then carry on by defining the acceptance of traces induced by a stochastic dynamical system. Note that this is not meant to be a comprehensive survey of V-automata and their related notions. For more details the reader is referred to Chapter 10 of [Zha94].  116  Definition 7.1 (Syntax of V-automata) A V-automaton A is a quintuple (Q, R, S, e, c) where Q is  afiniteset of automaton states, RC Q is a set of recurrent states and S C Q is a set of stable states. With each q G Q, we associate a state proposition e(q), which characterizes the entry condition under which the automaton may start its activity in q. With each pair q,q' G Q, we associate a state proposition c(q, q'), which characterizes the transition condition under which the automaton may move from q to q'. R and S are the generalization of accepting states to the case of infinite inputs. We denote by B = Q — (R U S) the set of non-accepting (bad) states. Let T be a discrete time structure, A be a domain and v : T —> A be a trace. A run of A over v is a mapping r : T —> Q such that (1) v(0) \= e(r(0)); and (2) for all t > 0, v(t) \= c(r(pre(t)),r(t)). We call a V-automaton complete iff we have ' V e Q e(«) is valid 9  • For every q G Q, V<,' Q (<?> <?') valid. c  is  G  For the remainder of this chapter, we will only consider complete automata as any automaton can be transformed into a complete automaton by introducing an error state. When displaying incomplete V-automaton, we will assume that the error state is implicitly present, thus simplifying the representation. Obviously, any complete automaton guarantees that any discrete time trace has a run over it. If r is a run then let Inf(r) denote the set of automaton states which appears infinitely often in r. That is, Inf(r) = {g|Vt, 3to > t, r(t 0) = q}. If T has a greatest element to then we define q  Inf(r)  = {r(to)}. Therefore, Inf(r) can be seen as a generalization of the "final value" of a  system. Let A be a V-automaton. A run r of A is defined to be accepting iff it satisfies one of the two conditions:  „  1. Inf(r) f l f i ^ B , i.e., some of the states appearing infinitely many times in r belong to R, or 2. Inf(r) C S, i.e., all the states appearing infinitely many times in r belong to S.  117  Essentially, the notion of acceptance of traces states that in the long run, the system either always returns to the set of recurrent states R or it will remain forever within the stable set S. Systems which cannot guarantee this are deemed unsatisfactory for the requirements specified by the behavioural constraints. Based on this requirement, the semantics of V-automata follow: Definition 7.2 (Semantics of V-automata) A V-automaton A accepts a trace v, written v \= A, iff  all possible runs of A over v are accepting. One should note that these semantics differ in the way they handle non-determinism from the semantics of conventional automata, with which the reader might be more accustomed. A conventional automata C, which could, in this context, also be called a 3-automata, accepts a language if there exists at least one run over C which is accepting. However, in the context of behaviour verification, having at least one run satisfying the requirements is obviously not a strong enough statement as in the case of a safety requirement, this is generally not what we define as a safe system. For deterministic systems, which are defined completely by a single trace, it is meaningful to require the trace be accepted. However, when modeling a stochastic system, asking for all traces to be accepted (which we referred to as satisfying the requirements at level a — 1) might be too demanding. Indeed, there might be a very small probability that the system will move into a set of absorbing bad states, hence never satisfying the behavioural constraints. However, if this probability (which is equivalent to the measure of all sample traces leading to the absorbing bad states) is small enough so that these events rarely occur, one might be willing to accept the risk to work with a system which satisfy the requirements at a level a where f3 < a < 1 and (3 is the safety threshold. Although, for certain systems, we will be interested in satisfying some behavioural constraints at levels a < 1, we will postpone doing so until we discuss verification procedures with PATTL. The techniques for verifying V-automata specifications which will be presented shortly are better suited for level a = 1 verification. Therefore, we will limit ourselves to systems for which verification at level a = 1 is meaningful. At this point it might be helpful to motivate the notion of verification at level a = 1 of a stochastic dynamical systems. What type of restrictions on the system itself does this create? Intuitively, perfect satisfaction of a set of behavioural constraints amounts to the system not possessing any absorbing bad states. By absorbing we refer to the case where the system enters this bad state and  118  never leaves it. In practice, for a system to not possess any absorbing bad states requires that for any state of the stochastic dynamical system associated with a bad automaton state, there must exist a path with positive probability which leads to an accepting state (associated to either R or S). We will formally prove this result when introducing the verification rules later in this section. While requiring that systems do not have absorbing bad states may appear to be overly restrictive, we can apply a simple transformation to the state space of the system to remove such states. Indeed, for a large class of systems with absorbing bad states, these states corresponds to a situation where the robotic agent is down in one way or another. Hence, repair or restart would be needed to ensure that the system can continue operating. One could take this "repair" into account and modify the state space so that once the agent arrives to a absorbing bad state, a transition occurs with probability one which relocates the agent to "restart" state. This simple modification removes absorbing bad states and thus allows the verification method to be applied to a vast class of systems. For economy of space, we refer the reader to §10 of [Zha94] for a comprehensive introduction to behavioural verification with V-automata. Let us now present some common behavioural constraints of dynamical systems. Figure 7.1(a) represents a specification which accept the traces of a system which eventually will always satisfy the goal condition G. This is equivalent to ODG in temporal logic. Figure 7.1(b) is a safety constraint which states that an accepted system should never satisfy the unsafe condition B . An equivalent temporal logic representation would be  \3-iB.  Finally, Figure 7.1(c) is a bounded re-  sponse constraint. It states that whenever event E occurs, the response R will occur in bounded time. This requirement is slightly more complicated as indicated by the temporal logic equivalent: • (£?-> OiJ). Although V-automata are not equivalent in expressive power to temporal logic such as TLTL, these small examples demonstrate the simplicity and intuitiveness of V-automata for the specification of behavioural constraints.  7.1.1  Average-timed automata  Meaningful behavioural constraints often encompass temporal components. Consider Figure 7.1(c), where one might be interested in a system satisfying a bounded response specification where the time bound is a known finite constant. In order to represent timeliness of behavioural constraints,  119  (a)  (b)  (c)  Figure 7.1: V-Automaton Specifications (a) goal achievement (b) safety (c) bounded response timed V-automata were proposed [ZM96]. Timed V-automata augment basic V-automata with timed automaton states and time bounds. This approach, however, is ill-suited for stochastic dynamical systems. Since we are interested in solving behavioural constraints on stochastic systems, we cannot talk about satisfying a given time constraint in an absolute way but rather we need to reason about satisfying that time constraint on average. We mentioned earlier that in order for a system to be accepting by a V-automata specification, it needs not have any absorbing bad states. This is characterized by the limiting behaviour: Umt->ooPr(Xt £ S U R\XQ G B) — 1. Although this requirement is sufficient to guarantee that for any run r over a given trace, Inf(r) fl R ^ 0 or that Inf(r) C S, it does not guarantee that it will happen in a finite time for every trace. However, for systems without any absorbing bad states, we are assured that the average time will be bounded, as stated in the proposition below. We prove the result for finite state space, but the result can be extended to countable state space at the price of a slightly more complicated proof.  Proposition 7.1 Assume afinitestate space S, and assume that, for all the bad states B, there is a positive probability of moving toward an accepting state R or S, i.e., the set of bad states is irreducible. Define RUS)  as the time needed to reach state s' from state s. Then, E(££|6 G B,s £  <oo.  A logical extension of time constraints is average time constraints. The idea behind average time constraints is that although we cannot prove that a stochastic dynamical system can always satisfy some given time constraint, we can show that the average behaviour of the system does satisfy the constraints. This is similar to the well-known concept of sample paths and expected sample paths of stochastic analysis. For completeness of this discussion, we summarize the notion of timed-V-automata prior to introducing the definitions of average-timed V-automata. 120  Definition 7.3 (Syntax of timed V-automata) A timed V-automaton TA is a triple {A, T, r) where  A= (Q,R,S, e, c) is a V'-automaton, T C Q is a set o/timed automaton states and r : Tu{bad} —» K  +  U {00} ('5 o time function. It is easy to show that any V-automaton is equivalent to a special timed V-automaton with T = 0  and r(bad) = 00. Graphically, a T-state is denoted by a nonnegative real number indicating its time bound. The conventions for complete V-automata are adopted for timed V-automata. Let v : T —» A be a trace. We define a run r of TA over w has being a run of .4 over v; r is accepting for  iff  1. r is accepting for A and 2. r satisfies the time constraints. If I C T is an interval of T and g* : I —> Q is a segment of run r, i.e., q* — r\j, let p(q*) denote the measure of q*, i.e., p(q*) = /:*(/) = T,t^ip(t) since 7 is discrete. Furthermore, let PB{q*) denote the measure of bad automaton states in q*, i.e., PB(Q*)  =  S / .( ) BM(*)- L te  jg  t  G  e t  Sg(q) be the set of segments of consecutive q's in r,  i.e., <j* £ Sg(q) implies Vi € 7, q*(t) — q. Let S S be the set of segments of consecutive B 1  and 5-states in r, i.e., q* e BS implies Vi e 7, g*(i) £ B U S . The run r satisfies the time condition iff (a) (local time constraint) V<j £T,q* £ Sg(q), p{q*) < r(q) and (b) (global time constraint) V<7* £ BS, PB(Q*) < r(bad). The first condition stipulates that for a local time constraint, the system will not stay continuously in a given state q £ T for longer than its local time bound r(q) . The second condition requires the system to leave the set of bad states within r(bad) time units. Definition 7.4 (Semantics of timed V-automata) A timed' V-automaton TA accepts a trace v, written v J= TA, iff all possible runs ofTA over v are accepting. We now present the syntax and semantics of average-timed V-automata, which extend the definitions for timed V-automata presented above. Average-timed V-automata allow for the verification of behavioural constraints for systems exhibiting uncertainty.  121  Definition 7.5 (Syntax of average-timed V-automata) An average-timed V-automa-  ton ATA is a triple (A, T, T) where A — (Q, R, S, e, c) is a V-automaton, T C Q is a set of average-timed automaton states and r : T U {bad} —> K U {00} is an average-timing function. +  Once again, we can easily show that any V-automaton is equivalent to a special average-timed V-automaton with T — 0 and r(bad) = 00. A T-state is denoted by a nonnegative real number indicating its average-time bound. However, unlike with typical V-automata, or even timed-V-automata, we cannot define the acceptance of a single trace by an average-timed V-automata. In fact, due to the stochastic nature of the systems of interest, we are no longer interested in the behaviour exhibited by individual traces but rather in the behaviour of a set of traces. Expected time constraints should be satisfied by the average behaviours of systems, hence we need to look at the ensemble of traces induced by those systems. Let B be the behaviour of a system. We define a run r of ATA over B has being a run of A over every trace v : T —• A in the behaviour B. A run r is accepting for ATA iff 1. r is accepting for A and 2. r satisfies the expected time constraints. If I C T is an interval of T and q* : I —» Q is a segment of run r, i.e., q* = r\i, let p(q*) denote the measure of q*, i.e., p(q*) = p(I) = E £ip(t) t  since I is discrete. Furthermore, let  states in q*, i.e., HB{Q*)  —  PB{Q*)  denote the measure of bad automaton  Ste/,(j*(t)eBA (*)- Let Sg{q) be the set of segments of consecutive t  q's in r, i.e., q* e Sg(q) implies V i £ I,q*(t) — q. Let BS be the set of segments of consecutive B and 5-states in r, i.e., q* € BS implies Vi e I,q*(t) £ B U S. The run r satisfies the time condition iff (a) (local time constraint) Vq £T,q* <E Sg(q), E(p(q*)) < r(q) and (b) (global time constraint) Vq* £ BS, E(p (q*)) < r(bad). B  where E ( ) denotes the expectation over all traces v of B. Definition 7.6 (Semantics of average-timed V-automata) An average-timed  V-automaton ATA accepts a set of traces B, written B |= ATA, iff all possible expected runs of ATA over B are accepting.  122  Figure 7.2: Timed V-Automaton Specifications: real-time bounded response As an example, Figure 7.2 depicts the real-time response constraint which states that R will be reached within 40 time units of B, where the time in S is not accounted for.  7.2  Model-Checking Approach  Before formally discussing the notion of behavioural constraint verification, it is necessary to discuss the relationship between stochastic dynamical systems and their behaviours. Intuitively, the behaviour of a stochastic dynamical system is the set of observable input/output traces of a given system. Let V(I, O) be a PCN module, where (I, O) is the tuple of input and output locations of the module. Formally, an input/output pair (i, o) is an observable trace of V(I, O) iff 3F & \V(I, O)] such that o — F(i). The reader should note that, within the PCN framework, the function F can be deterministic, non-deterministic or probabilistic, depending on what type of transductions (deterministic or probabilistic) and locations (hidden or not) are used to model the system. We define the behaviour of V(I, O) as the set of all observable traces and we denote it as {V(I, O)]. We will abbreviate {V(I, O)] to [V\ if I = I(V), O = 0{V) and no ambiguity arises. The notion of equivalency of PCN modules stems directly from their behaviour. Two PCN modules, V\ and V2, are equivalent, denoted V\ ^ 7 ^ 2 , iff they exhibit the same behaviour: [Pi] =  Now that we have formally defined the behaviour of a system, we introduce the definitions of time-invariant and Markovian behaviours. Definition 7.7 (Time-Invariant) Let B = {v\v : T —> A} be a behaviour. B is a time-invariant  123  behaviour if for any ai, a £ A, Vu £ B such that v(t\) — a\ and v(t\ + s) = a\ then Pr(v(t ) = 1  2  2  a-2\v(h) = a\) = Pr(v(t + s) = a \v(ti + s) — a{)forti 2  2  <t £T. 2  A PCN model is time invariant iff all transductions (deterministic or probabilistic) from the set of transductions are independent of the time parameter t. Another important type of behaviour, which has been studied extensively, is called Markovian behaviour. Definition 7.8 (Markov Property)  Let B be a behaviour. Given any time point t G T, v G B  and a £ A, such that v(t) = a, the behaviour is called Markovian (of order 1) if the probability distribution of the next state, v(t + 1), is independent of the past history of the system except for the state v(t) = a. More generally, a behaviour is Markovian of order n if the probability of the next state depends only on the n previous states. We will say that a system is strictly Markovian if it is Markovian of order 1. Proposition 7.2 (Markovian PCN)  Observe a PCN model at the finest level of details, i.e., looking  at the network created by the connections between locations and basic transduction (transliterations, delays and generators). A PCN model induces a Markovian behaviour of order n iff all the delays, for which the output location is included in the state space, are of finite length of at most n or if the sum of the strictly consecutive delays with output location not included in the state space is at most n. Recall that any Markovian system of finite order can be transformed into a strictly Markovian system. We can do so by augmenting the domains of the locations to contain a finite history of length equal to the order of the original system. Hence this allows us to solely consider strictly Markovian systems without loss of generality. These two type of behaviours constitute a very important class of behaviours and a vast number of physical systems exhibit them. In this chapter, we are interested in analyzing the behaviour of time invariant and Markovian systems. As we mentioned earlier, the PCN framework is very general and encompasses a much larger class of behaviour. We will thus restrict ourselves to the sub-class of PCN which are time invariant and Markovian. 1  This is sometimes also referred to as time homogeneous behaviour.  124  The class of discrete time stochastic dynamical systems exhibiting a time-invariant Markovian behaviour is very vast. In fact, any discrete-time, time invariant Markovian stochastic dynamical system corresponds to what we call a stochastic transition system. Since we are interested in the verification of properties specified as an automaton, we restrict ourselves to discrete-time. A stochastic state transition system is a tuple (<S, P, G) where <S is a set of states, P : «S x <S is an evolution kernel representing the transition probability distribution between two states, i.e., P ( s i , S2) is the probability of a transition occurring between s\, s 6 5 and 0 represents the distri2  bution of the initial state of the system . Notice that due to the time invariance and Markovian prop2  erties, P is independent of the time parameter and transitions depend only on the current state. For any discrete time structure T,v : T —» S is a trace of (S, P, 0) iff Vt > 0, ¥(v(pre(t)),  v(t)) > 0,  where pre(t) represents the time value preceding t. We will denote an allowed transition from v(pre(t)) to v(t) by v(pre(t)) ~> v(t). A behaviour B corresponds to a stochastic state transition system (S, P, 0) iff B is equal to the set of all traces of (S, P, 0). Stochastic state transition systems constitute a compact representation of time invariant Markovian behaviours. Let us now introduce the notion of probability measure on a behaviour. To do so, we introduce the notion of a Borel space on traces of a system, which follows [HJ94, BHHK03]. Let B be the behaviour of a stochastic state transition system S T S — (S, P, 0). A distribution 0 on the initial state of S T S induces a probability measure p& on its traces in the following way. First let so, si, • • •, s  n  G S with Si ~> Si i, (0 < i < n). Then denote C ( s o , s i , • • •, s ) by the cylinder set +  n  of all traces v G B such that v(Q) — s o , . . . , v(n) — s . Define ^(B) to be the smallest cr-algebra n  on the behaviour B which contains all cylinder sets C ( s o , . . . , s ) . The probability measure pe on n  F{B) is the unique measure defined by induction on n with base case n = 0 : ^ ( C ^ s o ) ) — © ( o ) s  and induction hypothesis for n > 0:  Pe{C{s , • - -, s„, s i)) = pe{C{s ,s )) 0  n+  0  n  • P(s„, s ) n+1  We define a behavioural constraint (or requirements specification) Be for a stochastic system STS — (S, P, 0) as a set of allowable input/output traces of the system, i.e., Be C Xiuo-Af- Let B = {STS] be the behaviour of STS. We say that STS satisfies the behavioural constraints Be at a level a, denoted by B (= Be iff p(B n Be) = p({v G B\v f= Be}) > a, where v |= Be a  2  This does not preclude the initial state of the system to be set in advance.  125  is the predicate for v satisfying the behavioural constraint Be- Perfect satisfaction of behavioural constraints is indicated by a = 1, which means that all traces of B (with the possible exception of a zero measure set) are allowable traces. Satisfaction at a level of a < 1 results in a subset of the traces of B being undesirable. Obviously, a satisfaction at level a = 0 is equivalent to a total absence of satisfaction, i.e., none of the possible traces of the system will ever satisfy the constraint Be (/z = 0  (fi fl Be) =  0). Behavioural constraint satisfaction of deterministic system is equivalent to satisfaction at level a = 1. However, in the presence of probabilistic behaviour, requiring satisfaction at level a — 1 might be too strict as one might be willing to accept a small risk of not satisfying the requirement (e.g., a = 0.95), rather then flat out rejecting the system. For the remainder of this chapter, we will elaborate a method which performs behavioural constraint satisfaction at level a = 1 for untimed constraints while temporal behavioural constraints will be satisfied on average. Methods for behavioural specification of average-time V-automata properties at any level a < 1 are being investigated and constitute future work. In Chapter 8, we will develop methods for verification at arbitrary level a. Now that we formally defined the notion of behavioural verification at level a, we can introduce the notion of robustness and complexity of systems. The robustness of a system is a notion defined on parameterized probabilistic constraint nets. We say that a parameterized system V[ is less robust than a second system V% with respect to a behavioural constraint Be, denoted by "Pf -<B P2'> C  iff Vp e xpDp, IVfKp) \= B a  c  => \V \ip)  equivalent w.r.t. B , written by V[ ~ s c  2  c  \=p B , for a < (3. The two systems above are c  P f , iff  <s  c  ^2  a n d  ^2 ^B  c  'Pi -  I n  t s case, hi  both systems would satisfy the behavioural constraint at level a = (3. Note that this definition of equivalence is somewhat more subtle than for deterministic systems. Equivalence of deterministic systems requires that the behaviours of the two systems be the same. This implies that the traces of the two systems are exactly the same. For probabilistic systems, we relax this assumption by requiring only that the measure over allowable traces be the same for both systems. However, it is easy to construct two equivalent systems for which their respective set of allowable traces is different, even though both sets have equal measure. Let us now define the complexity of a behaviour. Behavioural complexity is defined with respect to a given measurement on the size of a stochastic dynamical system. This measurement could be  126  the number of transductions, the number of delays or the maximum number of delay element in any path (which is equivalent to the order of the Markovian property of the system). Therefore, given a measurement n, denote \V\ as the size of the system V with respect to K. We then define K  the complexity of the behaviours satisfying the requirements specification Be, w.r.t. K and level a, written |ficl£» to be the smallest stochastic dynamical system which respects Be- That is, \Bc\% = "w«{|PU}p>lKBcWe have defined above the concepts of the behaviour of a system and of behavioural constraints. Given the behaviour B of a stochastic dynamical system and a behavioural constraint Be, the behaviour satisfies the requirements at level a, written as B \= 1ZS, iff p({v e B\v \= US}) = a. a  As mentioned before, we will restrict ourselves to the satisfaction of the behavioural constraints of a system at "average" level only. Therefore, given the probabilistic constraint net model of a system and an average-timed V-automaton specification of behavioural constraints, we say that the behaviour of the system satisfies the behavioural constraints if and only if the all traces of the system are accepting for the average-timed V-automaton. The formal behaviour verification method consists of a set of model-checking rules. The rules are a generalization of the rules for dynamical systems developed by [Zha94], which themselves extended the rules developed for concurrent programs [MP87].  7.2.1  Behavioural Constraint Verification Rules for Discrete-time  For sake of simplicity, we will introduce the verification rules for the simplest possible situation: discrete time and discrete domain. A generalization of the rules for arbitrary time and domain will follow shortly. As we mentioned earlier, any time-invariant Markovian behaviour B in discrete time corresponds to a stochastic state transition system (<Sg,P, 0) for which we denoted an allowed transition from state s to state s' by s ~* s'. We also write {ip}B{ip} iff the consecutive condition: <p(s) A (s ~> s') —> t/>(s') is valid. This relation is different from the one defined for deterministic systems in that it is valid not if there is a transition from s to s' but more generally if there is an allowed transition (non-zero probability of transition) from s to s'. Our verification method is composed of three types of rules: Invariance rules (I), Stability (Lyapunov-based) rules (S) and Average Timeliness rules (AT). Assume ATA is a V-automaton 127  (A, T, T) representing the behavioural constraints for the stochastic dynamical system: (S, P, G). (I) Invariance Rules We define a set of propositions {a } ^Q as a set of invariants for the behaviour B and specification q  q  .4 iff 1. Initiality: Mq £ Q, 0 A e(q) —> a , and q  2. Consecution: Mq,q' £ Q, {a }fi{c(g,q') —> a '}g  9  Proposition 7.3 Let {a } Q be invariants for B and A. Ifr is a run of A over a trace v £ B, then q  qe  VteT,v{t)\=a . r(t)  Note that this proposition stipulates that no matter which (uncertain) transition occurs, the destination state must always satisfy the invariant condition. This is consistent with the notion of invariants, regardless of whether the dynamics of the underlying systems are deterministic or stochastic. (S) Stability Rules Let {a } £Q be a set of invariants for B and A as defined above. A set of partial functions {p } ^Q q  q  q  q  is called a set of Lyapunov functions for B and A iff p : SB —> R satisfies the following condi+  q  tions: 1. Definedness: Vq £ Q, a —> 3w £ E , p = w. +  q  q  2. Non-increase: \/q £ S,q' £ Q, {a A p = w}B{c(q, q') —> E(p /) < q  g  q  3. Decrease: 3e > 0, Vq £ 5 , 3<?' £ Q, { a A p , = ui}fi{c(g, g') —> p ' — w < —e.} g  q  Those three conditions are derived from [ZM96]. However, the last two have been adapted for stochastic dynamical systems. Condition (S2) requires that for each stable state q £ 5, the transitions from q lead on average to a state for which the value of the Lyapunov function is less than or equal to the current value. Condition (S3) is similar in that it requires that for each bad state q £ B, there exists at least one allowed transition (i.e., with positive probability) leading to a state with strictly smaller Lyapunov value. This is a formal requirement that can only be satisfied if there are no absorbing bad states in the system under study, as discussed previously. 128  Proposition 7.4 Let {a } Q be a set of invariants for B and A. Let r be a run of A over a trace q  q&  v G B. Also, let VB = {r\r is a run of A over v G B} be the set of runs induced by B. If{p } eQ q  q  is a set of Lyapunov functions for B and A, then • V i G T , E . .(/o .( )(u*(i))) < p (pre(t))( (P ('t)))> where r* and v* denote all r' G VB v  r  )i;  r  t  re  r  and » ' E B such that v(pre(t)) ~> v'(t) and c(r(pre(t)),r'(i)), when r(pre(t)) G S; ' 3e > 0, Vi G T,3v' G B,r' G V , {[pr>{t){v\t))-p B  {v{pre{t))) < -e]A[(«(pre(t)) ~>  r{pre{t))  v'(t)) A c(r{pre(t)),r'(t))}) when r(pre(t)) G B. Theorem 7.1 Let {ct } £Q fee a sef of invariants for B and A q  q  Let r be a run of A over a trace  v G B. If{pq}q£Q is a set of Lyapunov functions for B and A, then • ifBS is the set of segments of consecutive B-states and S-states in r, then \/q* G BS, q* has a finite number of B-states; The results of Proposition 7.4 need to be applied to a set of traces, and thus to a set of runs. In the context of stochastic dynamical systems, we cannot guarantee that for every trace a transition from a bad (or stable) state to any other state will yield an immediate decrease (or non-increase) in the value of the Lyapunov function; nevertheless, we can show that there is a positive probability of this happening at any time point. Hence, we know that at any time i there exists at least one trace whose transition from v(t) to v(t + 1) will yield a decrease in the value of the Lyapunov function. (AT) Average-Timeliness Rules Let ATA = (A, T, r) be an average-timed V-automata. Assume, without loss of generality, that time is encoded in the stochastic state transition system. We assume that it is defined in a general sense as A : <Sg —> T ; i.e., as a function of time measure on states returning the time until the next transition. Note that for the special case of discrete time systems on N , A = 1 uniformly. We now define two different types of timing functions, associated with the local and global average-time bounds respectively. Once again, let { a } g be a set of invariants for B and A. A set of partial functions {7<j} er q  g e  g  is called a set of local timing functions for B and ATA iff j  q  conditions: 129  : SB —» K  +  satisfies the following  (LI) Boundedness: Vq £ T, a —> A < 7 < r(q). g  q  (L2) Decrease: Vq G T, { a A 7, = to A E(A) = /}fi{c(g, q) -> E(7 ) - UJ < - / } . g  q  A set of partial functions {7?g}<je<2 is called a set of global timing functions for fi and .4X4 iff rj : SB —>• K satisfies the following conditions: +  q  (Gl) Definedness: Vq £ Q,a  3w G K , r; = UJ. +  q  (G2) Boundedness: Vq £ B,a  q  q  ^ rj < r(bad). q  (G3) Non-increase: Vq £ S,q' £ Q, {a  q  Ar) = w}B{c(q,q') —• E(77 /) q  g  <  UJ}.  (G4) Decrease: Vq £ B,q' £ Q, {a Ar] — w A E(A) = Z}fi{c(q, q') -> E(?7 /) - UJ < -Z}q  Proposition 7.5  Ler  q  g  {ct } ^Q be a set of invariants for q  q  fi  and A, and r be a run of A over a trace  v £ A. If there exist local timing functions, {7q}qgT. and global timing functions, {n } Q, for fi q  qe  and ATA, then 1. if Sg(q) is the set of segments of consecutive q's in r, then Vq G T, q* £ Sg(q), E(/x(q*)) < r(q), and 2. ifBS is the set of segments of consecutive B and S-states in r, then Vq* G BS, E(/LXs(q*)) < r(bad).  The following is the set of verification rules (Invariance (I), Stability (S) and Average-Timeliness (AT)) for a behaviour fi and an average-timed automaton ATA = (A, T, r): (I) Associate with each automaton state q £ Q a state formula a , such that {a } Q is a set of q  q  q€  invariants for fi and A. (S) Associate with each automaton state q G Q a partial function p , such that {p } eQ is a set q  q  q  of Lyapunov functions for fi and A. (AT) Associate with each average-timed automaton state q G T a partial function 7,, such that {lq}q€T is a set of local timing functions for fi and .4734. Associate with each automaton state q G Q a partial function 77 , such that {Vq}qeQ is a set of global timing functions for fi g  and .4734.  130  Let us now present the main result of this section. The following theorem stipulates that if we are equipped with a set of invariants, Lyapunov functions and local and global timing functions, then the behavioural verification is sound and complete. Theorem 7.2 (Verification Rules) For any state-based and time-invariant behaviour B with an  infinite time structure and a complete average-timed V-automaton ATA, the verification rules are sound and complete, i.e., B (= ATA iff there exist a set of invariants, Lyapunov functions and timing functions. 7.2.2  Automatic Behaviour Verification  The above rules do not guarantee the existence of an automatic verification method. However, for finite domain probabilistic constraint nets, we can fully automate the process in order to verify an average-timed V-automata constraint on the behaviour. We will briefly describe the algorithm and then will utilize it to verify the elevator system augmented with probabilistic passenger arrivals. First, let us assume that PCN = (Lc, Td, Cn) is a probabilistic constraint net made solely of transliterations and unit delays. We denote an acceptable state by PCN(s) iff for every equation of the form l = f(li, • • • , l ) —> PT(S(IQ) = f(s(l\), • • • , s(l ))) > 0, and denote an acceptable 0  n  n  transition by PCN(s, s') if and only if PCN(s) and PCN(s'), and if for every delay equation 1' = I, s'(lo) = s(l). Let us also denote a reachable pair (q, s) by r(q, s) where q G Q and 0  s G xLA C  SI  .  Furthermore, let K be the evolution kernel associated with the set of reachable pairs  for q £ B U S and let T be the matrix summarizing the time for each transition within the set of reachable pairs with q G B U S. In addition, let K\ represent the evolution kernel for the set of reachable states with q G B U S which constitute the PS-boundary of the set B U S. This set is composed of all the bad and stable states which have a direct transition to a state r G R or to an absorbing S state. Proceed similarly to define T\. Finally, let L and L\ be the matrix T and T\ respectively where the non-zero entries have been replaced by the value 1. The algorithm follows the verification rules and has four steps which we describe below: 1. Invariant Generation: We can show that invariants can be constructed by finding the fixpoint of the sets of Equation A.4. This fixpoint can be obtained with the following two steps: (a) Initiality: Generate r(q, s) if 6(s), e(q)(s), PCN(s). 131  (b) Consecution: Generate r(q', s') if r(q, s),PCN(s, s'), c(q, q')(s'). 2. Non-Absorbness and Stability:  • Verify that the set of bad states is irreducible. That is, ensure that for every bad state b e B there is a path with non-zero measure leading to a P-state or an 5-state. If it is not the case, proceed to the modification of the state space discussed earlier such that absorbing bad states are removed. • For q G R, let p = 0. q  • Solve the set of linear equations for the average number of transitions taken to enter the set of recurrent states or absorbing 5-states. The solution is the set of Lyapunov functions, {p } eBuSq q  Practically, to solve for {p } £BuS,  u = diag([T,T\] * [K,K\\),  q q  define A — —K + I and n  then solve Ap = u. Here diag denotes the diagonal  operator, which returns the diagonal elements of a matrix. 3. Global Average Timing:  • For q e R, let rj — 0. q  • Similarly to the method for stability, solve the set of linear equations for the average time measure to leave the set of bad and non-absorbing stable states, not accounting for time spent in an 5-state. The solution is the set of Global timing functions  {ri } BuSq qe  Verify that n < r(bad),Vq e Q. q  4. Local Average Timing:  • For each q e T, solve the set of linear equations for the average time measure to leave q. This is similar to solving for the Lyapunov functions where we only consider states q € T. This leads to the local timing functions {7q}qer- Verify that 7 < r(q), V<7 € T. q  It is possible, given this method, to obtain a bound on the probability that a certain time bound will be exceeded. The bound is obtained from the well known equation Pr(X > r) < We can calculate E(X ), 2  E(X )/T . 2  2  where X is the average time to reach a i?-state or an absorbing 5-state, as  obtained in the global and local average timeliness rules. With the equation  u = 2[T, TI]. * [K, K1]*T]  + diag([T, TI]. * [T, TI] * [P, P I ] ' , ) 132  where the operator .* denotes the element-wise matric multiplication, solve AY — u for Y to calculate the value of the probability bound.  7.3  Generalizing the Approach  In this section, we generalize the concept of average-timed V-automata on discrete time structures to arbitrary time structures. This allows us to apply our method to behavioural constraint verification of stochastic hybrid dynamical systems, which generate traces on general time structures. Note that the common time structures of continuous and discrete time both act as special cases. Essentially, the set of verification rules for general time structures follows closely that of discrete time systems, however, the definitions of invariants, Lyapunov functions and timing functions are generalized. For any trace v : T. —> A, let {ip}v{ip} denote the validity of the following two consecutive conditions: • {tp}v~ {tp}: for all t > 0, 3t' < t, Vt", t' < t" < t, v(t") \= ip implies v(t) f= ip; • {<p}v {ip}: for all t < oo, v(t) |= <p implies 3t' > t, Vt", t < t" < t', v(t") (= ip. +  If T is discrete, these two conditions are reduced to one, i.e., Vt > 0, v(pre(t)) f= <p implies v{t) \= Given B as a behaviour, let Q = {v(0)|t> e B} denote the set of initial values in B. Let A = (Q, R, S, e, c) be a V-automaton. A set of propositions {a } Q is called a set of invariants q  q€  for B and A iff • Initiality: \/q G Q, Q A e(q) —• a . q  • Consecution: Vv G B, Vq, q' G Q, {a }v{c(q, q') —> a '}. q  q  Proposition 7.6 Let {a } ^Q be invariants for B and A. If r is a run of A over v G B, Vt G q  q  T,v(t) \= a . r{t)  Without loss of generality, we assume that time is encoded in domain A by A : A —> T . Given that {a } Q is a set of invariants for B and A, a set of partial functions {p } ^Q • A —> R is +  q  qe  q  q  called a set of Lyapunov functions for B and A iff the following conditions are satisfied: 133  • Definedness: Vq £ Q, a —> 3w £ E , / 9 = iu. +  g  q  • Non-increase: Wv £ B,Vq £ S, q' £ Q, {a Ap q  = w}t>~ { (g, </) -> E(p /) < w}  q  c  g  and Vq € Q,q' £ S,  {a A p = w}v {c(q, q') -> E(p >) < w}. +  q  • Decrease: Vu e  q  q  3e > 0, Vg £ B,q' £ Q,  {a A p — w A E(A) = q  { c (  q  9  , -  <-  e  }  and V<? £Q,q' £B,  {a, A „ = «, A E(A) =  ,<)  < _ . }  Proposition 7.7 Ler { a } Qfeeinvariants for B and A and r be a run of A over a trace v £ B. If g  {Pq}qeQ i  sa  set of Lyapunov functions for B and A, then  • E(p (v(t ))) r[t2)  •  ge  <p  2  M([ti,t ))  r(tl)  (t;(ti)) whenVh < t < t ,r(t) £ B U S, 2  ~  2  ew / z e n  i x  <*  ^h  2and  <t< t2,r(t) £ B, and  • if BS is the set of segments of consecutive B and S-states in r, then V<j* £  BS,PB{Q*)  finite. Let ATA = (A, T, T). Corresponding to the two types of time bounds, we define two timing functions. Let {ct } zQ be invariants for B and A. A set of partial functions {7q} eT is called a set q  ql  q  of local timing functions for B and ATA iff 7 : A —> R satisfies the following conditions: +  q  • Boundedness: Vv £ B,\/q £ Q,q' £ T, {a }v-{c(q,q') -» 7,/ < r(g')} q  and Vg £T,q' £ Q, {a A E(A) = t A-y — w}v~{c(q, q') -» w > /z([0, E(A)))}. q  q  134  is  • Decrease: Vv G B, Vq £ T, {a A q  = UJ A E(A) = i}u{c(q, q) -> ^ f e g )  7 G  < -1}-  A set of partial functions {n } £Q is called a set of global timing functions for B and .4.T.4 iff q  rj : A —» K  +  q  q  satisfies the following conditions:  • Definedness: Vq e Q , a , - » 3UJ G M ,775 = UJ. +  • Boundedness: Vq e B . a , - * E(?7 ) < r(bad). g  • Non-increase: Vv £ B,Vq £ S,q' G Q, { a Ar) — w}v~{c(q, q') -> E ^ / ) < UJ} g  q  and Vq £ Q,q' £ S,  {a Ar] = uj}u {c(q, q') -> +  q  q  E(r7 /) < g  UJ}.  • Decrease: Vv £ B,Vq £ B,q' £ Q, {a Arj — w A E(A) = q  and Vq £Q,q'  { (q, q')  q  < -1}  C  £ B, {«, A „ = „ A E(A) = <  K  « W ) -  E(r) >) - UJ q  <-!}.  Proposition 7.8 Let {ct } ^Q be invariants for B and A and r be a run of A over a trace v £ B. If q  q  there exist local and global timing functions for B and ATA, then • ifSg(q) is the set of segments of consecutive q's in r, then Vq £T,q* £ Sg(q), E(u(q*)) < r(q), and  • ifBS is the set of segments of consecutive B and S-states in r, then Vq* £ BS,  E(PB(Q*))  r(bad). The following theorem is a generalization of the soundness and completeness of the set of verification rules.  Theorem 7.3 The verification rules (I), (S) and (AT) are sound if the following conditions on B and ATA are satisfied: 135  <  • T is an infinite time structure. • All traces in B are specifiable by ATA. The verification rules are complete if the following conditions on B and ATA are satisfied: • {(v, r)\v G B,r is a run over v} is time-invariant. • All transitions from R to non-R-states are left-closed, i.e., ifr is a run, and there is a transition from a R-state to a B-state or a S-state at t, then r(t) G B U S.  The conditions for the completeness of the rules are imposed so as to be able to define Lyapunov functions for a behaviour and an automaton, as long as the behaviour satisfies the automaton. The second condition for completeness is always satisfied for traces with discrete time structures. More generally, the following proposition may apply. Proposition 7.9 All transitions from R to non-R-states are left-closed, if the following conditions are satisfied: • ATA is open and complete. • Mq G R, q\ G R and q G R, c(q, qi) A c(q, q ) is not satisfiable. -  2  2  • All traces in B are right-continuous.  These definitions are essential to provide understand of general behaviours of stochastic hybrid dynamical systems. At the present time, however, we have yet to develop an algorithm, either semi-automatic or automatic, based on these rules. Work in progress includes the development of such algorithms along with the augmentation of the behavioural constraint verification technique to perform quantitative probabilistic verification.  136  Chapter 8  Behavioural Verification with a Probabilistic Temporal Logic Probabilistic logics provide a simple yet powerful specification methodology for (dynamical) systems with temporal behaviour. We develop the Probabilistic Arbitrary-time Timed Temporal Logic (PATTL) for specifying the specifications on behaviours of uncertain systems, where timed refers to the notion of temporal evolution of the systems viewed as a metric distance. First we generalize the probabilistic computational tree logic (PCTL) [HJ94] into the Probabilistic Arbitrary-time Temporal Logic (PATL) to incorporate both discrete and continuous time, so that properties of arbitrary traces can be specified and reasoned about. Then we augment the modal operator so that real-time properties can be specified. Finally, we develop a first order PATTL (FPATTL) for arbitrary time and domain structures. The PATTL logic is intended to specify timed quantitative probabilistic behavioural constraints applied to a PCN model of a system. For example, in an elevator system, the building manager might want to reason about the overall service level of his elevator. He might be interested in maintaining a basic service level at all times where the incoming requests are served within 40 time units but might agree, as a cost saving strategy, to accept that the probability of the service level dropping below the basic level for at most a specified short period of time be less than 10%. This requirement is often referred to as quality of service and is a very important requirement for many applications. Therefore, it is essential to be equipped with a specification language that is powerful enough to allow us to explicitly state such requirements but also offers computational efficiency so that one  137  can verify that the desired requirements for real-world practical applications are indeed satisfied. Our PATTL logic arises from the well known logic called Computational Tree Logic (CTL) [Eme90, EC82], a branching time logic, which some of the most important probabilistic logics extend. While CTL is not expressive enough to specify quantitative properties of uncertainty, it has a natural correspondence with the computation of concurrent and non-deterministic programs. The fact that CTL is a branching time logic rather than a linear time logic makes it a good candidate to serve as the basis of our probabilistic temporal logic. Indeed, the underlying structure of time being tree-like provides an ideal framework as one can view each branch as a different probabilistic (or non-deterministic) successor of the current node. In our probabilistic approach to systems, we view the branching property of such logic as a representation of each possible event ui taken from the event space fl. We will elaborate on this notion later. However, for a more in depth (and classic) comparison of linear time logic and branching time logic, the reader is referred to [Eme90]. Recently, with the increasing number of practical applications where the inherent uncertainty is modeled, the need to specify quantitative, rather than qualitative, probabilistic properties has lead to the development of full probabilistic logics, most of them directly extending the CTL framework. PATTL is such an extension and is related to the Probabilistic Computational Tree Logic (PCTL) of [HJ94]: it generalizes PCTL into a logic for arbitrary time and domain structures which allows formulae to be interpreted over a general probabilistic constraint net model.  8.1  Introduction to CTL and PCTL  As mentioned above, our logic emerges from one of the simplest branching time logics: Computational Tree Logic (CTL) first introduced by [EC82], It introduces formulae where one of the two 1  path quantifiers, A (for all future paths) and E (for some future paths), are followed by the usual linear temporal operators that can be found in most temporal logics: X (next time), U (until), •  (always) and 0(sometime or eventually). C T L was developed as a modal (temporal) logic for reasoning about qualitative program correctness. Typical sentences in CTL expressing properties are: "pi will hold continuously on some  future execution path" (EDpi), "p will eventually hold on all future executions paths" (A<>p ), 2  2  'CTL can also be extended to a more expressive logic CTL* which allow the path operators to be followed by a boolean combination or nesting over formulae made with F, G, X or U. Here we will restrict ourselves to CTL. For more details on the extended logic CTL*, the reader is referred to [Eme90]. 138  and " p 3 will always hold on all future execution paths" (Anp ). Emerson etal. [EMSS92] have 3  extended C T L with RTCTL which is suited for specification of hard deadlines. Alur extended CTL in a way similar to RTCTL, but in their logic (TCTL) formulae are interpreted over models with continuous time [ACD90]. C T L formulae are generally interpreted over a structure M — (S, R, L) where S is a set of states, R is a binary relation on S x S and L : S —> 2 * is a labeling function which associates s  with every state in 5 a subset of atomic propositions srf which are known to be true in that state. Intuitively M can be seen as a directed labeled graph with S representing the set of nodes, R being the arc set representing transitions and L labeling every node of the graph. It is important to note that CTL formulae can also be interpreted on more general structures, a property that we will use when developing the logic to represent properties of a PCN model. The PCTL logic [HJ94] is based on Emerson, Clarke and Sistla's Computation Tree Logic (CTL) [CES86], an extension of the above mentioned CTL logic ([EC82]) with the added property of fairness included into the model. However, unlike with CTL, in PCTL one is interested in soft  deadlines such as: after a request for service is received, there is at least a 95% probability that the service will be carried out within 5 seconds. Note that using probabilities of 0 and 1 makes PCTL also suitable for expressing hard deadlines. Within PCTL, time is assumed to be discrete, with one time unit corresponding to one transition along an execution path. To allow for reasoning about soft deadlines, the authors have replaced path quantifiers by probabilities. Examples of PCTL properties are: with at least 35% probability,  property p\ will hold within 10 time units  (O|J  0 3 5  will hold continuously for at most 20 time units  )  and with at least 95% probability, property p  (•^J°  2  9 5  ).  PCTL uses the same operators as CTL  with the exception that the operators are augmented with probabilities and time intervals. PCTL properties are interpreted over structures that are discrete time Markov Chains. Figure 8.1 summarizes the relationship between the aforementioned logics. A l l these logics extend CTL to handle either hard or soft deadlines, hence quantitative properties. The arrow from PCTL to PATTL highlights the relationship between the logic introduce in this dissertation and its ancestor logic PCTL.  139  TCTL  RTCTL  CSL  PCTL  continuous time  discrete time  continuous time  discrete time  Soft Deadlines  Hard Deadlines  Figure 8.1: Comparing the logics extending CTL.  8.2  Probabilistic Arbitrary-time Temporal Logic (PATL)  As mentioned above, we are interested in developing a temporal probabilistic logic that will be used in the specification of behavioural constraints for systems modeled as Probabilistic Constraint Nets. Since one of the PCN framework's main advantages is that one can consider abstract time and domain structures, we would like our logic to assert semantics appropriate for such abstraction while also being powerful enough to specify temporal and probabilistic constraints. As PCTL allows for temporal and probabilistic reasoning, it is a perfect candidate to base our logic on. Moreover, a powerful verification method for PCTL behavioural constraints exists and this method can be extended to operate on a certain class of PCN structures. The algorithm will be described in the following sections. First of all, let us start by assuming that we are equipped with a finite set of atomic propositions srf — { a i , • • , a „ } which will represent atomic properties of states. Such atomic properties could be, in an elevator system for example, a : the number of passengers in the elevator at a given time  does not exceed the maximum number allowed. Formulae in our logic are built from a combination of atomic propositions with the propositional, temporal and probabilistic operators introduced below. Definition 8.1 (PATL Syntax) Let si be a set of atomic propositions, p G [0,1] be a real number 140  known as a probability bound (or threshold) and © G {>,>,<,<} be a comparative operator defined on the reals. The syntax ofPATL, given srf, is obtained inductively via the following rules: • true is a state formula; • each atomic proposition a G siis a state formula; • if<pi and <p are state formulae, then so are -np\ and <p\ A <p i 2  2  • ifipi and ip are state formulae, then (fiS(p , <pfU<p and Xip are path formulae, and, 2  'iff  2  2  is a path formula, then {pQ (<p) is a state formula. P  Here U, S and X denote the until, since and next temporal operators, while S? denotes the  probabilistic operator, with {pQp((p) asserting that the probability measure of the paths tha satisfy the path formula <p satisfies the bound ©p. In this definition, the logic operators -> and A hold the usual meaning of negation and conjunction. The probabilistic formula £pQyp((p) specifies that the probability of the path formula tp being true satisfies the bound Op. As we will see later in this section, this probability is obtained by computing the measure of the paths over which (p is satisfied. The reader should note that the operator @> replaces the common C T L quantifiers A and E. Instead of reasoning on the qualitative satisfaction of a given formula on one, some or all computation paths, we adopt a quantitative approach which allows us to infer about the probability of a formula being satisfied. Note that this quantitative approach does not preclude us from specifying qualitative properties of systems. We keep the ability to represent "there exists" and "for all" by using specific values for p within 3 Qp. The s  CTL requirement (E<p) which means "there exists a path on which <p is satisfied" can be expressed with ^ >o(y) while the CTL requirement (Atp), which means "tp is satisfied for all possible paths ,  (except, perhaps, for a zero measure set of paths)", can be represented by &>i(tp), given a fair interpretation of the CTL formula (A<p) [EL87]. In a fair interpretation of CTL, one only considers the set of paths that satisfy a certain fairness constraint (e.g., visit every state infinitely often). Hence, the satisfaction of the formulae is only with respect to those fair paths. For an elaborate discussion on the relation between fairness and probability, the reader is referred to [BK98a].  141  It is important to note that in the presence of uncertainty, the meaning of trace differs significantly from its usual deterministic meaning. Indeed, when dealing with systems behaving unpredictably, even if all the parameters of the system are fixed and no inputs are hidden, a trace does not represent the exact behaviour of the system but rather one of the possible behaviours. It is for this reason that instead of talking about the behaviour of a system, one often refers to its average behaviour. Therefore, to be able to assess the behaviour of a given system, one needs to observe many traces so that an average behaviour can emerge. Our goal in defining an interpretation function is to be able to relate every state (value in the domain A) to the set of atomic propositions that is satisfied in that state. We say that a pair (v(t), <p) is an element of the satisfaction relation |= if and only if the state formula tp is valid in state v(t). We denote the satisfaction of ip in state v(t) by v(t) |= <p. To introduce the semantics of a PATL path formula ip augmented with the probabilistic operator Jfr and state v(t), a probability measure p^  we need to define, for each PCN structure  on the set of traces starting from v(t).  Definition 8.2 (Probability distribution induced by a PCN)  Assume the following PCN structure  Jff —< L,T,C > with n locations, L = {Li}f , each with domain A^. =1  Let us denote the set  of probabilistic transductions by P C T. The set of probabilistic transductions induces a prob2  ability distribution F  x  : (xjAi) x (xjAi) —> [0,1] which is a well-defined probability distribu-  tion due to the well-definedness of each probabilistic transduction in T. Hence, F ^(at , J  i  ^ Li(0'ti i\^Li(ti)) >  +  a< ) = j+1  x • • • x PL (a>ti \lLn(ti)) denotes the probability of the cross-product x j L j  having values a<  n  j+1  +1  at time ti+i while having values a at time ti, where 1^ denotes the input ti  locations of transduction Pi . t  Based on this definition, let us consider the well-definedness of a PCN trace. Definition 8.3 (Well-definedness of a trace)  Let v : T —» XjAi be a trace and let J^f =<  L, T,P,C > be a Probabilistic Constraint Net structure with n locations and P C T as defined above. We say that a trace v is well-defined, given JC, iff v(t) is well-defined for all t e T (i.e., v(t) £ XjAi, Vt G T), and for every consecutive time point ti,ti \ +  G T we have  F (v(t ),v(t ))>0. J(r  i  i+1  The set of deterministic transductions also comes into play but each deterministic transduction indu a probability distribution with 0 and 1 values only, hence not affecting directly the result. 2  142  n )  Consider the PCN Jif = ({x}, {5 }, P(xt, x +i), C), which is  Example 8.1 (Undefined trace)  X  t  defined on T — N and A — {1,2} with sole deterministic transduction being the unit delay 8  X  and unique probabilistic transduction defined as: P ( l , and  P(2,2)  trace v —  =  0.  1) =  For this simple PCN model, we have  0.4, P ( l , 2 )  =  0.6, P ( 2 , 1 )  =  1  = P. Now consider the partial  Although this trace is defined for every value it takes over  {1, 2, 2 , 1 , 1 , 2,1}.  t =  the transitionfromv(2) to v(3) is ill-defined with respect to the probability distribution  { 1 , . . . , 7},  P^. Indeed, transitionsfromdomain value 2 to 2 has probability 0 in the model, and hence is n a valid transition. In fact, it is easy to show that such a trace has measure 0. • Let B^ denote the set of traces of the PCN  Definition 8.4 (Probability measure of trace)  =<  L, T,P,C > starting at a € x / A j , i.e., v(0) — a. In accordance with measure theory, we define: • For any sequence vo, V\,..., v , with Vi  l -f{0)(i J  V£  B  f \  XiA  €  n  \<n =  it  {V0,...,v })  V  n  = ¥ {v ,v ) x  0  x ••• x P - * > _ i , u )  l  n  n  where U|< denotes the restriction of v onto {t' £ T\0 < t' < n}. That is, the measure N  of a cylinder set of all traces v e B^ such that v[i] = vu (i < n) is equal to the product nr= ^("i.«i+i) 0  • For n — 0  pf ({v £ B^\v^ 0  • For any countable set  0  {5;}i / e  = {v }) = 1. 0  of disjoint subsets ofB^  i€l  16/  The sum is well-defined. This follows because it is bounded by 1 and each term in the su mation is non-negative. • The measure of the complement set B^\S where S is a subset ofB^ can be obtained via  143  pf (Bf \s) 0  0  =  i-pf (s) 0  As stated in the preceding section on the relationship between uncertainty and time, we will assume non-Zenoness of the probabilistic transductions. However, given a PCN model with con3  tinuous time and discrete domain structures, where the transport delays are of positive duration and the probabilistic transductions are event-based on exponential distributions, we can show that this assumption is superfluous. Indeed, the following proposition shows that the set of infinite traces where all the transitions are taken in a finite amount of time has probability measure 0. The proof of this result is similar in essence to the proof of Proposition 1 of [BHHK03]. This result can also be obtained from a not so simple application of Theorem 3 of [dA97a]. Proposition 8.1 (Measure of non-time-divergent infinite traces) Assume a well-defined continuous time T and discrete domain X[Ai PCN model (with positive transport delays to avoid algebraic loops) where all probabilistic transductions Pi are event-based with exponential distribution exp(Xi). For any initial value VQ £ XjAi, the probability measure of the set of infinite traces B :  VQ,V\,V2T  • for which time is convergent, i.e., ^2i>0U  < oo where ti denote the time ofevents  i > 0, converges to 0. Now let us present the definition of the structures over which PATL formulae will be interpreted. As stated above, we are interested in specifying the behaviour of probabilistic dynamical systems. In order to do so, we introduce a few notions that will be useful for the rest of this section. Definition 8.5 (Frame) A frame of PATL over a PCN structure J(f is a tuple (T, A, P ^ , V) where T denotes a time structure, A is a composite domain,  is the probability distribution induced by  Jff and V is referred to as an interpretation . Formally we define an interpretation as a function 4  V : A —> 2^. We will use v \= ip to denote ip £ V(v). Definition 8.6 (Model) A model of PATL is a tuple (F, v) where F = (T, A, P , x  V) is a frame  and v :T —> A is a trace generated by the PCN Jfr. Formally, the semantics of the PATL logic is defined as follows. 3 4  A system is called Zeno if it takes infinitely many discrete transitions in a finite time interval. /n the temporal logic and model checking literature, V is often referred to as a labeling function  144  Definition 8.7 (Semantics of PATL) Let T = (T, A,  V) be a frame and (J , v) be a model 7  of PATL. Let tp and tp be PATL state and path formulae respectively, v \=t <p denotes that state v satisfies tp at time t. Specifically the satisfaction of PATL formulae is defined as follows. V  h t true,  Vt G T.  V  ht a  iff[v{t) h a] = [a G V(v{t))] with a G si.  V  K -•</>  iffv(t) h tp.  V  ht <pi A ip  iffv{t) h <Pi A v(t) h P2-  V  H  iff3t' > t , V t " , i  V  ht <pi$V2  iff 3? <t,v  ht' f2 and Vt", t' < t" <t,v h t » <Pi  V  h t <Plb(<P2  iff 3? >t,v  ht' f2 and Vt", t < t" < t', v ht" <Pi  V  ht  2  x<p  < t" < t',v ht" <P-  W^ )({v\v(0)  PQPM  = v(t) A v h t  t  ip})Op.  To specify the initial satisfaction of a PATL formula tp, i.e., v h o <P, we will use the simplified expression v \= cp. A PATL formula tp, not extended with the probabilistic operator, is said to be valid over a frame J , iff for any model (J , v),v h </>• f is va/W, iff for any frame F , tp is valid 7  7  over F . tp is satisfiable over a frame  iff for some model (J , v),v h V - Finally, we say that tp is 7  7  satisfiable, iff for some frame J , tp is satisfiable over F . 7  8.2.1  Common Logical Connectives and Temporal Operators  With our syntax of PATL we are able to represent many other commonly used logical connectives and temporal operators. Here are a few of them represented within the PATL syntax. False: false  = ->true.  Disjunction: tp  x  V  tp  =  2  Implication: tp\ —> tp  =  2  Equivalence: tpi <-> tp  =  Eventually: Otp  =  Previous: Qtp  =  2  _ ,  (  _ , <  /'i A  ->tp2)-  -npi V tp . 2  (tp —> tp ) A (tp —> tpi). x  2  2  trueUtp. tpStp.  The propositional connectives V , —>, <-», have the usual logical meaning. The reader is referred to [Eme90] for the definitions of stronger and weaker variants of these temporal operators. The commonly used Always operator (Dtp), which is equivalent to -iO-np in the CTL* logic,  145  cannot be derived similarly in our logic. However, we can derive it using the duality of lower and upper probability bounds, e.g., £P> (nip) — £P<i- (0-«p). p  p  Based on the definitions above, we can present the semantics of these commonly used connective and temporal operators. Given a frame T = ( T , A , P^, V ) , a model (J , v) and a PATL 7  formula <p, the semantics are specified as follow.  False:  V  Disjunction:  V  Implication:  V  Eventually:  V  Previous:  V  Always:  V  Ft false. ift-v(t)\=<p vv(t)\=<p2x  H <Pi -> <P2 iff v \= <Pi implies v \= ip . t  t  2  iff 3t' >t,v \=f f. K  iff 3 i ' < t,W,t' < t" <t,v \= » tp t  iff Vt' >t,v \= > <p. t  For the elevator task mentioned earlier and which is described in detail in Appendix B, let E be the proposition denoting the entrance of a passenger in the elevator and let D representing the proposition which is true when the passenger is delivered to the floor of his selection. A desired property for the elevator is • ( £ • —> ^>i(true UD)). That is, whenever a passenger enters the elevator (event satisfying E), then with probability 1 the passenger will be delivered to his destination floor (D). Other important properties of behaviours mentioned earlier can also be specified using PATL. • Safety: Let B denote a proposition representing an undesirable (bad) situation: D-^B. • Goal achievement: Let G be a proposition representing a final absorbing goal of a system: one • Recurrence: Let R be a proposition of a recurrent condition: OOR.  8.3  Probabilistic Arbitrary-time Timed Temporal Logic (PATTL)  In the preceding section we have introduced a logic that can represent probabilistic properties over general traces. However, in this logic we did not take time explicitly into account. Many important behavioural requirements of dynamical systems contain a temporal component. For example, the 146  elevator property • (£--> 3?>i(\.rueUD)) would be more meaningful if one could stipulate an upper bound on the time of delivery. In this section, we develop the PCN timed temporal logic (PATTL), an extension of PATL which allows to specify the metric properties of time. The basic syntax and semantics of PATTL are the same as those of PATL with the exception that we augment PATL with two real-time operators: timed until, W and timed since, S , where r > 0 is a nonnegative real-number. T  Before defining the semantics of these operators, we need to introduce the concept of time evolution. Even though PATTL is based on a branching time logic (CTL), we still maintain our notion of linear time. As mentioned earlier, we see the branching property of the logic as representing the probabilistic choices within the system's dynamics rather than representing the evolution of time. With this notion of linear time, we need to define the notion of subsets of the time domain. The reason for this definition will become evident once we define the semantics for the real-time logical operators U and <S . T  r  Definition 8.8 (Time Domain Subset) Let r > 0 be a nonnegative real number representing a time duration. Let Tt+ = {t'\t < t',d(t,t') < T } and Tt- = {t'\t' < t,d(t',t) < r}, where T  T  d(-,-) is the usual Euclidean metric between two real numbers.  This definition allows us to reason about the time instances that are included between two fixed times. Remember that PCN is defined on an arbitrary time structure. Hence when reasoning about time intervals, it is useful to be able to assess what other times are included between time t and t + r. For example, for T = N and * i = 1, T = 4, the set T  t+T  is simply {1, 2,3,4, 5} while for the  time domain T — R, this set is the closed interval [1,5]. Given PATTL formulae tp\ and <p , the two real-time operators that augment PATL are defined 2  as follows: real-time Until: v \= <piU ip iff 3i' G T , v \= , ip and V i " , t < t" < t', v \= » (pi. (8.1) T  t  2  t  t+T  2  t  real-time Since: v \= <PiS (p iff 3t' G T - , v |= / <p and V i " , t' < t" <t,v (=t" <PiT  t  2  t  T  t  2  (8.2)  As one can observe from their syntax, the timed versions U and S simply extend U and S T  T  to allow for temporal reasoning. The formula tpiU (p expresses the property that ip will become T  2  2  true within r time units, period during which tp\ will remain true continuously, while the formula 147  (pi S Tip2 expresses the property that tp was true at some time value t' within the interval [i — r , t], 2  and for every subsequent time value, tpi held continuously until time i . Based on those two real-time operators, we can define other real-time operators that can prove useful for specifying real-time properties of systems. Eventually: 0 <p =  true U <p.  T  0 tp  T  EE true S (p. T  T  Always: The real-time Always operators n (p and D (p can also be derived using the duality of lower T  T  and upper probability bounds: & > (U ip) = 0 <i_ (O -Mp) and <?> (U ip) = 8P<\- {<> -y<p) >  T  B  p  T  p  p  T  v  T  The semantics of these new real-time operators is defined by Eventually:  • v |=t 0 ip v h t Orf T  iff 3t' G T , v ht' t+T  iff 3t' G T _ , v h t ' <Pt  T  Always:  v\=tD <p iff Vt' eT ,v\= , <p. T  t+T  t  v h t n <P iff Vt' G Tt- , v ht' ¥>• T  T  Now let us come back to the specification for the elevator task introduced in the PATL section. With the real-time temporal operators, we are now able to express the property: "Whenever a passenger enters the elevator (E), then there is a probability of at least 95% that the passenger will be delivered to his destination floor (D) within r = 30 time units. In PATTL, this specification is denoted by n(E  - > ^> 0 .95(true  U D)). 30  We define the size of a PATTL real-time logical operator as log(k) where k is the number of time steps included within T  , with r being the time threshold of the real-time operator. For instance,  t+T  assume that we have the PATTL formula <pi U'V2 with T — N. In this case, k = r = 7, i.e., there is 7 transitions that will occur between time t and time t + r . For general time structures with equally spaced increments of A , there will be maxfc{/cA < r} time steps. The logarithmic property of the size of the real-time operators comes from the fact that in matrix representation P , which is fc  the probability transition matrix after A; steps, can be computed in 0(log k) matrix multiplications. Finally, we also define the size \<p\ of a PATTL formula <p as the number of propositional connectives  148  and temporal operators added to the sum of the sizes of the real-time operators in (p.  8.4  Time Structure and Verification Issues  In order to proceed with the verification of PATL and PATTL formulae we need to first specify the meaning of the time bound r in the temporal formulae U and S . Indeed, since PCN models are T  T  denned over arbitrary time structures, one needs to have a clear understanding of the time bounds. We view the time bound r as the absolute time reference, independent from the time structure of the PCN model under study. For instance, assume PCN with a time structure such as T = {3t\t 6 N | | . A trace on such a time structure would have a value for every time point in T = {0,3,6,9,...}. If one desires to verify a property such as <pi U ip , it is important to note that one should only be 2  2  interested in the first value of the trace, as any other value occurs after time t = 2 and hence has no effect in the satisfaction (or dissatisfaction as it may be the case) of the behavioural constraint. However, one could ask what is the value of the trace at time t — 2 since it may have a crucial effect on the satisfaction of the formula <pi U <p - Thus, one needs to pay close attention to the meaning 2  2  of time along with the various time structures that a system can have.  8.4.1  The Structure of Time  Time is an essential component of dynamical systems. Hence, understanding time is absolutely necessary to understanding-dynamics. Our approach consists of using an abstract structure to formalize time so that it encompasses its most important aspects. In general, we view a time structure as a linearly ordered set with a well-defined start point, a metric associated to the distance between any two time points and a measure defined over the duration of an interval of time. This approach is analogous to the results obtained for the C N framework [Zha94], which we summarized in Chapter 2. As mentioned earlier, the PCN framework allows for the modeling of hybrid systems. A hybrid system consists of subsystems acting on different time structures. Hence when considering the verification of temporal logic formulae, one needs to ensure that the systems' traces are well defined for any value of the time bound r, which is independent of the time structures of the system. For systems modeled with multiple clocks, sampling and extending are common transductions that are used to synchronized the various components. 149  When verifying multi-clock systems, we will assume that the trace of the system has been extended to the most dense time structure that takes into account all the various clocks of the system. Based on this assumption, we present the following proposition which guarantees that no matter what reference time is used, the verification method is still sound. Proposition 8.2 Assume a PCN trace v : T —> A which satisfies a PATTL formula tp. Let T be r  any reference time of T with a reference time mapping h : T —> T . Then the extension trace r  v : T —> Aofv onto T also satisfies ip. r  r  The effect of an extending transduction on the transition probability of a given location is very simple. For every extended value, the probability that a location remains in the same state is one, while the probability that the location takes on any other value in its domain is 0. Hence, we generalize the notion of probability distribution induced by a PCN with multiple clocks, which we denote with Pf, where r refers to the most dense reference time structure of the system. Definition 8.9 Transition probability distribution induced by a PCN with multiple clocks  Assume a PCN X = (L, T, P, C), a frame (T, A, P , V) and a reference time T of T with a x  T  reference time mapping h. The probability distribution induced by the PCN over the reference time structure T is defined as follows: r  P  x  (at , a-ti+i) if no locations are extended at ij+i t  0 Pf(ati,a ,ti i) ti+1  +  if any Ljare extended at ti i and a° . ^ Of  PLI x • • • Pi  +  k  if L\,...  t  Lk are not extended at ti+\ and for all  extended locations Lj,j = k + l,...n, we have ai — ai We need to add an extra parameter to the transition probability distribution to take into account the current time value. Indeed, the temporal location of the system will influence the value of the transition probabilities. If a location is getting extended at time t, then its value will be the same than at the previous time step with probability one. This will be important when describing the verification algorithms as it enable us to verify systems with arbitrary non -continuous time structures. The reader should note that these results not only apply to multi-clocked systems but also to event-driven systems. An event-driven transduction works in a way that is very analogous 150  to a transduction with input of different clocks. Indeed, for an event-driven transduction, the input trace with reference time T is first sampled onto the sample time T generated by the event trace e. e  Secondly, the primitive transduction is performed on the time structure T . Finally, the output trace e  is extended from % back to the initial time structure T . Another important fact is that the total number of different probability distributions induced by a PCN with multiple clocks is always finite. The following proposition summarizes this result.  Proposition 8.3 For any PCN with multiple clocks, the number of different induced probability distribution isfiniteand at most 2" — 1, where n is the size of the location set L. 8.5  Model Checking of PATTL over finite PCN and non-continuous time structure  In this section we present an iterative model checking algorithm, which, for a given frame T = (T, A,  , V) and a PATTL formula tp determines whether a PCN model with finite domain satis-  fies the formula tp. The algorithm terminates when each state is labeled with the set of sub-formulae of tp that are true at that state. We will focus on PATTL formulae since as we will show, the verification of PATL formulae collapses into the PATTL verification procedure with r = oo. As mentioned previously, the PCN framework is designed to handle arbitrary time structures, whether, discrete, continuous or event-based. Although our algorithm is based on CTL model checking [CES86] and on the PCTL model checker of [HJ94], it is more general as it can handle arbitrary discrete and event-based time structures. For PCN models with a continuous time structure, we will present another algorithm, based on the CSL model checker of [ASSBOO, BHHK03], which can handle continuous-time systems where the time delay between transition events is distributed exponentially. In this setting, we will also discuss another probabilistic operator,  ^©p(v),  as introduced  within the CSL logic, which allows for the specification of steady-states requirements. Similarly to other model checking algorithms, in order to proceed with the iterative labeling of each state, our algorithm introduces a variable valid(vt) that indicates which sub-formulae of tp have been shown to be true at state v . As a base case, each state vt is labeled with the atomic t  propositions that are true in vt, that is, we have at first valid(v ) = V(vt),\/v G A, where V(-) is t  t  the interpretation function obtained from the frame T. Then, the labeling of the states is performed  151  starting with the smallest sub-formulae of tp that remain to be labeled until the algorithm labels the states with tp itself. Obviously, the labeling of composite formulae depends on the labeling of their parts. For instance, the label ipi A ip is added to valid(v ) if ipi G valid(vt) and tp G valid(vt). 2  t  2  The other logical connectives such as ->, V, —>,<-> are handled similarly.  8.5.1  Verifying modal formulae in PATTL  Lets us now present the algorithms for handling the labeling of states with modal formulae of the form ^Q (X(p),^ Q (ipiU ip ) >  T  P  p  2  and £?Q (ipiS ip ).  We will verify these formulae for the different  T  P  2  sets of values of p: p > 0, 0 < p < 1 and p > 1 and for the different possible values of r: r = 0, 0 < r < oo ( r is finite) and r = oo. One should note that, as mentioned before, the cases of p = 0 and p > 1 correspond to the existential and universal quantifiers, respectively, with the exception that these operators are restricted to non-measure zero sets of paths. The case of r = oo collapses to the PATL version of the operators. Hence PATTL verification encompasses PATL verification as one of its sub-case. For the remaining of this section, we will assume that the system's coarsest time structure T is r  known and the increments of the time structure are regularly spaced by A > 0. The increments r  of an event-based time structure, or even multi-clocked, might not be equally spaced. However, the results provided here will also hold for this case with the exception that the notation is more complicated and harder to follow. Hence, without loss of generality, we only depict the case for regular increments. Operator  T  Method  Xip  -  use Pf  0  equivalent to v J= <p compute 3§(T, v(t), t) simple clock: compute 38 (oo, v(t)) multiple clocks with cycles: compute 38{po, v(t),n) multiple clocks without cycles: approximate ^ ( o o , v(t), t)  (0,oo) <PlU ip T  2  00  <P\S f T  2  -  [0,t)  2  reversible distribution: see ipiU <p non-reversible distribution: compute 3>(T,v ) T  2  t  Table 8.1: Comparing the diverse cases  Table 8.5.1 summarizes the different cases that we will consider, along with a brief description 152  of the method used.  8.5.2  Model checking of the X operator  Given a PCN frame (T, A, Pf,V),  the verification of a PATTL formula of the form &>Qp(Xip)  directly involves the transition probability distribution matrix P ^ . To obtain the set of states which satisfy ^^(Xif), {0,1} given by ^(v)  simply construct a column vector n  v  : A —>  = 1 if (p e valid(v) and 0 otherwise. Then, the set of satisfactory states is  obtained via {v G A\x 0 p} where x — P ^ • n . v  v  v  Note that in the case of multi-clocked systems, the next operator is exclusively defined at the coarsest time structure of the system, i.e., the time at which the next transition occurs in one of the locations of the system. Hence we need to use the generalized transition probability matrix ff  in  our calculations.  8.5.3  Model checking of the U operator when r = 0 r  In this case, it is easy to show that the satisfaction of £?Qp(<pi U°ip ) collapses to showing v J= tp , 2  2  regardless of the value of the probability threshold p [HJ94].  8.5.4  Model checking of the U operator when 0 < r < oo r  Here we consider the case where the time threshold r is finite and the probability threshold p is comprised between 0 and 1 inclusively. The special cases for p — 0 and p = 1 can be solved more efficiently if handled separately as they correspond to the RTCTL model checking for the formula E(ipiU- ip2) T  and A((piU- <p2), respectively [EMSS92]. For an example of such improvements T  for systems modeled as Markov chains, the reader is referred to [HJ94]. Assuming that we have done the labeling of the states for formulae <p and <p , we now give x  2  an algorithm for labeling states with the PATTL formula gpQpiipx U <p ), for 0 < r < oo. We T  2  generalize the approach of Hansson and Jonsson [HJ94], by extending the notion of measure of paths satisfying until formulae to handle arbitrary non-continuous time structures over a PCN model.  Definition 8.10 Path Measure for until formulae satisfaction  153  Assume a PCN structure J(f, a frame (T, A, Pf, V) and a real value r > 0. For a state v of t  the system, the function 38(T,v ,t) defines the measure of the set of traces v in B^ for which t  ci tf2- For negative values ofr, r  v f=t fi  we define 38(T, vt, t) = 0.  Proposition 8.4 For r > 0, 38 (T, v , t) is obtained via the following recurrence relation: t  1  if f2 £ valid(vt)  38(T,v ,t) = { 0  elseifipi ^ valid(vt)  t  ^2Pf(v ,v',t t  Note that since A  r  + A ) x 38(T - A ,v',t + A ) otherwise r  r  r  is positive, we are guaranteed that the recurrence relation will be well-  defined and always terminate (after k > 0 steps when r — A;A becomes negative). r  Based on the result above, we have that 38(T, v , t) provides the probability with which a certain t  state, namely v , satisfies the PATTL formula LpiU tp . Hence, we label state v with tpxU <p2 if T  t  38{T,  T  2  t  v ,t)Qp. t  In Figure 8.5.4, we provide an algorithm, based on Equation A.5 for calculating the value of 38(T, v , t). Let us assume that time is equally spaced by A and that k G N is the maximal value t  r  such that fcA < r. r  The worst case complexity of the above algorithm is ((fc + 1) x (\A\ + 1) x 2 x \A\), where | A| is the size of the state space of the PCN model. The most outer loop is run k + 1 times while the for all loops are run through \A\ times, with an additional assignment statement within the first if statement. Finally, the factor 2 arises from the two additional assignments performed in the second for all loop. Therefore, we can conclude that our algorithm requires 0(k x \ A\ ) operations. 2  However, for most PCN, the diverse deterministic transductions will be such that a large portion of the state space (the cross-product of all location domains) will generate transitions with probability 0. Indeed, consider an addition transduction which adds up the value of location x and y and for which the domain of the output location z is N . In this case, the probability of reaching any state of the form {x = 1, y = 1, z — z'}, will be 0 for every z' G N\{2}. Hence, we can significantly improve the performance of the algorithm. In fact, if one considers the transitions with non-zero probability, the algorithm now requires Q(k x (\A\ + \B\)), where \B\ denotes the number of  154  for (i :=0tofc){ for all (v G A){  if (if 2 & valid(v)){ @(iA ,v,t + (i + l)A ) := 1 r  r  }  else{  3S(iA ,v,t + (i + \)A ) :=0; r  r  if(y?i G valid(v)){ for all (v' G A){ <%(iA ,v,t + (i + l ) A ) := ^ ( i A , u , t + (t + l ) A ) + P f («, v', t+(i + l ) A ) x - l ) A , v,t + (i + 2 ) A ) }} r  r  r  r  r  }  }  }  r  r  Figure 8.2: Algorithm for calculating B8(-) transitions in  with positive probability. One can easily see that, for a PCN where the whole  state space A is reachable from any state v £ A, the two complexity expressions are equivalents as in that case \B\ = \A\ . 2  Using the expression for the size of PATTL formulae and real-time operators, we can now define the complexity of deciding whether a PCN model satisfies a formula of the form (p — <pi U (f2- It T  is easy to show that the worst case complexity for this problem is 0(kmax  x  (|-<4| + \B\) x \tp\),  where \<p\ is the size of the PATTL formula tp,fcmaxis the maximum number of transitions between the current time and the time threshold r and \ A\ and \B\ are as previously denned.  8.5.5  Model checking of the U operator when r = oo r  Any PATTL formula with r = oo is equivalent to the PATL counterpart. Hence model checking of <p = ipi M°°<P2 is also applicable to the model checking of <p U(f2. x  However, it is easy to see that the preceding algorithm cannot be used to model check formulae with r = oo. Indeed, in this case, the algorithm would require infinite calculations hence rendering the model checking procedure infeasible. Therefore, we have to adopt a different approach to solve formulae with infinite time threshold (or absence of temporal component as in PATL). Let us define 38(oo, vt) as the measure of the set of paths w G B^ which satisfy ip.  155  In this section, we develop algorithms to model check PCN models of systems with arbitrary non-continuous time structures. We will consider three distinct families of time structure, each of which will require a different algorithm to be solved. The three families under consideration are: 1. single-clock system, which amounts to stationary transition distributions; 2. multi-clock systems where the induced transitions distribution are cyclical, i.e., the order of the probability distribution repeats itself according to a finite period; 3. multi-clock systems with no structure in the order of the probability transition distribution.  Single-clock systems For a system within the family of single-clocked systems, a sole probability transition distribution is induced. This distribution results from the cross-product of all the probability distribution of the locations of the system. Let A* = {vi,... ,v } denote the state space of a PCN JC. We partition the space A n  x  into  three subsets, denoted by A^f, A^ and A^ in the following way: • Aiff: The valid states which satisfy tp with probability exactly 1. • A^j? : The failure states which satisfy tp with probability exactly 0. • A^  = A^\(A^  U A^).  The equivocal states are states for which a conclusion cannot  be reached but which have a positive probability of reaching a valid state. For every state v G  , we have that tp\ G valid(v) and tp £ valid(v) with the extra requirement that 2  there exists a path with non-zero measure from v to a state w G A^.  Hence, these states will  form the paths that lead to a valid state. The first problem is the identification of the partitions. It has been showed that A^  and A^  can be determined by ordinary fix point computation [CY95]. Once A% and Aj^ are known, it is easy to compute  by using set difference. Once these partitions have been identified, the next  step consists of solving the set of linear equations defined by:  156  1  if v e  Af  0  elseif v € Af  t  t  Pf(v ,v') t  x ^(oo, v')  otherwise  The solution of the linear equation systems can be computed by any direct method such as Gaussian elimination, or iterative method like Jacobi or Gauss-Seidel. For Gaussian elimination, the problem can be solved with a complexity of 0((\A\ - \ Af \ - \ A* | )  2 8 1  ) [AHU74].  Multi-clock systems with cycles In the general case of multi-clock PCN models, we have a non-stationary set of linear equations with a parameter of infinite domain. Hence Gaussian elimination is not feasible in this case. However, for systems where the order of the induced transition probability distributions is cyclical, we can still use iterative methods such as Gaussian elimination. Let us assume that the period of the cycle is equal to n . Then, using the same partitions as above, we need to solve the following set of linear p  equations: 1  if v e Af  0  elseif v G A^f  t  t  3§(oo,v p)  P f («t,«',(p + l ) m o d n )  u  (  0  )  p  v'&A  x  x£§(oo, v', [p + 1) mod n ) p  otherwise  where mod is the usual modulo operator. In fact, any system, for which its clocks have equally spaced increments, will generate a cycle in the order of the induced transition probability distributions. The following proposition demonstrates this fact and indirectly also shows that we can use the set of linear Equations 8.3 to solve for •^(oo, vt, n). Proposition 8.5 Let Cl — {ci, c ,..., c } be the set of all clocks of a PCN W. If for every clock in 2  k  Cl the time increments are equally spaced, then there exist afinitecycle in the order of the transition distributions  .  157  Multi-clock systems without cycles For fully non-stationary systems without any well-defined structure such as cycles, the model checking problem is more complicated. In fact, the following theorem shows that for such systems, there does not exist a finite iterative method that can solve the following model checking task exactly.  1  38(oo,v ,t) = <j  if  v eAf t  t e Af  0  e l s e i f v  t  Y  ^f{v  v', t + A )x  u  r  3S(oo, v', t + A ) r  (8  .4)  otherwise  Let us first introduce the corollary which will be used in proving the theorem below. Corollary 8.1 Vp G (0, l),p G R, and any k G N, there exist a subset I C { 1 , . . . , k} such that the following inequality holds:  p - ^ ^ Y . l i -  -  p  (8 5)  Theorem 8.1 There does not exist a finite iterative procedure to solve the set of linear Equations 8.4 for any order on the induced probability distributions. The results of Theorem 8.1 indicates that the measure of the paths satisfying <p cannot be computed exactly. Hence, we need to proceed with an approximate procedure. To do so, we will once again refer to the partition of the state space that was described in the previous section. Based on these three partitions, we define the |^4| x |A|-matrices Mt by PfiVi,  Vj,t + A ) r  •M [vi, Vj] = I 1  if Vi G  if  t  0  V i  Af  <£Af A i ^ j  (8.6)  otherwise  For nonnegative values of t, we define £S{r,£) as the column vector of size |^4| whose ith element 38(r, i)i is 38(T, Vi, t). Therefore, we have that 38(0, t)i is 1 if Vi G Af From this, we to obtain the following proposition: 158  and 0 otherwise.  Proposition 8.6 For any r > 0, we have 38(T, t) = M x  Mt+A  t  x • • • x Mt+rAr  r  x ^ ( 0 , t) — M  x 38(0, t)  (8.7)  For finite values of r, it is easy to show that Equation 8.7 yields the same result as recurrence equation of Proposition 8.4 [HJ94]. However, for T — oo, it would take an infinite number of matrix multiplication to obtain the exact result (as it was shown in Theorem 8.1). However, we can proceed to an number of multiplication, knowing that in the limit, the result will converge to the true path measure. In fact, lim _,oo 38(T, t)i = 38(oo,Vi,t). Using this method, we can bound the result r  from both above and below, hence obtaining a good assessment of the value of the approximation. Proposition 8.7 Bounds on path measure  Let M be the number of matrix multiplication such that we have 38(oo,t) = Ait x -Mt+A  x r  ••• x M +M A T  i e {k\vk  x r  e  38(0,t) = M x 38(0,t). Furthermore, let 5f  = 2~2j\ &Ai^  j], with  Vj  A?}.  Then, for every equivocal state V{, the true measure 38(oo, Vi, t) of the paths satisfying tp lies between the following bounds:  3S(oo, t)i < 38(oo, v t) < J(oo, t) + 5™ u  8.5.6  Model checking of the S  T  (8.8)  t  operator  In this section, we will show how to model check the PATTL formula ipi S ip2- Note that the unT  timed PATL formula ip Stp is equivalent to the PATTL formula with r = t. Indeed, because the x  2  since operator looks back in the past, the maximum value of r, for the formula to be well-defined, is t. Hence, we will always deal with finite time threshold when model checking a formula with the since operator. One interesting case arises when the induced probability distribution of the PCN model is time stationary and a limiting distribution it exists. In this case, and for large enough t values, the induced probability distribution is reversible and we can solve the formula 159  ^  2 ,  Q ((/?I p  S <p ) using T  2  the method of the until formula with the reversed induced transition probability distribution =2 which is obtained via the following equation:  (8.9)  Note that we are using the fact that the system is running at stationary level, hence the assumption of a large t value. It is also well-known that if a system is reversible, then the rate of transitions i —> j is equal to the rate of the inverse transition j —> i. This result comes from the fact that the operators U and <S are symmetric. Hence for reversible systems, both formulae are solved by the same technique. Example 8.2  Reversible system  As an example of a system that has reversible dynamics, consider a PCN model Jfc of a random wa on the set  {0,1,2,...,  N}. A random walk on the integers (or a subset of the integers) is define  as moving either one step up or one step down during each time step. In general, the trans probabilities are of the form ^ ( i , i + l) = ai = I ^(0,1)  =00 =  &>*(N,N)  =a  N  (i,i - 1) (8.10)  1-^(0,0)  = 1- &>*(N,N  -I)  To show that this system is reversible, consider a process that moves up from position 0 < i  N. If the process is to move up from i once again, then it had to have moved down from i + 1 sin  there is only one way back to state i and that state is via i + 1. Therefore, for each move up a there must have been a move down from i + 1. Hence, the rates up and down across the i <-> boundary are equal. • For non-reversible systems, the problem of model checking the formula ^Q (ip\ S ip2) amounts T  P  to finding the measure of all the paths for which a tp state is reached within [t — r, t] and, for all 2  subsequent time step up to t, the paths never enter &-xpi state. Let us define 3>(T, v , t) as the measure of the paths, ending at state v at time t which satisfy t  t  the PATTL formula of the form tp\ S ip . T  2  Definition 8.11  Path Measure for since formulae satisfaction 160  Assume a PCN structure J^, a frame (T, A, Pf, V) and a real value r > 0. Also assume that  the time structure of the system has regular time increments, namely A . Furthermore, for sak r  of simplicity, let us assume that the induced transition probability distribution is stationary. Th  extension to the non-stationary case is straightforward although it is notationally more involved.  For a state Vt of the system, the function 3)(T, u ) defines the measure of the set of traces v i t  B^ for which v  \=t ipi S ip2T  For negative values ofr, we define  @(T, v ) = 0. t  The following proposition presents how to calculate the results for the paths measure defined above.  Proposition 8.8 Let us consider two separates case for the value of the threshold r. • For r < A ort = 0, @(T, Vi) is obtained via the following equation: r  (8.11)  @( ,v )=n'  P2  T  i  • For T > A , the paths measure is obtained with r  M  2  + n (i)  x {&>f) ^  V2  M  $>(T, Vi) =  (8.12)  and where 3?^ is the vector of the system's initial state's distribution, Mi — ]J ^ nax  Mi = [ ^' ^\, and the matrices Jvi _> are defined as follows: mi  T  A  lf>k  lfil  3Pf{yi, Vj)  0  8.6  ifvi (= <p and Vj (= <pi t  otherwise  k  t  (8.13)  Model checking of PATTL formulae with continuous time structures  When dealing with a system acting on a finite state space and a continuous time structure, some properties of the probabilistic behaviour is required. For instance, the numbers of transitions in non-overlapping intervals should be independent for all intervals. The occurrence of a transition  161  t  T  )j  )  should in fact be independent of when the previous one occurred. Furthermore, to ensure fairness, the probability of two or more transitions in a sufficiently small interval should converge to 0 as the interval's width diminishes. As we have seen earlier, a way to ensure that the above properties are satisfied is to model the system via event-driven probabilistic transductions where the events are subject to an exponentially distributed random delay. We explained how one arrives to the choice of the exponential distribution as the model for transitions delays in Chapter 4 Within a PCN model, the event-driven probabilistic transductions can be triggered by the same event generator or there can be one event generator per transductions. This is characterized by the respective behaviour of the system: 1- the value of every locations changes at every event, 2- a transition of the system only implies one location changing value. Hence in the second case, we are faced with a local transition within the state space. Let us now approach the problem of verifying PATTL formulae for continuous time systems. As we have shown previously, given a PCN model where the probabilistic transductions are driven by events with exponentially distributed random delays, we can construct a rate matrix 1Z : A x A —> M  +  which denotes the rate at which the system will transition to a new state. Once the rate matrix  is generated, the verification method is essentially analogous to model checking continuous-time Markov chains (CTMC). Hence, in the remainder of this section, we will restrict ourselves to a brief description of the method. For a thorough introduction to the methods, the reader is referred to the seminal work on verifying temporal formulae on CTMCs [BHHKOO, ASSB96]. Using the rate matrix 71, let us define the matrix (8.14) £(v) denotes the the rate with which the system will be taking a transition from state v to any other state (including state v itself). Given the exponential nature of the transition delays, it is easy to show that the probability that the system will take a transition from state v within t time units is equal to 1 — e ^ * . The probability of the destination state v' is obtained via what is called a race _ £  condition. That is, the probability P(v, v') for the system to transition from state v to v' equals the probability that the delay of the transition for going to v' ends before any other transition delay. Hence we have that P(v, v') = 1Z(v, v')/£(v) if £(v) > 0. Otherwise, P(i>, v) = 1 since the state v 162  is absorbing and the transition delay is infinite. In this setting, the time between transition is unknown (exponentially distributed). Therefore, when proceeding to the verification of PATTL formulae, we need to consider the time between transition explicitely. Hence we define a path in a PCN with exponentially distributed transition delays as an augmented trace  ^0*0^1*1^2 • • • where 1Z(vi, v i) > 0 and ti G R + for all i > 0. i+  Note that since a transition can lead from Vi to  itself, U does not represent the time spent in Vi  but rather the time spent in state Vi before a transition occurs. The need to explicitely take selftransition comes from the construction of the rate matrix TZ from the underlying PCN model, as it was explained earlier. Denoting the set of infinite traces starting in state v by B , we define the probability measure as v  in [BKH99]. Assuming that the states vo, • • •, v G A satisfy the condition mathcalR(vi, i>j+i) > n  0, for all 0 < i < n and that the time intervals 7o,..., I -\ n  are non-empty, nonnegative intervals  of the real line, then we can define the cylinder set C(vo, IQ, • • •, I -i,v ) n  n  as the set containing all  traces votov\tiV2 • • • where U G Ii for i < n. This measure is made unique by simply completing the cylinders to the least cr-algebra. With this notion of path measure, one can use the PATTL logic introduced earlier to analyze the transient (state of the system at a specific time instant) behaviour of systems, as it was in the previous sections. Moreover, to analyze the steady-state (in the long-run) behaviour of the system, the CSL logic [ASSB96, BKH99] introduced a steady-state operator y , with syntax y^op).  The  semantics of the steady-state operator is defined as follows  (8.15) v'\=ip  As one can see from Equation 8.15, in order to model check the steady-state operator formula, we must compute the steady-state probabilities TT (V') for all states v and v'. These probabilities are v  independent of the initial state if the system is ergodic. Otherwise, the solution is more involved as it necessitate the computation of bottom strongly connected components. It is a standard result [Ste94] that steady state probabilities can be obtained by solving a system of linear equations:  fr • Q = 0 and  (8.16) v'eA  163  where Q is called the generator matrix and is obtained via the following equation:  (8.17) Equation 8.16 can be solved using standard iterative or direct method [Ste94]. Let us now turn to model checking the PATTL formula 3?Qp(ip\U ip2), for finite values of r. T  The path measure can be obtained via a technique called uniformization, which is also known as Jensen's method [BHHKOO]. This method proceeds to the transformation of the original CTMC (which our P C N is equivalent to in this case) to a uniformized discrete-time Markov chain with transition matrix P, yielding an infinite summation to computer the transient probability vector ir  st  obtained via: 00  (8.18) where 7^.4 = e~  qt  • (q • t) /i\ denotes the ith Poisson probability with parameter q • t. l  The complexity of PATTL augmented with the steady-state operator model checking has been showed to be linear in the size of the formula, polynomial in the state space, linear in the maximum time threshold in the formula, and linear in the largest entry of the generator matrix Q [BHHKOO].  8.7  First Order PCN timed temporal logic (FPATTL)  We can define a first-order PCN timed temporal logic (FPATTL). The syntax of FPATTL is obtained by combining the rules of the propositional PATTL logic with a multi sorted first-order language. That is, in addition to atomic propositions, logical connectives, temporal and probabilistic operators, we now also have predicates, functions, individual constants and individual variables interpreted over an appropriate domain. Intuitively, the structure on which the propositional language is used is extended in such a way that each state is associated with an interpretation of local and global symbols. The semantics is obtained via the usual (Tarskian) definition of truth while the validity and satisfaction are defined in the usual way. Here, we present briefly the details of the technical formulation of FPATTL, which extend those introduced in [Eme90, Zha94] for a non-probabilistic linear propositional logic.  164  In order to define a first order language, we need to define the notion of terms which relies on the concepts of signature and E-domain structure defined in Chapter 2. In addition to the notions of signature and E-domain, we also need to specify a set of global variables on which the first-order quantifiers will apply, and a set of local variables. In the PCN framework, global variables, X , will g  be represented by parameter variables while local variables, Xi, will be trace variables of the model. The set of global and local variable will add up to the set of S-sorted variables X = X / U X . The g  set of terms of sort s £ S induced by E and X, denoted by T ( E , X) , is the least set of strings that 3  satisfies one of the following: • if x £ X , t h e n x £ T ( E , X ) , s  S  • if x £ Xi n X , then pre(x), x - r <E T ( E , X) s  s  for r > 0,  • if / £ F with type -> s, then / £ T(E, X ) , s  • if / € F with type s* -» s where s* : I -* S, then / ( T ) e T ( E , X ) where T : I —> s  T ( E , X ) with Tj e T ( E , X ) , . . Given E = (S, F) as a signature, let $ be a set of S-sorted predicate symbols, such that for each p e $ , the type of p is a tuple s*": I —> S. The syntax of FPATTL can be defined given E and  Definition 8.12 (Syntax of FPATTL) The basic syntax of FPATTL can be defined as:  (p ::= true | T = T | p(T) | -><p | <p A<p I V i ^ x  s  2  x  s  2  <P2  \ <fii S <p I <Pi U <p \ <pi S <p \ &> (ip) \ T  2  T  2  2  Qp  where T G T(E, X) is a term of sort s, p £ $ is a predicate symbol with type s* : I —> S, s  s  T : I —* T(E, X ) wifA Tj € T(E, X ) . and x G X . s  g  For defining the quantifiers in FPATTL, we adopt a rather different approach in that instead of specifying the syntax and semantics for the Universal (Vx £ X ) and Existential (3x £ g  X) g  quantifiers, we generalize those notions to introduce a Probabilistic quantifier (&Qp{<p), x £ X ), g  i.e., (p is quantified over global variables. 165  To properly define this quantifier, we need to introduce the notion of prior on the domain of the global variables x G X . Therefore, within FPATTL, we will assume the existence of a prior g  probability distribution T : A —> [0,1], a; G X , with the restriction that 2 ^ e A ^x(a) = 1- Intux  g  a  itively, when quantifying over a global variable x, we assume that T is a well-defined probability x  distribution (a prior) that assess a probability to each value that x can take within its domain. A frame of FPATTL over a PCN structure X is a tuple (T, A,V , X  structure, A is a S-domain structure, V  is the probability distribution (possibly parameterized)  x  induced by  V) where T is a time  and V is an interpretation that assigns to each predicate symbol p £ $ a subset V(p)  of xiA *, given that the type of p is s* : I —> S. s  A model of FPATTL is a pair (F, a) where F = (T, A, V *, V) is a frame and a = (cri,o ) is 1  g  a valuation for X = X ; U X , i.e., <r : X —> A and <TJ : X ; —> A " , where A 7  s  fl  T  9  is the trace space  of the system. By extending the valuation a from variables to terms, we have a : T(F, X) —> A , T  such that for any < € T : • cr(x)(i) = c (a;) for any x G X , 3  f l  • o-{x)(t) = ai(x)(t),a(pre(x))(t) = o"j(x)(pre(i)),cr(x - r)(t) = <J;(a;)(t - r) for any z G Xj,  • a(/(T))(i) = f (cr{T)(t)) for any / G F . A  Definition 8.13 (Semantics of FPATTL) Let F = (T, A, V) be a frame and (F, a) be a model of FPATTL. Let F be an FPATTL formula, a \=t F denotes that a satisfies F at time t: a \= true  Vi G T.  t  # V v i G aCZ; ),^ G a(T2), (i) = ^ ( i ) . 1  Wl  a\= p(T),pe^  ' J V u G a(T),v(t) G V(p).  t  'Jo- h t fiff°~ h t O- h t  U <P2  0-\=tf\S <p  « " ^ C T h t <P2-  #f 3 i ' > i , a h t ' <^2 andVt", t < t" < t', a h t " ¥>i; J 3 i ' <t,a h t ' </>2 andVt",t' < t" <t,a h t "  2  a h t F<piU L? T  2  # 3 i ' G Tt+ , IT h t ' <^2 and\/t",t < t" < t',a h t " <PiT  166  a H V\S f2  iff3t' e T -r, <r H <P2 and V i " , i ' < i " < i , a h " P i -  ffK^QpW  iffe<r\v\=t1>})  r  t  a \=t ^Qptf, x £ X (1 Xg  OP-  iff there is a value a in A , such that we have  s  s  (E  a £  A  s  r (a)-((r \=t<p[a/x]))Qp, x  where (p[a/x] stands for substitution of x in ip by a. Note that the usual quantifiers (Bx, Vx) are represented by the respective special instances of our probabilistic operator: 3x = <^> and Vx = 5 > . Moreover, various logical connectives, 2>  0  1  temporal and real-time operators can be defined as for PATTL. As it is usually the case, we will use o \= F to denote that o satisfies F initially, i.e., o [=o F. F is valid over a frame F, iff for any model (F, o), o \= F. F is valid, iff any frame F, F is valid over F. F is satisfiable over a frame F, iff for some model (F, o), o (= F. F is satisfiable, iff for some frame F, F is satisfiable over F.  Example 8.3 Consider the property, as introduced before, of an elevator system to deliver any new passenger to its destination within a certain amount of time r = 30, with at least probability p = 0.95: n(E —> <^>o.95(in<e U D)). Obviously, the satisfaction of this formula depends on 30  the rate of arrivals of news passenger, on the number offloorsof the building being deserved by the elevator and on the velocity of the elevator. Let R be the domain for the parameter of the probability distribution modeling the rate r (r G X ,i.e., r is a global variable in the PCN model) of g  the passengers' arrival. Then the FPATTLformula  (U(E —> ^^(true U D))) will be true if r  the measure of all possible parameter values in Rfor which the formula is true is greater or equal to the threshold p. • The verification offirst-orderformulae in FPATTL is closely analogous to that of PATTL formulae with the exception that one has to be careful when defining the prior probability distribution over uncountable domains. The technical issues arising in this case will not be addressed in this dissertation. Another issue that is important to note is the notion of open state specification. Zhang ([Zha94]) showed that the openness of state formulae (formulae without temporal and real-time operators) was important for requirements specification. For example, consider the safety constraint n-np{x). This formula stipulates that the system, in order to satisfy the behavioural constraint requirement, should 167  never satisfy the predicate tp(x), no matter the value of x. However, if the predicate ip is open, it means that -up is closed and hence an undefined value of x would satisfy the property. This can be extremely problematic, especially for critical constraints on the behaviour such as safety. For more details on this topic, the reader is referred to the aforementioned reference.  168  Chapter 9  Control Synthesis 9.1  Introduction  In this chapter, we will briefly review the concept of control synthesis within the PCN framework. Similarly to the verification problem discussed earlier in this dissertation, the task of control synthesis is extremely complex. In fact, there does not exist a unique algorithm that can be used to synthesize controllers in all classes of systems. Hence, various methods have been developed over the years, each of which are specifically designed to handle a certain type of control problem. Due to the complexity of the problem, the field of control synthesis is a wide area that includes among others, optimal control, stochastic control, robust control and also planning. We do not intend to cover all those areas here, but rather we wish to introduce the reader to specific techniques that are especially well suited to synthesize controllers when in the presence of a PCN model. As described earlier, we view a robotic system as a combination of a body, a controller and an environment (Figure 1.1). We defined the behavior of such a robotic system as the set of observable robot/environment traces of the system. Moreover, we introduced the notion of requirements specification as the subset of all possible traces which satisfy a given property. Therefore, within the PCN framework, the problem of control synthesis can be described as follows: given a requirements specification 71, the model of the body BODY and the model of the environment ENVIRONMENT,  synthesize a model of the controller CONTROLLER,  such that the be-  haviour of the resulting system satisfies the requirement, i.e.,  [X = BODY(U,Y),  U = CONTROLLER^,Y), 169  Y = ENVIRON'MENT(X)}  \= Tl.  Historically, planning and control have been studied as different problems. The planning problem [DW91] is denned as using a model to formulate sequences of actions to achieve a certain goal. The control problem [DW91] is considered as finding a policy to achieve a goal or minimizing a functional. Planning is normally restricted to symbolic domains in discrete time; while control is often for numerical domains, particularly n-dimensional Euclidean spaces, in either discrete or continuous time, both of which can be handled under the umbrella of the PCN framework. The solution to a planning problem (traditionally) is a trace (sequence) of inputs to a system for approaching a final goal; the solution to a control problem (closed-loop control) is a transduction from the sensor traces to the command traces for minimizing a required functional, such as time, energy, cost for approaching a goal. Common techniques for planning include search algorithms and theorem proving while calculus of variations and optimization have been developed for control. It is to be noted that even though in general the notions of control and planning are often considered as different problems, they can be seen as specific instances of our control synthesis formalism. Examples of typical planning and control problems include respectively, approaching a final goal and minimizing a global function over time (for example, energy). Both problems can be seen as setting constraints over the possible traces of the system, which clearly relates the notion of control with that of verification. There are many types of control synthesis problems, depending on the information that is available to the decision maker. These types can be classified in three main categories: 1. Open-loop: only the initial state of the system X  to  2. Feedback: both the initial state X  to  is known.  and the current state X are known. t  3. Closed-loop: the whole trajectory of the system is known, i.e., {X }, to < s <t. s  It is clear that open-loop controllers form a subset of feedback controllers while feedback controllers are a subset of closed-loop controllers. However, for Markovian systems, feedback control and closed-loop control are in fact equivalent since the state of the system is completely determined by the previous state. In this chapter, we will restrict ourselves to feedback control as most systems of interest possess the Markovian property. Before formally introducing the notions of stochastic and robust control, let us enumerate some standard definitions of stability in the sense of Lyapunov. 170  Equilibrium point: A state x is an equilibrium point (state) of the system x = f(x, u, t) if e  x(t) = x then the system stays in x for all time. This essentially means that f(x , u, to) = e  e  e  0 , V i > t. o  Stability: This property denotes a system that remains close to an equilibrium given that the initial condition was reasonably close to the equilibrium in the first place. Formally, the equilibrium point x  e  = 0 is said to be stable if, for any e > 0, there exists a constant  5(to, e) > 0 such  xt \\ < S(t ,e) 0  \\x(t)\\ <e,\/t>t .  0  0  Otherwise the system is deemed unstable. Stability is a very important concept since trying to control an unstable system is useless.  x = 0 is said to be attractive at time to if, for some 6 > 0  Attraction: The equilibrium point  e  and each e > 0, there exists a finite time interval T(to, 5, e) such that  l^tol <  ||zt|| < e,Vt > t + T(t ,5,e).  <5  0  0  Asymptotic Stability: The equilibrium point x = 0 is asymptotically stable if (1) it is stable e  at time to, and (2) it is attractive, or equivalently, there exists 6 > 0 such that  ||x || < S => ||a; || -» x as t —> oo. to  t  Exponential Attraction and Stability: x  e  — 0 is exponentially attractive at time to if, for  e  some 5 > 0, there exist two strictly positive numbers a (6) and f3 such that  ,-/3(t-t ) t \\ < =^ \\ t\\ < a(5)exp  x  5  x  0  0  The equilibrium point is said to be exponentially stable if, for some 6 > 0, there exists constants a > 0 and /3 > 0 such that  171  e  e'  e  (a)  (b)  (c)  Figure 9.1: Comparing a) stability, b) attraction, and c) asymptotic stability. The above definitions are presented for systems whose equilibrium points are located at the origin. However, they can easily be extended to systems with a known but nonzero equilibrium. The concepts of uniform stability, uniform attraction and uniform asymptotic stability imply that stability and performance properties of many systems are independent of the initial time to- Figure 9.1 compares the concepts of stability, attraction and asymptotic stability for a two-dimensional system.  9.2  Stochastic and Robust Control  As mentioned above, the field of control is vast and has evolved considerably since its origin which dates all the way back to 300 B C with the preoccupation of the Greeks and Arabs with keeping accurate track of time. For an interesting introduction to the origin of automatic control, the reader should consult §1: Introduction to Modern Control Theory of [Lew92]. Obviously, we cannot cover most of the control theory field in this dissertation. Therefore, we will simply provide a brief introduction to two areas of control theory: stochastic control and robust control, with an emphasis on robust control. We will then illustrate the use of robust control on a simple package delivery robotic system. In stochastic control the uncertainties in the system are modeled as probability distributions. One desires to combine these distributions to yield the control law. Stochastic control deals essentially with the expected value of control: controlling the system on average, based on the stochastic nature of the uncertainty. However, large deviations can occur and thus move the system away from its (average case) optimum. Hence, this type of control might not be acceptable for embedded con172  trol systems that have safety implications: when we want to have probability one that one event will not occur. In general, stochastic control is interested in minimizing a cost function. The most common stochastic control techniques include Linear Quadratic Gaussian (LQG), Quadratic Stability and Hamilton-Jacobi-Isaacs Equations. An introduction to stochastic control can be found in [Lew86]. Robust control involves a method of measuring the performance changes of a control system in the presence of changing system parameters. The main concern of robust control is uncertainty and how the control system can deal with the presence of uncertainty. Unlike stochastic control, robust control seeks to bound the uncertainty rather than express it in the form of a distribution. Given a bound on the uncertainty, the control can deliver results that meet the control system requirements in all cases. This can be seen as worst-case control. Note that some performance may be lost to guarantee the system meets some requirements. It has been shown that controller arising from robust control can be more conservative than stochastic controllers. However, when one cannot accept the system entering certain states (as in safety critical systems), loss of performance may be a small price to pay. The field of robust control has yielded many different techniques beyond the scope of this dissertation. Here, we simply attempt to catalog the major ones and briefly describe the basic concepts behind each techniques. The reader is referred to the specific citations for more detail on a particular technique. • Adaptive Control: An adaptive control system sets up an observer for each significant state variable. The system can adjust each observer to account for time varying parameters of the system. The output is to be brought closer to a desired input while at the same time the systems continues to learn about changes in the system parameters. This method may suffer from convergence problems. [K.J96]. • H and H°°: Hankel norms are used to measure control system properties. A norm is an 2  abstraction of the concept of length. Both these techniques were originally developed for the frequency domain but have been recently extended for the state space domain [PUSOO]. H  2  control seeks to bound the power gain of the system while H°° control seeks to bound the energy gain of the system. Gains in power or energy in systems indicate operation of the  173  system near a pole in a transfer function which is an unstable situation. For more on these techniques, consult [Cha96]. • Parameter Estimation: This method establishes boundaries in the frequency domain that cannot be crossed to maintain stability. These boundaries are evaluated by given uncertainty vectors. This technique is similar in essence to the root locus method [Eva54] in that it is also a graphical method which observes the movement of the system. A detailed treatment of the parameter estimation techniques can be found in [Ack93]. • Lyapunov: This is possibly the only universal technique for assessing nonlinear systems. Lyapunov techniques focus on stability. One constructs Lyapunov functions, which can be seen as energy functions that model the behavior of the system. Then one evaluates these functions along the trajectory to ensure that the first derivative is always dissipative in energy. A gain in energy means that the system is near a pole and will therefore be unstable. Qu describes these methods in detail [Qu98] and focuses on the Lyapunov first and second methods for analyzing the stability of systems.  9.2.1  Robust Control of Package Delivery Robot  In this section, we present an example illustrating the use of control synthesis on a system, encompassing multiple sources of uncertainty, acting on a continuous time structure. Suppose that the system to be controlled consists of a robot pushing a package to a desired location as shown in Figure 9.2. The arms of the robotic agent are built on a spring structure. The spring constant A: has a nominal value of ko = 1.25, but may vary and is considered uncertain: k e [0.5,2.0]. Furthermore, there is an external disturbance, u>(t), affecting the dynamics of the package as it is pushed by the agent. This disturbance is modeled as a Wiener process with identity covariance. For simplicity, we assume that both the robot and the package have the same mass m  r  = m  p  = 1. Due to the  uncertain nature of the spring constant along with the stochastic nature of the external disturbance on the package, we model the system as a stochastic uncertain system (see [PUS00] for a thorough description of stochastic uncertain systems). Then the system can be described by the following set of equations:  174  Environmental Disturbances  Figure 9.2: Package delivery robotic system.  x = (A + F{k{t) - k )S)x + Bm + b u{t); 0  z = Cx  2  4-  Du  where z is called the uncertainty output of the system, x = [x\ x xi x }' €E R is the state 4  2  2  space, u is the control force, and 0  0  1  0  0  0  0  1  0 0 ;  -k  0  ko  fco  -ko  0  B  X  =  0 ;  B  2  =  0  0  0  l  0  0  1  0  1  0  0  0  0  1 0  0  0  0  0  0  175  0  ; D =  0  I  0  F =  0 -1  5 =  1 - 1 0  0  1 It has been shown that a controller corresponding to the nominal value k = ko can be found that guarantees a H°° norm bound with 7 = 2.5 for a closed-loop nominal transfer function [PUS00]. However, this norm bound holds only for the aforementioned nominal value of k and is not robust for variations in k. Suppose instead that a satisfactory controller must guarantee a disturbance attenuance bounds 7 = 2 robustly with respect to the variations in the value of k. Define the following: A ( i ) = k(t) — ko. Assume that A(£) is a Gaussian white noise process with zero mean and variance o . 2  It is easy to choose the value for the parameter a such that k will fall within the bounds [0.5, 2.0] with high probability. For instance, a selection of a = 0.25 with yield P(\k(t) — ko\ < 0.75) > 0.997. Obviously, this probability will increase as a  2  —> 0. Applying the result of Theorem 3  from [Ugr98], which involves solving algebraic Riccati equations that can be solved by homotopy methods [RHP93b], along with an infinite sequence of Lyapunov equations [GuoOl], we obtain the controller u = Kx, with K = [0.8547 - 3.7458 - 3.3103 - 3.1986]. It is possible to show that with this controller, the robotic package delivery system is exponentially stable. Figure 9.3 shows one realization of the system. In this example, the package needs to be moved by one unit forward. We can observe that the agent overshoot the target location a little before stabilizing around the desired target.  9.3  Planning under uncertainty  Coined as planning under uncertainty, decision-theoretic planning (DTP) has recently drawn a considerable amount of attention among the AI community. Basically, the problems which are of interest to decision-theoretic planners are those involving systems whose dynamics can be modeled as stochastic processes and where an agent, acting as a decision maker, can influence the system's behavior by performing (uncertain) actions. Resulting from the Markov property, the current state of the system and the choice of the action by the agent jointly determine a probability distribution over the possible next states. It is usually assumed that systems evolve in stages, where actions cause 176  Figure 9.3: One realization of the package delivery system. transitions from stage t to t +1. This progression through stages is analogous to the passage in time, with one clock for the whole system, if one assumes that every action takes unit time to complete. In general, it is also assumed that transition probabilities are stationary, although the results can easily be extended to the non-stationary case. The agent is assumed to prefer to be in a certain subset of the system's states, whose elements are often called the goal states. Therefore, one would like to construct a plan of action which dictates the action that is most likely to bring the agent to these goal states. Such a plan is called a policy and finding it is the goal of DTP. The representation used as an underlying model for such decision-theoretic problems is very often a Markov Decision Process. The reader should note that it is often the case that the agent does not know the exact state of the world. Hence, when it needs to choose an optimal action, the agent is forced to make decisions based on a probabilistic estimate of the current state of the system. Such problems have been modeled with a Partially Observable Markov Decision Process (POMDP), which can be seen as a fully observable MDP with the addition of an infinite state space consisting of probability distributions over the state,  177  each distribution representing the agent's belief of where it is at any point in time. For an in depth look at the theory of MDPs or for a survey on decision-theoretic planning (including POMDPs), the reader is referred to [Put94] and [BDH99] respectively. In the following section, we will briefly introduce the notion of (PO)MDPs. We will carry on to present how (PO)MDPs can be seen as a simple case of a PCN model and how the decision-theoretic algorithms can be used to generate a PCN controller.  9.4  Introduction to Markov Decision Processes  A Markov decision process (or sometimes referred to as a fully observable MDP) is defined by the tuple (S, A, P, R), where S is a finite set of states of the system, and where states are defined as a description (more or less precise) of the system at any point in time. In a MDP, these states can be exactly identified by the agent, i.e., at any given time the agent knows exactly which state it is in. A is a finite set of actions from which the agent can choose; P is the state transition model of the system which is a function mapping from elements of S x A into discrete probability distributions over S; and R is a stationary reward function mapping from S x A to E . R(s, a) specifies the immediate reward gained by the agent for taking action a in state s. Actions induce stochastic transitions, with P(s, a, t) denoting the probability with which state t is reached when, at the previous time step, action a is performed at state s. It is to be noted that the transitions of the model specify the resulting next state using only the state and action at the previous time step. This therefore assumes that the next state is solely determined by the current state and the current action and corresponds to the Markov assumption discussed earlier. It is worth mentioning that not all systems are Markovian in nature. The Markov assumption is merely a property of a particular model of that system, not of the system itself. However, one should note that the Markovian assumption is not too restrictive, since any non-Markovian model of a system can be converted to an equivalent Markov model. In the field of control theory, this conversion is referred to as the conversion to state form [Lue79]. A stationary policy it : S —> A describes a particular, time independent, course of action to be adopted by an agent, with n(s) denoting the action to be taken in state s. It is often assumed that the agent acts indefinitely (an infinite horizon) but the finite horizon case has also been studied extensively. In the finite-horizon case however, the optimal policy is typically non-stationary: the 178  agent's choice of action on the last step of his life will generally be very different than when it has a long life ahead of it. We will, in this short presentation of the MDP framework, assume infinite horizon, unless explicitly stated. A possible way to assess the quality of different policies is to adopt an expected total discounted reward as the optimality criterion wherein future rewards are discounted at a rate 0 < B < 1, and the value of a policy is given by the expected total discounted reward accrued. The expected value V^-(s) of a policy n at a given state s satisfies [Put94]:  V (s) = R(s, TT(S)) n  +  8 Y, P(s, 7r(s), t) • V„(t)  (9.1)  tes  A policy 7r is said optimal if V > V > for all s S S and policies TT\ The optimal value function n  n  V* is the value of any optimal policy. There exists many iterative algorithms for constructing optimal policies; the most popular ones are value iteration and policy iteration. For the purpose of this discussion, we will only briefly present the simplest one: value iteration. Value iteration [Bel57] is a simple iterative approximation algorithm for constructing optimal policies. It proceeds by constructing a series of n-stage-to-go value functions V . Setting V° = R, n  we define  V {s) = max J R(s, a) + 8 V Pr(s, a, t) • V (t) \ n+1  n  I The sequence of value functions V  n  tes  (9.2)  )  produced by value iteration converges linearly to the optimal  value function V*. For some finite n, the actions that maximize Equation 9.2 form an optimal policy, and V  n  approximates its value. A commonly used stopping criterion specifies termination  of the iteration procedure when _  <  (9.3)  (where \\X\\ = max{|:r| : x £ X} denotes the supremum norm). This ensures that the resulting value function V  n+1  is within | of the optimal function V* at any state, and that the resulting policy  is e-optimal [Put94].  179  PCN: Body + Environment  PCN: Controller  Figure 9.4: Similarities between MDP and PCN frameworks  9.5  MPDs correspondence to PCN  In the MDP setting, the agent behaves according to a policy that is computed off-line, typically by dynamic programming techniques such as those mentioned in Section 9.4. Once this policy is available, the agent simply transitions from one state to another, choosing at each time step the action that is specified by the policy. This is essentially a control problem where the policy represents a controller and the dynamics of the MDP (transition probabilities) denote the agent's behavior. In this view, one can see obvious similarities between a PCN model and a MDP. Figure 9.4 graphically represents this similarity. In the PCN framework one describes a robotic system as the coupling of a plant (robot's body), an environment and a controller. Applying this view to a MDP model, one can see that the M D P dynamics represent both the environment and the plant, while the policy represents the controller. In fact, in an MDP, the robot's body is included in the general (non-modular) description of the environment. This lack of modularity can significantly complicate the task of system designers. Indeed, modularity and hierarchy are two essential properties of a model for robotic systems, which PCN possesses. The reason why MDPs are so popular within the AI community is not mainly because of the modeling capabilities of the framework, but rather the algorithms with which one can compute optimal policies. As mentioned earlier, policies can be viewed as controllers; hence computing a policy can be seen as control synthesis. It would be extremely valuable to be able to merge the modeling simplicity and power of the PCN framework with the control synthesis capabilities of MDP. In this section, we will show that for a subclass of PCN models, which we call synchfin-PCN, there exists a one-to-one correspondence between the class of all MDPs and synchfin-PCNs. We 180  PCN class  PCN model  Figure 9.5: Control synthesis for the synchfin-PCN class will also show, for the synchfin-PCN class of models, how we can synthesize an optimal controller by taking advantage of the dynamic programming algorithms for MDPs. Figure 9.5 sketches out the algorithm for performing control synthesis on a synchfin-PCN model. The first step consists of translating the synchfin-PCN model into a MDP. Secondly, one performs off-line value or policy iteration on the resulting MDP to obtain a policy. Finally, the policy is converted into a controller for the initial synchfin-PCN model, thus synthesizing a controller. Before formally defining the synchfin-PCN class of models and the control synthesis algorithm, let us present an informal discussion on the correspondence between a PCN and a MDP model.  9.5.1  MDP to PCN Conversion  Within the MDP framework, one can use an explicit, or sometimes referred to as extensional, representation for the state space. In such a representation, states are enumerated explicitly, hence the designer has to describe the transition probabilities for every pair of states within the system. In this case, the transformation of an MDP into a PCN model is trivial: the state variable and the set of transition probability tables (one per action) can be seen as a single location with a generator, parameterized on the selected action and the current state of the system, which generates the next  181  Figure 9.6: PCN model of an extensional MDP (uncertain) state. Figure 9.6 displays a PCN model of a generic extensional MDP. However, for a problem with a state space <S such that |<S| = N is large, the task of specifying the transition probabilities can be overwhelming (0(N )) and quite impractical. A solution to this 2  problem is to use an intensional (or factored) representation rather than an extensional one. A n intensional representation is obtained by specifying a set of features that describes, at an acceptable level of detail, the state of a given dynamical system. Each feature takes a finite number of values (usually quite small) and an assignment of values to every feature of the system completely defines a state. Hence, the state space can be described as the cross product of all the features, and in general, it grows exponentially in the number of features used to described the system. Another important 1  advantage of adopting a factored representation for the state space within the MDP framework is that it allows one to also use a factored representation of actions, rewards and other components of an MDP. In such a representation, one describes the effects of an action on specific features rather than on the entire state of the system, often leading to a considerably more compact representation. Bayesian networks (BN) [Pea88] are graphical models used to represent causal and probabilis2  tic processes. In the recent years, BNs have become the tool of choice for probabilistic and statistical modeling. Moreover, they have become the norm for representing probability distributions in factored form as not only do BNs provide convenient means of expressing complex assumptions and relationships between variables but they also greatly facilitate an economical representation of joint 'Note that a PCN model is by definition a factored representation as the state space corresponds to the cross product of the domains of the locations of the model. In AI, Bayesian networks are also referred to as belief nets 2  182  probability functions. Formally, a Bayesian network is represented by a directed acyclic graph (DAG). Each element in the set of vertices denotes a set of random variables and edges between two vertices represent a direct probabilistic dependency between the two random variables. Obviously the absence of edges between vertices also reflects implicit independencies among the variables. Once the structure of the graph is specified, one must then quantify the network by specifying, for each variable in the graph, a conditional probability table (CPT) where a probability is given for each variable conditioned on all possible values of its immediate parents in the B N . When a variable does not have any parents, an unconditional distribution, also known as a marginal distribution, is specified for the parentless variable. Not only is the structure of the B N displaying the independence assumptions of the variables, but it can be shown that the network defines a unique joint distribution over the variables constituting the global state space of the system. For a thorough introduction to the theory of Bayesian networks, we refer the reader to [Pea88]. Temporal Bayesian networks (also called dynamic Bayesian networks) are special cases of Bayesian networks where the vertices represent features of the systems at different time points and edges denote dependencies over time. For Markovian systems only 2 time points are of importance: t and t + 1. Hence, temporal Bayesian networks for such systems are called two-stage temporal Bayes nets (2TBN). In a 2TBN, the set of features (variables) is partitioned into 2 sets: the features at time t and the features at time t + 1. We call diachronic arcs the arcs displaying dependencies being variables at time t and t + 1 while synchronic arcs denote dependencies between variables at time t + 1. For a thorough discussion on dynamics Bayesian networks, the reader is referred to [Mur02]. Similarly to the extensional representation, the conversion of an intensional MDP into a PCN model is rather simple. Given the conditional probability tables from the MDP description, one can build exactly one generator per feature (variable) where the inputs of the generator are the selected action and the parents of the variable, as specified by the two-stage temporal Bayes Net (2TBN). By acting this way, one takes advantage of the factored representation of the conditional independence assumption. It allows one to build generators that are simpler, since the size of their set of input variables is much smaller then the total number of variables in the system. Figure 9.7 shows a general case for a PCN obtained from converting a factored MDP. Note that to avoid algebraic loops  183  Transition Probability  PtVjjtpatCvJ), Aj  Controler Policy K  Reward Transduction  -©  Transition Probability P^lparCv"), A  (  Figure 9.7: PCN model of an intensional (factored) MDP in the PCN model, the use of unit delays is required. This is only a side effect of the fact that the probability distribution of a variable at time t is parameterized on the value of the same variable at time t — 1. Therefore, it is easy to conclude that unit delays will appear for every diachronic arc in the 2TBN representation of the MDP. In the event where the action effect on variable V /  + 1  is correlated with variable V? , as repre+1  sented by a synchronic arc in the 2TBN, we can represent this in a PCN by simply making location Vj an input of the probabilistic transduction of variable V,, without the use of a unit delay. This parameterizes the distribution of Vi on not only the values of its set of parents at time t but also on the value of Vj at time t + 1. The independence of the variables at time t given the values of the parents at time t — 1 no longer holds when in the presence of synchronic arcs. The conventional dynamic programming techniques, as presented in the previous section, do not work with arbitrary Bayes Net action description (ones with both diachronic and synchronic arcs). Therefore, in order to be able to perform control synthesis, we might have to require that one only specifies actions without correlation. However, this is not desirable as it would seriously diminish the number of systems that can be represented. Indeed, correlated effects are common in most systems so it is essential that we allow them in our models. More specifically, correlated effects will arise very commonly in the 184  PCN framework, thus the inability to solve MDPs with synchronic arcs would render our control synthesis algorithm impractical as it would only apply to a very small class of PCN models. Techniques to alleviate this problem have been suggested: one consists of clustering all variables affected by the correlation into a compound variable with size exponential in the number of variables in the cluster (see [BDH99] for more detail). However, this technique not only requires a radical transformation of the problem (thus we would no longer be able to use the original state variables, which could significantly complicate the conversion of the optimal policy into a PCN controller) but can also cause a blowup in the size of the state space. Another technique, proposed in [Lit97], suggests the transformation of the action representations into a STRIPS representation. However, this solution once again necessitates a drastic transformation of the problem which, as previously discussed, is not desirable in the context of control synthesis for PCN. These two methods, while solving the problem of synchronic arcs, are not useful for our purpose. Remember that we are trying to transform a PCN model into a MDP so that we can perform value or policy iteration to obtain an optimal policy. Once this policy is obtained, we wish to convert it into a PCN controller transduction. Therefore, if in order to solve the MDP we have to dramatically transform its state space, we might not be able to convert the policy into a PCN controller relevant to the original state space. It is important that we maintain the same structure so that the conversion of the optimal policy into a PCN controller remains possible. Fortunately, [Bou97] improved the structured policy iteration algorithm of [BDG95] so that it would not be hindered by synchronic arcs. This generalized structured policy iteration algorithm is important here as it conserves the original problem description when performing decision theoretic regression in the presence of synchronic arcs. Therefore, based on this result, we can assume that synchronic arcs do not represent a limitation in the control synthesis procedure that we are about to describe.  Making an extensional representation intensional As we have just seen, an extensional representation of a system can be very impractical to deal with, both in terms of the size of the transition matrix and the lack of information provided by that representation. System features in an intensional representation provide the designer with information on the local behavior of the system. Extensional representations only displays global behavior.  185  One could then ask if given an extensional representation, can one automatically decompose it into a potentially simpler intensional representation? That is, are there n independent variables whose joint distribution is the same as the state transition probability distribution? Using a set of independent variables that represent the state space can reduce considerably the amount of work the designer needs to perform. Since the state space is the cross product of the variables that constitute it, far fewer probabilities need to be specified in the system. Furthermore, an intensional representation of the system can provide some interesting insight on the behavior of the system, thus simplifying the troubleshooting of the system when need be. Obviously, not every state transition probability distribution can be decomposed into n independent marginal distributions; for example, if the number of states in the state space, N, is a large prime number. Then by the prime factorization theorem, one will not be able to represent the state space with independent variables. Indeed, other than by using one single variable with N distinct values, one would have to use a representation that would create more states than in the state space. For example, a state space with 5 states cannot be exactly represented by any combination of independent variables. The closest one can get is to represent the state space with one binary variable and one three-value variable, thus yielding 6 states. Therefore, one extra state would be artificially created, a state which has no equivalent in the state space representation. One could associate a transition probability of 0 to the values of the variables associated with this state, such that the transition probability to this value, from any other value is 0 and the transition from this value to itself is 1. This technique would maintain the existence of a transition probability distribution but would introduce problems when trying to solve the system of equations. Therefore, to solve this problem, one needs to introduce dependence between the variables, and the hope is that the dependency structure will be simple to extract. More work is currently being done to find an algorithmic solution to this problem. When N is not a prime number, it is possible to exactly represent the state space with a combination of variables; and a simple test which provides information about the existence of marginal distribution of those variables can be performed. The test simply consists of assigning each state to a set of assignments to the n variables for which we desire to generate the marginals. Under the assumption of independence, we obtain a set of nonlinear equations (2" for binary variables) of the type P{Si\Si) = P(v\\v\) • • • P(V \V ) where the set of probabilities P(Si\Si) is given in n  n  186  the extensional representation. Furthermore, we have the following constraints: each row of each marginal distribution must be a probability distribution (sum to 1) and each individual probability must be between 0 and 1. One is thus left with a nonlinear programming problem to resolve. However, it is possible to avoid dealing with nonlinearity by simply using a log-transformation on the equalities and the range constraints. This transformation appears not to change the situation much since the constraints on the rows of the marginals then becomes nonlinear in the log of the probabilities. But we can ignore those constraints, solve the now linear problem, and once a solution is reached, simply normalize the results, hence producing a probability distribution. We can represent this linear programming problem with a linear system of the form Ax = B. With such a system, three possibilities exist for the solution x: 1. No solution exists 2. Exactly one solution exists 3. Infinitely many solutions exist The following table summarizes the solvability of a system of equations in terms of the rank of matrices A and B: The system Ax = b has ...  when ...  no solution  rank(yl) < rank([A6])  one solution  rank(A) = rank([A6]) = N  infinite number of solutions  rankG4) = rank([A6]) < N  If the ranks are such that there are one or more solutions, then the existence of n independent variables is guaranteed and one simply needs to solve to obtain the marginal distribution. When an infinite amount of solutions can be found, one needs to decide on which solution to choose. An algorithm based on the linear interior point solver [FGW02], which is a variant of Mehrotra's predictor-corrector algorithm (a primal-dual interior-point method) [Meh91] can be used to minimize a function defined over the set of all marginals. The sum of all marginals appears to be a suitable heuristic for that function. Let us present a simple example where we show how one can apply the test:  187  Example 9.1 (Extensional to intensional representation conversion) Let us define a state space  as S — {Si, S , S 3 , S 4 } , with its transition probabilities defined as follows: 2  Si  s2  s3  S4  Si  0.18  0.12  0.42  0.28  s2  0.255  0.045  0.595  0.105  S3  0.3  0.2  0.3  0.2  S4  0.425  0.075  0.425  0.075  Let us assume that we want to model the state space with n = 2 binary variables (Vi, V ). Let 2  us represent the state space in the following way: 51 = Vi A V  2  5  = ViA V  53  = Vi A V  5  = ViA V  2  4  2  2  2  The marginals probability distributions for the variables (e.g. P(Vi\Vi)) are unknown: our goal is tofindthe transition probabilities of each variable. If the variables were to be independent, one couldfindout their marginals by simply solving, for the (8 — 2 x 2 2) different marginals P(Vi\Vi), P(V \V ), P(V\|Vi), P(V \V ), 2  2  2  P{Vi\Vi), P(V \V ), the following systems of equations :  2  2  P(Si\Si)  2  =P(ViAV2|ViAVr2)  = P ( V i | V i A V ) • P(V \Vi A V ) 2  2  2  = P(Vi|Vi)-P(V |V ) 2  ^ISi)  2  = P ( V i A V \Vi A V ) 2  2  = P(Vi|Vi)-P(V |V ) >  2  =P(V\AV2|ViAV^)  P(S |S ) 4  2  4  = P(V\|V i)-P(V |V ) 7  ?  2  subject to the set of constraints:  188  2  (9.4)  P{V \V ) + P(V \V ) = 1 X  X  l  l  p<y \v{)+ P(V \V ) = 1 x  1  l  P(V \V ) + P(V \V ) = 1 2  2  2  (9.5)  2  P{V \V ) + P(V \V ) = 1 2  2  2  2  and 0  < P(Vi|Vi) < 1  0  < P(V \V{) < 1  0  < P{Vy\V{) < 1  0  < P(V \V ) < 1  0  < P(V \V ) < 1  0  < P(V \V ) < 1  0  < P(V \V ) < 1  0  < P(V \V ) < 1  X  X  2  2  2  2  X  (9.6)  2  2  2  2  By using the log transform, Equations 9.4 become linear and Constraints 9.6 now restrict the log of the marginals to a non-positive number. By ignoring Constraints 9.5, we can then solve this linear system, which has an infinite number of solutions. The solution that minimizes the sum of all marginal probabilities, along with the normalized values of the solution are displayed in the table below. The column Real Values represents the values with which the initial state transition probabilities were computed (reverse engineering). Hence, one can now model the state space system with 2 binary variabl