# Chapter 1. Probability Systems

## 1.1. Likelihood*

### Introduction

Probability models and techniques permeate many important areas of modern life. A variety of types of random processes, reliability models and techniques, and statistical considerations in experimental work play a significant role in engineering and the physical sciences. The solutions of management decision problems use as aids decision analysis, waiting line theory, inventory theory, time series, cost analysis under uncertainty — all rooted in applied probability theory. Methods of statistical analysis employ probability analysis as an underlying discipline.

Modern probability developments are increasingly sophisticated mathematically. To utilize these, the practitioner needs a sound conceptual basis which, fortunately, can be attained at a moderate level of mathematical sophistication. There is need to develop a feel for the structure of the underlying mathematical model, for the role of various types of assumptions, and for the principal strategies of problem formulation and solution.

Probability has roots that extend far back into antiquity. The notion of “chance” played a central role in the ubiquitous practice of gambling. But chance acts were often related to magic or religion. For example, there are numerous instances in the Hebrew Bible in which decisions were made “by lot” or some other chance mechanism, with the understanding that the outcome was determined by the will of God. In the New Testament, the book of Acts describes the selection of a successor to Judas Iscariot as one of “the Twelve.” Two names, Joseph Barsabbas and Matthias, were put forward. The group prayed, then drew lots, which fell on Matthias.

Early developments of probability as a mathematical discipline, freeing it from its religious and magical overtones, came as a response to questions about games of chance played repeatedly. The mathematical formulation owes much to the work of Pierre de Fermat and Blaise Pascal in the seventeenth century. The game is described in terms of a well defined trial (a play); the result of any trial is one of a specific set of distinguishable outcomes. Although the result of any play is not predictable, certain “statistical regularities” of results are observed. The possible results are described in ways that make each result seem equally likely. If there are N such possible “equally likely” results, each is assigned a probability 1/N.

The developers of mathematical probability also took cues from early work on the analysis of statistical data. The pioneering work of John Graunt in the seventeenth century was directed to the study of “vital statistics,” such as records of births, deaths, and various diseases. Graunt determined the fractions of people in London who died from various diseases during a period in the early seventeenth century. Some thirty years later, in 1693, Edmond Halley (for whom the comet is named) published the first life insurance tables. To apply these results, one considers the selection of a member of the population on a chance basis. One then assigns the probability that such a person will have a given disease. The trial here is the selection of a person, but the interest is in certain characteristics. We may speak of the event that the person selected will die of a certain disease– say “consumption.” Although it is a person who is selected, it is death from consumption which is of interest. Out of this statistical formulation came an interest not only in probabilities as fractions or relative frequencies but also in averages or expectatons. These averages play an essential role in modern probability.

We do not attempt to trace this history, which was long and halting, though marked by flashes of brilliance. Certain concepts and patterns which emerged from experience and intuition called for clarification. We move rather directly to the mathematical formulation (the “mathematical model”) which has most successfully captured these essential ideas. This is the model, rooted in the mathematical system known as measure theory, is called the Kolmogorov model, after the brilliant Russian mathematician A.N. Kolmogorov (1903-1987). Kolmogorov succeeded in bringing together various developments begun at the turn of the century, principally in the work of E. Borel and H. Lebesgue on measure theory. Kolmogorov published his epochal work in German in 1933. It was translated into English and published in 1956 by Chelsea Publishing Company.

### Outcomes and events

Probability applies to situations in which there is a well defined trial whose possible outcomes are found among those in a given basic set. The following are typical.

• A pair of dice is rolled; the outcome is viewed in terms of the numbers of spots appearing on the top faces of the two dice. If the outcome is viewed as an ordered pair, there are thirty six equally likely outcomes. If the outcome is characterized by the total number of spots on the two die, then there are eleven possible outcomes (not equally likely).

• A poll of a voting population is taken. Outcomes are characterized by responses to a question. For example, the responses may be categorized as positive (or favorable), negative (or unfavorable), or uncertain (or no opinion).

• A measurement is made. The outcome is described by a number representing the magnitude of the quantity in appropriate units. In some cases, the possible values fall among a finite set of integers. In other cases, the possible values may be any real number (usually in some specified interval).

• Much more sophisticated notions of outcomes are encountered in modern theory. For example, in communication or control theory, a communication system experiences only one signal stream in its life. But a communication system is not designed for a single signal stream. It is designed for one of an infinite set of possible signals. The likelihood of encountering a certain kind of signal is important in the design. Such signals constitute a subset of the larger set of all possible signals.

These considerations show that our probability model must deal with

• A trial which results in (selects) an outcome from a set of conceptually possible outcomes. The trial is not successfully completed until one of the outcomes is realized.

• Associated with each outcome is a certain characteristic (or combination of characteristics) pertinent to the problem at hand. In polling for political opinions, it is a person who is selected. That person has many features and characteristics (race, age, gender, occupation, religious preference, preferences for food, etc.). But the primary feature, which characterizes the outcome, is the political opinion on the question asked. Of course, some of the other features may be of interest for analysis of the poll.

Inherent in informal thought, as well as in precise analysis, is the notion of an event to which a probability may be assigned as a measure of the likelihood the event will occur on any trial. A successful mathematical model must formulate these notions with precision. An event is identified in terms of the characteristic of the outcome observed. The event “a favorable response” to a polling question occurs if the outcome observed has that characteristic; i.e., iff (if and only if) the respondent replies in the affirmative. A hand of five cards is drawn. The event “one or more aces” occurs iff the hand actually drawn has at least one ace. If that same hand has two cards of the suit of clubs, then the event “two clubs” has occurred. These considerations lead to the following definition.

Definition. The event determined by some characteristic of the possible outcomes is the set of those outcomes having this characteristic. The event occurs iff the outcome of the trial is a member of that set (i.e., has the characteristic determining the event).

• The event of throwing a “seven” with a pair of dice (which we call the event SEVEN) consists of the set of those possible outcomes with a total of seven spots turned up. The event SEVEN occurs iff the outcome is one of those combinations with a total of seven spots (i.e., belongs to the event SEVEN). This could be represented as follows. Suppose the two dice are distinguished (say by color) and a picture is taken of each of the thirty six possible combinations. On the back of each picture, write the number of spots. Now the event SEVEN consists of the set of all those pictures with seven on the back. Throwing the dice is equivalent to selecting randomly one of the thirty six pictures. The event SEVEN occurs iff the picture selected is one of the set of those pictures with seven on the back.

• Observing for a very long (theoretically infinite) time the signal passing through a communication channel is equivalent to selecting one of the conceptually possible signals. Now such signals have many characteristics: the maximum peak value, the frequency spectrum, the degree of differentibility, the average value over a given time period, etc. If the signal has a peak absolute value less than ten volts, a frequency spectrum essentially limited from 60 herz to 10,000 herz, with peak rate of change 10,000 volts per second, then it is one of the set of signals with those characteristics. The event "the signal has these characteristics" has occured. This set (event) consists of an uncountable infinity of such signals.

One of the advantages of this formulation of an event as a subset of the basic set of possible outcomes is that we can use elementary set theory as an aid to formulation. And tools, such as Venn diagrams and indicator functions for studying event combinations, provide powerful aids to establishing and visualizing relationships between events. We formalize these ideas as follows:

• Let Ω be the set of all possible outcomes of the basic trial or experiment. We call this the basic space or the sure event, since if the trial is carried out successfully the outcome will be in Ω ; hence, the event Ω is sure to occur on any trial. We must specify unambiguously what outcomes are “possible.” In flipping a coin, the only accepted outcomes are “heads” and “tails.” Should the coin stand on its edge, say by leaning against a wall, we would ordinarily consider that to be the result of an improper trial.

• As we note above, each outcome may have several characteristics which are the basis for describing events. Suppose we are drawing a single card from an ordinary deck of playing cards. Each card is characterized by a “face value” (two through ten, jack, queen, king, ace) and a “suit” (clubs, hearts, diamonds, spades). An ace is drawn (the event ACE occurs) iff the outcome (card) belongs to the set (event) of four cards with ace as face value. A heart is drawn iff the card belongs to the set of thirteen cards with heart as suit. Now it may be desirable to specify events which involve various logical combinations of the characteristics. Thus, we may be interested in the event the face value is jack or king and the suit is heart or spade. The set for jack or king is represented by the union JK and the set for heart or spade is the union HS. The occurrence of both conditions means the outcome is in the intersection (common part) designated by . Thus the event referred to is

(1.1)E=(JK)∩(HS)

The notation of set theory thus makes possible a precise formulation of the event E .

• Sometimes we are interested in the situation in which the outcome does not have one of the characteristics. Thus the set of cards which does not have suit heart is the set of all those outcomes not in event H . In set theory, this is the complementary set (event) Hc .

• Events are mutually exclusive iff not more than one can occur on any trial. This is the condition that the sets representing the events are disjoint (i.e., have no members in common).

• The notion of the impossible event is useful. The impossible event is, in set terminology, the empty set . Event cannot occur, since it has no members (contains no outcomes). One use of is to provide a simple way of indicating that two sets are mutually exclusive. To say AB= (here we use the alternate AB for AB) is to assert that events A and B have no outcome in common, hence cannot both occur on any given trial.

• Set inclusion provides a convenient way to designate the fact that event A implies event B , in the sense that the occurrence of A requires the occurrence of B . The set relation AB signifies that every element (outcome) in A is also in B . If a trial results in an outcome in A (event A occurs), then that outcome is also in B (so that event B has occurred).

The language and notaton of sets provide a precise language and notation for events and their combinations. We collect below some useful facts about logical (often called Boolean) combinations of events (as sets). The notion of Boolean combinations may be applied to arbitrary classes of sets. For this reason, it is sometimes useful to use an index set to designate membership. We say the index J is countable if it is finite or countably infinite; otherwise it is uncountable. In the following it may be arbitrary.

(1.2)

For example, if then is the class , and

(1.3)

If then is the sequence . and

(1.4)

If event E is the union of a class of events, then event E occurs iff at least one event in the class occurs. If F is the intersection of a class of events, then event F occurs iff all events in the class occur on the trial.

The role of disjoint unions is so important in probability that it is useful to have a symbol indicating the union of a disjoint class. We use the big V to indicate that the sets combined in the union are disjoint. Thus, for example, we write

(1.5)
Example 1.1Events derived from a class

Consider the class of events. Let Ak be the event that exactly k occur on a trial and Bk be the event that k or more occur on a trial. Then

(1.6)

The unions are disjoint since each pair of terms has Ei in one and Eic in the other, for at least one i. Now the Bk can be expressed in terms of the Ak. For example

(1.7) B2 = A2A3

The union in this expression for B2 is disjoint since we cannot have exactly two of the Ei occur and exactly three of them occur on the same trial. We may express B2 directly in terms of the Ei as follows:

(1.8) B2 = E1 E2E1 E3E2 E3

Here the union is not disjoint, in general. However, if one pair, say is disjoint, then E1E3= and the pair is disjoint (draw a Venn diagram). Suppose C is the event the first two occur or the last two occur but no other combination. Then

(1.9) C = E1 E2 E3cE1c E2 E3

Let D be the event that one or three of the events occur.

(1.10) D = A1A3 = E1 E2c E3cE1c E2 E3cE1c E2c E3E1 E2 E3

Two important patterns in set theory known as DeMorgan's rules are useful in the handling of events. For an arbitrary class of events,

(1.11)

An outcome is not in the union (i.e., not in at least one) of the Ai iff it fails to be in all Ai, and it is not in the intersection (i.e. not in all) iff it fails to be in at least one of the Ai.

Example 1.2Continuation of Example 1.1

Express the event of no more than one occurrence of the events in as B2c.

(1.12)

The last expression shows that not more than one of the Ei occurs iff at least two of them fail to occur.

## 1.2. Probability Systems*

### Probability measures

In the module "Likelihood" we introduce the notion of a basic space Ω of all possible outcomes of a trial or experiment, events as subsets of the basic space determined by appropriate characteristics of the outcomes, and logical or Boolean combinations of the events (unions, intersections, and complements) corresponding to logical combinations of the defining characteristics.

Occurrence or nonoccurrence of an event is determined by characteristics or attributes of the outcome observed on a trial. Performing the trial is visualized as selecting an outcome from the basic set. An event occurs whenever the selected outcome is a member of the subset representing the event. As described so far, the selection process could be quite deliberate, with a prescribed outcome, or it could involve the uncertainties associated with “chance.” Probability enters the picture only in the latter situation. Before the trial is performed, there is uncertainty about which of these latent possibilities will be realized. Probability traditionally is a number assigned to an event indicating the likelihood of the occurrence of that event on any trial.

We begin by looking at the classical model which first successfully formulated probability ideas in mathematical form. We use modern terminology and notation to describe it.

Classical probability
1. The basic space Ω consists of a finite number N of possible outcomes.

• There are thirty six possible outcomes of throwing two dice.

• There are different hands of five cards (order not important).

• There are 25=32 results (sequences of heads or tails) of flipping five coins.

2. Each possible outcome is assigned a probability 1/N

3. If event (subset) A has NA elements, then the probability assigned event A is

(1.13)

With this definition of probability, each event A is assigned a unique probability, which may be determined by counting NA, the number of elements in A (in the classical language, the number of outcomes "favorable" to the event) and N the total number of possible outcomes in the sure event Ω.

Example 1.3Probabilities for hands of cards

Consider the experiment of drawing a hand of five cards from an ordinary deck of 52 playing cards. The number of outcomes, as noted above, is N=C(52,5)=2598960. What is the probability of drawing a hand with exactly two aces? What is the probability of drawing a hand with two or more aces? What is the probability of not more than one ace?

SOLUTION

Let A be the event of exactly two aces, B be the event of exactly three aces, and C be the event of exactly four aces. In the first problem, we must count the number NA of ways of drawing a hand with two aces. We select two aces from the four, and select the other three cards from the 48 non aces. Thus

(1.14)

There are two or more aces iff there are exactly two or exactly three or exactly four. Thus the event D of two or more is D=ABC. Since are mutually exclusive,

(1.15) ND = NA + NB + NC = C ( 4 , 2 ) C ( 48 , 3 ) + C ( 4 , 3 ) C ( 48 , 2 ) + C ( 4 , 4 ) C ( 48 , 1 ) = 103776 + 4512 + 48 = 108336

so that P(D)≈0.0417. There is one ace or none iff there are not two or more aces. We thus want . Now the number in Dc is the number not in D which is NND, so that

(1.16)

This example illustrates several important properties of the classical probability.

1. P(A)=NA/N is a nonnegative quantity.

2. P ( Ω ) = N / N = 1

3. If A,B,C are mutually exclusive, then the number in the disjoint union is the sum of the numbers in the individual events, so that

(1.17)P(ABC)=P(A)+P(B)+P(C)

Several other elementary properties of the classical probability may be identified. It turns out that they can be derived from these three. Although the classical model is highly useful, and an extensive theory has been developed, it is not really satisfactory for many applications (the communications problem, for example). We seek a more general model which includes classical probability as a special case and is thus an extension of it. We adopt the Kolmogorov model (introduced by the Russian mathematician A. N. Kolmogorov) which captures the essential ideas in a remarkably successful way. Of course, no model is ever completely successful. Reality always seems to escape our logical nets.

The Kolmogorov model is grounded in abstract measure theory. A full explication requires a level of mathematical sophistication inappropriate for a treatment such as this. But most of the concepts and many of the results are elementary and easily grasped. And many technical mathematical considerations are not important for applications at the level of this introductory treatment and may be disregarded. We borrow from measure theory a few key facts which are either very plausible or which can be understood at a practical level. This enables us to utilize a very powerful mathematical system for representing practical problems in a manner that leads to both insight and useful strategies of solution.

Our approach is to begin with the notion of events as sets introduced above, then to introduce probability as a number assigned to events subject to certain conditions which become definitive properties. Gradually we introduce and utilize additional concepts to build progressively a powerful and useful discipline. The fundamental properties needed are just those illustrated in Example 1.3 for the classical case.

Definition

A probability system consists of a basic set Ω of elementary outcomes of a trial or experiment, a class of events as subsets of the basic space, and a probability measure P(⋅) which assigns values to the events in accordance with the following rules:

 (P1): For any event A, the probability P(A)≥0. (P2): The probability of the sure event P(Ω)=1. (P3): Countable additivity. If is a mutually exclusive, countable class of events, then the probability of the disjoint union is the sum of the individual probabilities.

The necessity of the mutual exclusiveness (disjointedness) is illustrated in Example 1.3. If the sets were not disjoint, probability would be counted more than once in the sum. A probability, as defined, is abstract—simply a number assigned to each set representing an event. But we can give it an interpretation which helps to visualize the various patterns and relationships encountered. We may think of probability as mass assigned to an event. The total unit mass is assigned to the basic set Ω. The additivity property for disjoint sets makes the mass interpretation consistent. We can use this interpretation as a precise representation. Repeatedly we refer to the probability mass assigned a given set. The mass is proportional to the weight, so sometimes we speak informally of the weight rather than the mass. Now a mass assignment with three properties does not seem a very promising beginning. But we soon expand this rudimentary list of properties. We use the mass interpretation to help visualize the properties, but are primarily concerned to interpret them in terms of likelihoods.

 (P4): . This follows from additivity and the fact that (1.18) (P5): P(∅)=0. The empty set represents an impossible event. It has no members, hence cannot occur. It seems reasonable that it should be assigned zero probability (mass). Since ∅=Ωc, this follows logically from (P4) and (P2). (P6): If A⊂B, then P(A)≤P(B). From the mass point of view, every point in A is also in B, so that B must have at least as much mass as A. Now the relationship A⊂B means that if A occurs, B must also. Hence B is at least as likely to occur as A. From a purely formal point of view, we have (1.19) (P7): The first three expressions follow from additivity and partitioning of A∪B as follows (see Figure 1.1). (1.20)A∪B=A⋁AcB=B⋁ABc=ABc⋁AB⋁AcB If we add the first two expressions and subtract the third, we get the last expression. In terms of probability mass, the first expression says the probability in A∪B is the probability mass in A plus the additional probability mass in the part of B which is not in A. A similar interpretation holds for the second expression. The third is the probability in the common part plus the extra in A and the extra in B. If we add the mass in A and B we have counted the mass in the common part twice. The last expression shows that we correct this by taking away the extra common mass. (P8): If is a countable, disjoint class and A is contained in the union, then (1.21) (P9): Subadditivity. If , then . This follows from countable additivity, property (P6), and the fact (Partitions) (1.22) This includes as a special case the union of a finite number of events.

Some of these properties, such as (P4), (P5), and (P6), are so elementary that it seems they should be included in the defining statement. This would not be incorrect, but would be inefficient. If we have an assignment of numbers to the events, we need only establish (P1), (P2), and (P3) to be able to assert that the assignment constitutes a probability measure. And the other properties follow as logical consequences.

Flexibility at a price

In moving beyond the classical model, we have gained great flexibility and adaptability of the model. It may be used for systems in which the number of outcomes is infinite (countably or uncountably). It does not require a uniform distribution of the probability mass among the outcomes. For example, the dice problem may be handled directly by assigning the appropriate probabilities to the various numbers of total spots, 2 through 12. As we see in the treatment of conditional probability, we make new probability assignments (i.e., introduce new probability measures) when partial information about the outcome is obtained.

But this freedom is obtained at a price. In the classical case, the probability value to be assigned an event is clearly defined (although it may be very difficult to perform the required counting). In the general case, we must resort to experience, structure of the system studied, experiment, or statistical studies to assign probabilities.

The existence of uncertainty due to “chance” or “randomness” does not necessarily imply that the act of performing the trial is haphazard. The trial may be quite carefully planned; the contingency may be the result of factors beyond the control or knowledge of the experimenter. The mechanism of chance (i.e., the source of the uncertainty) may depend upon the nature of the actual process or system observed. For example, in taking an hourly temperature profile on a given day at a weather station, the principal variations are not due to experimental error but rather to unknown factors which converge to provide the specific weather pattern experienced. In the case of an uncorrected digital transmission error, the cause of uncertainty lies in the intricacies of the correction mechanisms and the perturbations produced by a very complex environment. A patient at a clinic may be self selected. Before his or her appearance and the result of a test, the physician may not know which patient with which condition will appear. In each case, from the point of view of the experimenter, the cause is simply attributed to “chance.” Whether one sees this as an “act of the gods” or simply the result of a configuration of physical or behavioral causes too complex to analyze, the situation is one of uncertainty, before the trial, about which outcome will present itself.

If there were complete uncertainty, the situation would be chaotic. But this is not usually the case. While there is an extremely large number of possible hourly temperature profiles, a substantial subset of these has very little likelihood of occurring. For example, profiles in which successive hourly temperatures alternate between very high then very low values throughout the day constitute an unlikely subset (event). One normally expects trends in temperatures over the 24 hour period. Although a traffic engineer does not know exactly how many vehicles will be observed in a given time period, experience provides some idea what range of values to expect. While there is uncertainty about which patient, with which symptoms, will appear at a clinic, a physician certainly knows approximately what fraction of the clinic's patients have the disease in question. In a game of chance, analyzed into “equally likely” outcomes, the assumption of equal likelihood is based on knowledge of symmetries and structural regularities in the mechanism by which the game is carried out. And the number of outcomes associated with a given event is known, or may be determined.

In each case, there is some basis in statistical data on past experience or knowledge of structure, regularity, and symmetry in the system under observation which makes it possible to assign likelihoods to the occurrence of various events. It is this ability to assign likelihoods to the various events which characterizes applied probability. However determined, probability is a number assigned to events to indicate their likelihood of occurrence. The assignments must be consistent with the defining properties (P1), (P2), (P3) along with derived properties (P4) through (P9) (plus others which may also be derived from these). Since the probabilities are not “built in,” as in the classical case, a prime role of probability theory is to derive other probabilities from a set of given probabilites.

## 1.3. Interpretations*

### What is Probability?

The formal probability system is a model whose usefulness can only be established by examining its structure and determining whether patterns of uncertainty and likelihood in any practical situation can be represented adequately. With the exception of the sure event and the impossible event, the model does not tell us how to assign probability to any given event. The formal system is consistent with many probability assignments, just as the notion of mass is consistent with many different mass assignments to sets in the basic space.

The defining properties (P1), (P2), (P3) and derived properties provide consistency rules for making probability assignments. One cannot assign negative probabilities or probabilities greater than one. The sure event is assigned probability one. If two or more events are mutually exclusive, the total probability assigned to the union must equal the sum of the probabilities of the separate events. Any assignment of probability consistent with these conditions is allowed.

One may not know the probability assignment to every event. Just as the defining conditions put constraints on allowable probability assignments, they also provide important structure. A typical applied problem provides the probabilities of members of a class of events (perhaps only a few) from which to determine the probabilities of other events of interest. We consider an important class of such problems in the next chapter.

There is a variety of points of view as to how probability should be interpreted. These impact the manner in which probabilities are assigned (or assumed). One important dichotomy among practitioners.

• One group believes probability is objective in the sense that it is something inherent in the nature of things. It is to be discovered, if possible, by analysis and experiment. Whether we can determine it or not, “it is there.”

• Another group insists that probability is a condition of the mind of the person making the probability assessment. From this point of view, the laws of probability simply impose rational consistency upon the way one assigns probabilities to events. Various attempts have been made to find objective ways to measure the strength of one's belief or degree of certainty that an event will occur. The probability P(A) expresses the degree of certainty one feels that event A will occur. One approach to characterizing an individual's degree of certainty is to equate his assessment of P(A) with the amount a he is willing to pay to play a game which returns one unit of money if A occurs, for a gain of (1–a), and returns zero if A does not occur, for a gain of a. Behind this formulation is the notion of a fair game, in which the “expected” or “average” gain is zero.

The early work on probability began with a study of relative frequencies of occurrence of an event under repeated but independent trials. This idea is so imbedded in much intuitive thought about probability that some probabilists have insisted that it must be built into the definition of probability. This approach has not been entirely successful mathematically and has not attracted much of a following among either theoretical or applied probabilists. In the model we adopt, there is a fundamental limit theorem, known as Borel's theorem, which may be interpreted “if a trial is performed a large number of times in an independent manner, the fraction of times that event A occurs approaches as a limit the value P(A). Establishing this result (which we do not do in this treatment) provides a formal validation of the intuitive notion that lay behind the early attempts to formulate probabilities. Inveterate gamblers had noted long-run statistical regularities, and sought explanations from their mathematically gifted friends. From this point of view, probability is meaningful only in repeatable situations. Those who hold this view usually assume an objective view of probability. It is a number determined by the nature of reality, to be discovered by repeated experiment.

There are many applications of probability in which the relative frequency point of view is not feasible. Examples include predictions of the weather, the outcome of a game or a horse race, the performance of an individual on a particular job, the success of a newly designed computer. These are unique, nonrepeatable trials. As the popular expression has it, “You only go around once.” Sometimes, probabilities in these situations may be quite subjective. As a matter of fact, those who take a subjective view tend to think in terms of such problems, whereas those who take an objective view usually emphasize the frequency interpretation.

Example 1.4Subjective probability and a football game

The probability that one's favorite football team will win the next Superbowl Game may well be only a subjective probability of the bettor. This is certainly not a probability that can be determined by a large number of repeated trials. The game is only played once. However, the subjective assessment of probabilities may be based on intimate knowledge of relative strengths and weaknesses of the teams involved, as well as factors such as weather, injuries, and experience. There may be a considerable objective basis for the subjective assignment of probability. In fact, there is often a hidden “frequentist” element in the subjective evaluation. There is an assessment (perhaps unrealized) that in similar situations the frequencies tend to coincide with the value subjectively assigned.

Example 1.5The probability of rain

Newscasts often report that the probability of rain of is 20 percent or 60 percent or some other figure. There are several difficulties here.

• To use the formal mathematical model, there must be precision in determining an event. An event either occurs or it does not. How do we determine whether it has rained or not? Must there be a measurable amount? Where must this rain fall to be counted? During what time period? Even if there is agreement on the area, the amount, and the time period, there remains ambiguity: one cannot say with logical certainty the event did occur or it did not occur. Nevertheless, in this and other similar situations, use of the concept of an event may be helpful even if the description is not definitive. There is usually enough practical agreement for the concept to be useful.

• What does a 30 percent probability of rain mean? Does it mean that if the prediction is correct, 30 percent of the area indicated will get rain (in an agreed amount) during the specified time period? Or does it mean that 30 percent of the occasions on which such a prediction is made there will be significant rainfall in the area during the specified time period? Again, the latter alternative may well hide a frequency interpretation. Does the statement mean that it rains 30 percent of the times when conditions are similar to current conditions?

Regardless of the interpretation, there is some ambiguity about the event and whether it has occurred. And there is some difficulty with knowing how to interpret the probability figure. While the precise meaning of a 30 percent probability of rain may be difficult to determine, it is generally useful to know whether the conditions lead to a 20 percent or a 30 percent or a 40 percent probability assignment. And there is no doubt that as weather forecasting technology and methodology continue to improve the weather probability assessments will become increasingly useful.

Another common type of probability situation involves determining the distribution of some characteristic over a population—usually by a survey. These data are used to answer the question: What is the probability (likelihood) that a member of the population, chosen “at random” (i.e., on an equally likely basis) will have a certain characteristic?

Example 1.6 Empirical probability based on survey data

A survey asks two questions of 300 students: Do you live on campus? Are you satisfied with the recreational facilities in the student center? Answers to the latter question were categorized “reasonably satisfied,” “unsatisfied,” or “no definite opinion.” Let C be the event “on campus;” O be the event “off campus;” S be the event “reasonably satisfied;” U be the event ”unsatisfied;” and N be the event “no definite opinion.” Data are shown in the following table.

Survey Data

 S U N C 127 31 42 O 46 43 11

If an individual is selected on an equally likely basis from this group of 300, the probability of any of the events is taken to be the relative frequency of respondents in each category corresponding to an event. There are 200 on campus members in the population, so P(C)=200/300 and P(O)=100/300. The probability that a student selected is on campus and satisfied is taken to be P(CS)=127/300. The probability a student is either on campus and satisfied or off campus and not satisfied is

(1.23) P ( C SO U ) = P ( C S ) + P ( O U ) = 127 / 300 + 43 / 300 = 170 / 300

If there is reason to believe that the population sampled is representative of the entire student body, then the same probabilities would be applied to any student selected at random from the entire student body.

It is fortunate that we do not have to declare a single position to be the “correct” viewpoint and interpretation. The formal model is consistent with any of the views set forth. We are free in any situation to make the interpretation most meaningful and natural to the problem at hand. It is not necessary to fit all problems into one conceptual mold; nor is it necessary to change mathematical model each time a different point of view seems appropriate.

### Probability and odds

Often we find it convenient to work with a ratio of probabilities. If A and B are events with positive probability the odds favoring A over B is the probability ratio P(A)/P(B). If not otherwise specified, B is taken to be Ac and we speak of the odds favoring A

(1.24)

This expression may be solved algebraically to determine the probability from the odds

(1.25)

In particular, if O(A)=a/b then .

O(A)=0.7/0.3=7/3. If the odds favoring A is 5/3, then P(A)=5/(5+3)=5/8.

### Partitions and Boolean combinations of events

The countable additivity property (P3) places a premium on appropriate partitioning of events.

Definition. A partition is a mutually exclusive class

(1.26)

A partition of event A is a mutually exclusive class

(1.27)

Remarks.

• A partition is a mutually exclusive class of events such that one (and only one) must occur on each trial.

• A partition of event A is a mutually exclusive class of events such that A occurs iff one (and only one) of the Ai occurs.

• A partition (no qualifier) is taken to be a partition of the sure event Ω.

• If class is mutually exclusive and , then the class is a partition of A and .

We may begin with a sequence and determine a mutually exclusive (disjoint) sequence as follows:

(1.28)

Thus each Bi is the set of those elements of Ai not in any of the previous members of the sequence.

This representation is used to show that subadditivity (P9) follows from countable additivity and property (P6). Since each BiAi, by (P6) . Now

(1.29)

The representation of a union as a disjoint union points to an important strategy in the solution of probability problems. If an event can be expressed as a countable disjoint union of events, each of whose probabilities is known, then the probability of the combination is the sum of the individual probailities. In in the module on Partitions and Minterms, we show that any Boolean combination of a finite class of events can be expressed as a disjoint union in a manner that often facilitates systematic determination of the probabilities.

### The indicator function

One of the most useful tools for dealing with set combinations (and hence with event combinations) is the indicator function IE for a set EΩ. It is defined very simply as follows:

(1.30)

Remark. Indicator fuctions may be defined on any domain. We have occasion in various cases to define them on the real line and on higher dimensional Euclidean spaces. For example, if M is the interval on the real line then IM(t)=1 for each t in the interval (and is zero otherwise). Thus we have a step function with unit value over the interval M. In the abstract basic space Ω we cannot draw a graph so easily. However, with the representation of sets on a Venn diagram, we can give a schematic representation, as in Figure 1.2.

Much of the usefulness of the indicator function comes from the following properties.

 (IF1): IA≤IB iff A⊂B. If IA≤IB, then ω∈A implies IA(ω)=IB(ω)=1, so ω∈B. If A⊂B, then IA(ω)=1 implies ω∈A implies ω∈B implies IB(ω)=1. (IF2): IA=IB iff A=B(1.31) (IF3): IAc=1–IA This follows from the fact IAc(ω)=1 iff IA(ω)=0. (IF4): (extends to any class) An element ω belongs to the intersection iff it belongs to all iff the indicator function for each event is one iff the product of the indicator functions is one. (IF5): (the maximum rule extends to any class) The maximum rule follows from the fact that ω is in the union iff it is in any one or more of the events in the union iff any one or more of the individual indicator function has value one iff the maximum is one. The sum rule for two events is established by DeMorgan's rule and properties (IF2), (IF3), and (IF4). (1.32) (IF6): If the pair is disjoint, IA⋁B=IA+IB (extends to any disjoint class)

The following example illustrates the use of indicator functions in establishing relationships between set combinations. Other uses and techniques are established in the module on Partitions and Minterms.

Example 1.7Indicator functions and set combinations

Suppose is a partition.

(1.33)

VERIFICATION

Utilizing properties of the indicator function established above, we have

(1.34)

Note that since the Ai form a partition, we have , so that the indicator function for the complementary event is

(1.35)

The last sum is the indicator function for .

### A technical comment on the class of events

The class of events plays a central role in the intuitive background, the application, and the formal mathematical structure. Events have been modeled as subsets of the basic space of all possible outcomes of the trial or experiment. In the case of a finite number of outcomes, any subset can be taken as an event. In the general theory, involving infinite possibilities, there are some technical mathematical reasons for limiting the class of subsets to be considered as events. The practical needs are these:

1. If A is an event, its complementary set must also be an event.

2. If is a finite or countable class of events, the union and the intersection of members of the class need to be events.

A simple argument based on DeMorgan's rules shows that if the class contains complements of all its sets and countable unions, then it contains countable intersections. Likewise, if it contains complements of all its sets and countable intersections, then it contains countable unions. A class of sets closed under complements and countable unions is known as a sigma algebra of sets. In a formal, measure-theoretic treatment, a basic assumption is that the class of events is a sigma algebra and the probability measure assigns probabilities to members of that class. Such a class is so general that it takes very sophisticated arguments to establish the fact that such a class does not contain all subsets. But precisely because the class is so general and inclusive in ordinary applications we need not be concerned about which sets are permissible as events

A primary task in formulating a probability problem is identifying the appropriate events and the relationships between them. The theoretical treatment shows that we may work with great freedom in forming events, with the assurrance that in most applications a set so produced is a mathematically valid event. The so called measurability question only comes into play in dealing with random processes with continuous parameters. Even there, under reasonable assumptions, the sets produced will be events.

## 1.4. Problems on Probability Systems*

Let Ω consist of the set of positive integers. Consider the subsets

Describe in terms of A,B,C,D, E and their complements the following sets:

1. The even integers greater than 12.

2. The positive integers which are multiples of six.

3. The integers which are even and no greater than 6 or which are odd and greater than 12.

Let Ω be the set of integers 0 through 10. Let , B= the odd integers in Ω, and C= the integers in Ω which are even or less than three. Describe the following sets by listing their elements.

1. AB

2. AC

3. ABcC

4. ABCc

5. ABc

6. ABCc

7. ABC

8. AcBCc

1. AB={5,7}

2. AC={6,8}

3. ABCC=C

4. ABCc=AB

5. ABc={0,2,4,5,6,7,8,10}

6. ABC=

7. AcBCc={3,9}

Consider fifteen-word messages in English. Let A= the set of such messages which contain the word “bank” and let B= the set of messages which contain the word “bank” and the word “credit.” Which event has the greater probability? Why?

BA implies P(B)≤P(A).

A group of five persons consists of two men and three women. They are selected one-by-one in a random manner. Let Ei be the event a man is selected on the ith selection. Write an expression for the event that both men have been selected by the third selection.

A=E1E2E1E2cE3E1cE2E3

Two persons play a game consecutively until one of them is successful or there are ten unsuccessful plays. Let Ei be the event of a success on the ith play of the game. Let A,B,C be the respective events that player one, player two, or neither wins. Write an expression for each of these events in terms of the events Ei, 1≤i≤10.

(1.36) A = E1E1c E2c E3E1c E2c E3c E4c E5E1c E2c E3c E4c E5c E6c E7E1c E2c E3c E4c E5c E6c E7c E8c E9

Suppose the game in Exercise 5. could, in principle, be played an unlimited number of times. Write an expression for the event D that the game will be terminated with a success in a finite number of times. Write an expression for the event F that the game will never terminate.

Let F0=Ω and for k≥1. Then

(1.37)

Find the (classical) probability that among three random digits, with each digit (0 through 9) being equally likely and each triple equally likely:

1. All three are alike.

2. No two are alike.

3. The first digit is 0.

4. Exactly two are alike.

Each triple has probability 1/103=1/1000

1. Ten triples, all alike: P=10/1000.

2. 10×9×8 triples all different: P=720/1000.

3. 100 triples with first one zero: P=100/1000

4. C(3,2)=3 ways to pick two positions alike; 10 ways to pick the common value; 9 ways to pick the other. P=270/1000.

The classical probability model is based on the assumption of equally likely outcomes. Some care must be shown in analysis to be certain that this assumption is good. A well known example is the following. Two coins are tossed. One of three outcomes is observed: Let ω1 be the outcome both are “heads,” ω2 the outcome that both are “tails,” and ω3 be the outcome that they are different. Is it reasonable to suppose these three outcomes are equally likely? What probabilities would you assign?

.

A committee of five is chosen from a group of 20 people. What is the probability that a specified member of the group will be on the committee?

C(20,5) committees; C(19,4) have a designated member.

(1.38)

Ten employees of a company drive their cars to the city each day and park randomly in ten spots. What is the (classical) probability that on a given day Jim will be in place three? There are n! equally likely ways to arrange n items (order important).

10! permutations. 1×9! permutations with Jim in place 3. P=9!/10!=1/10.

An extension of the classical model involves the use of areas. A certain region L (say of land) is taken as a reference. For any subregion A, define P(A)=area(A)/area(L). Show that P(⋅) is a probability measure on the subregions of L.

John thinks the probability the Houston Texans will win next Sunday is 0.3 and the probability the Dallas Cowboys will win is 0.7 (they are not playing each other). He thinks the probability both will win is somewhere between—say, 0.5. Is that a reasonable assumption? Justify your answer.

P(AB)=0.5is not reasonable. It must no greater than the minimum of P(A)=0.3 and P(B)=0.7.

Suppose P(A)=0.5 and P(B)=0.3. What is the largest possible value of P(AB)? Using the maximum value of P(AB), determine , , and P(AB). Are these values determined uniquely?

Draw a Venn diagram, or use algebraic expressions

(1.39)

For each of the following probability “assignments”, fill out the table. Which assignments are not permissible? Explain why, in each case.

 P ( A ) P ( B ) P ( A B ) P ( A ∪ B ) P ( A ) + P ( B ) 0.3 0.7 0.4 0.2 0.1 0.4 0.3 0.7 0.2 0.3 0.5 0 0.3 0.8 0
 P ( A ) P ( B ) P ( A B ) P ( A ∪ B ) P ( A ) + P ( B ) 0.3 0.7 0.4 0.6 -0.1 0.3 1.0 0.2 0.1 0.4 -0.1 -0.2 -0.3 0.3 0.3 0.7 0.2 0.8 0.1 0.5 1.0 0.3 0.5 0 0.8 0.3 0.5 0.8 0.3 0.8 0 1.1 0.3 0.8 1.1

Only the third and fourth assignments are permissible.

The class of events is a partition. Event A is twice as likely as C and event B is as likely as the combination A or C. Determine the probabilities .

P(A)+P(B)+P(C)=1, P(A)=2P(C), and P(B)=P(A)+P(C)=3P(C), which implies

(1.40)

Determine the probability P(ABC) in terms of the probabilities of the events A,B,C and their intersections.

P(ABC)=P(AB)+P(C)–P(ACBC)

(1.41) = P ( A ) + P ( B ) – P ( A B ) + P ( C ) – P ( A C ) – P ( B C ) + P ( A B C )

If occurrence of event A implies occurrence of B, show that .

P(AB)=P(A) and implies .

Show that P(AB)≥P(A)+P(B)–1.

Follows from P(A)+P(B)–P(AB)=P(AB)≤1.

The set combination AB=ABcAcB is known as the disjunctive union or the symetric difference of A and B. This is the event that only one of the events A or B occurs on a trial. Determine P(AB) in terms of P(A), P(B), and P(AB).

A Venn diagram shows .

Use fundamental properties of probability to show

1. P ( A B ) ≤ P ( A ) ≤ P ( AB ) ≤ P ( A ) + P ( B )

ABAAB implies P(AB)≤P(A)≤P(AB)=P(A)+P(B)–P(AB)≤P(A)+P(B). The general case follows similarly, with the last inequality determined by subadditivity.

Suppose are probability measures and are positive numbers such that c1+c2=1. Show that the assignment P(E)=c1P1(E)+c2P2(E) to the class of events is a probability measure. Such a combination of probability measures is known as a mixture. Extend this to

(1.42)

Clearly P(E)≥0. P(Ω)=c1P1(Ω)+c2P2(Ω)=1.

(1.43)

The pattern is the same for the general case, except that the sum of two terms is replaced by the sum of n terms ciPi(E).

Suppose is a partition and is a class of positive constants. For each event E, let

(1.44)

Show that Q(⋅) us a probability measure.

Clearly Q(E)≥0 and since AiΩ=Ai we have Q(Ω)=1. If

(1.45)

Interchanging the order of summation shows that Q is countably additive.

Solutions