\magnification 2400
\parindent 0pt
\parskip 10pt
\nopagenumbers
\hsize 7.5truein
\vsize 10 truein
\hoffset -0.45truein
\voffset -0.45truein
\def\u{\vskip -10pt}
\def\v{\vskip -5pt}
\input ../../../PAPERS/GOLD/psfig


$p_{ij}$ = $P(O_t = u_j ~|~ O_{t-1} = u_i)$ \hfil \break
= probability that the outcome at time $t$ is $u_j$ given 
that the outcome at time $t-1$ is $u_i$ \hfil \break
= State at time $t$ is $u_j$ given that the state at time 
$t-1$ is $u_i$.

$p_{ij}^{(t)}$ = $P(O_t = u_j ~|~ O_{0} = u_i)$\hfil \break
 = probability that
state at time $t$ is $u_j$ given that the chain starts in state $u_i$ (at time 0).
\hfil \break= probability that you go from state $u_i$ to $u_j$ in $t$ steps given that you are at 
$u_j$.
\hfil\break= i,j entry of $P^t$ where $P$ is the transition matrix.

A set C is closed if $p_{ij} = 0$ for all $u_i \in C$ and $u_j \not\in C$

A set E is ergodic if it is a minimal closed set  (i.e.  E is closed and no proper subset of E is 
closed) = strong component which is closed.

Transient set = strong component which is not ergodic.

A state is ergodic if it is in an ergodic strong component.
\u
A state is transient if it is in a transient strong component.

If an ergodic strong component consists of only a single vertex $u_i$, then $u_i$ is an absorbing 
state.


A Markov chain is ergodic iff it is strongly connected 

A Markov chain is absorbing if each of  its ergodic strong components have only one element.

Entering an absorbing state is called absorption.


\eject
\eject

Every Markov chain has an ergodic set.

Thm 5.4:  In any (finite) Markov chain, the probability after $t$ steps that the process is in an 
ergodic state approaches 1 as $t$ approaches $\infty$.

Thm 5.3:  A Markov chain is absorbing if and only if it has at least one absorbing state and from every 
nonabsorbing state it is possible to reach some absorbing state

Corollary to 5.4:  In an absorbing Markov chain, the probability of absorption is 1.


Questions:
\v
2.)  
The expected number of times the process will be in a given non-absorbing state $u_j$ starting
from a given non-absorbing state $u_i$
\v
2.5)  The expected number of steps before absorption 
starting 
from a given non-absorbing state $u_i$


Cor 5.5:  If $I$ is the identity matrix and $Q$ is any square matrix of real numbers such that \break
$lim_{n \rightarrow \infty} Q^n = 0$, the zero matrix, then $I-Q$ is invertible and $(I-Q)^{-1} = I +
Q + Q^2 + ... = \Sigma_{n=0}^\infty Q^n$.


\end

1.)  What is the probability of entering a given absorbing state $u_j$ starting from a given 
non-absorbing 
state $u_i$ in a finite number of steps.

3.)  What is the probability of entering an absorbing state starting from a given 
non-absorbing 
state $u_i$ in a finite number of steps.







\end