The aim of this book is to outline the recent development of Markov chain
models for modeling queueing systems, Internet, re-manufacturing systems,
inventory systems, DNA sequences, genetic networks and many other practical
systems.
This book consists of eight chapters. In Chapter 1, we give a brief introduction to the classical theory on both discrete and continuous time Markov
chains. The relationship between Markov chains of finite states and matrix
theory will also be discussed. Some classical iterative methods for solving
linear systems will also be introduced. We then give the basic theory and
algorithms for standard hidden Markov model (HMM) and Markov decision
process (MDP).