This book develops systematically and rigorously, yet in an expository and lively manner, the evolution of general random processes and their large time properties such as transience, recurrence, and convergence to steady states. The emphasis is on the most important classes of these processes from the viewpoint of theory as well as applications, namely, Markov processes. The book features very broad coverage of the most applicable aspects of stochastic processes, including sufficient material for self-contained courses on random walk in one and multiple dimensions; Markov chains in discrete and continuous times, including birth-death processes; Brownian motion and diffusions; stochastic optimization; and stochastic differential equations. Most results are presented with complete proofs, while some very technical matters are relegated to a Theoretical Complements section at the end of each chapter in order not to impede the flow of the material. Chapter Applications, as well as numerous extensively worked examples, illustrate important applications of the subject to various fields of science, engineering, economics, and applied mathematics. The essentials of measure theoretic probability are included in an appendix to complete some of the more technical aspects of the text. Audience: This book can be used for a number of different courses for graduate students of mathematics, statistics, economics, engineering, and other fields who have some background in probability and analysis. It is also intended as a reference for researchers and professionals in many areas of science and technology whose work involves the application of probability.