As a Sub, Paul's is a good way to go.
In this method, I show how you can use the code as a sub or as an array formula. It can be tweaked to remove the nRows input.
Sub Test_fMarkov() Dim a
a = fMarkov([G5:I7], [D12:D14], 3, 15)
[E18].Resize(UBound(a), UBound(a, 2)) = a
End Sub
'array formula: =fMarkov(G5:I7, D12:D14, 3, 15)
Function fMarkov(rMatrix, rVector, nRows, nCols)
Dim a, b, d, m, r As Long, c As Integer
nCols = nCols + 1
ReDim a(1 To nRows, 1 To nCols)
ReDim b(1 To nRows, 1 To 1)
'Set first column
m = WorksheetFunction.Mmult(rMatrix, rVector)
For r = 1 To nRows
a(r, 1) = m(r, 1)
Next r
'Fill other columns
For c = 2 To nCols
For r = 1 To nRows
b(r, 1) = a(r, c - 1)
Next r
m = WorksheetFunction.Mmult(rMatrix, b)
For r = 1 To nRows
a(r, c) = m(r, 1)
Next r
Next c
'Slice Array a: 1 to nRows, 2 to nCols)
d = Application.Index(a, Evaluate("row(1:" & nRows & ")"), _
Application.Transpose(Evaluate("row(1:" & nCols - 1 & ")")))
fMarkov = d
End Function