Problem 1

A = {{1,-1,0},{0,1,-1},{0,0,1}};
MatrixForm[A]

MatrixForm[Inverse[A]]

either of these forms of the characteristic polynomial is fine

CharacteristicPolynomial[A, λ]//FullSimplify
CharacteristicPolynomial[A, λ]

Eigensystem[A]
MatrixForm[Eigensystem[A][[1]]]
MatrixForm[Eigensystem[A][[2]]]

So all the eigenvalues are the same.  This means that there is only one independent eigenvector.  So the matrix is not diagonalizable, even though it is invertible.  Remember, there are three eigenvalues since the matrix is 3x3.  It just so happens they are all the same.

Problem 2

this is of the form

a + a r + a r^2 ... where a = PA and r = (1 - P).  So the sum is PA(1-(1-P)^r)/(1-(1-P)) = PA/P = A.
To get full credit you need to 1) explain how this geometric series arises, step by step; 2) the physical interpretation is that after an infinite number of steps the entire substance decays.

So, after 0 steps there is 0 decayed and A undecayed
After 1 steps there is PA decayed and hence (1-P)A undecayed
After 2 steps there is P(1-P)A = PA(1-P) decayed and (1-P) (1-P)A undecayed
After 3 steps there is P(1-P)^2A decayed and (1-P)^3 A undecayed ....
So after N steps the total decayed is the sum:

PA + PA(1-P) + PA(1-P)^2 ... PA(1-P)^N  = A

The interpretation of this is that 100% of the substance decays eventually, no matter what P is.  If you simply gave the sum of the series, which you could get from the book, you lost 10 points.

Problem 3

s[x_] = Normal[Series[Sin[x],{x,0,5}]]

error[x_] = x^3/6

Since the x^5 term is about 10,000 times smaller than the x^3 term, we can neglect it.  So the error entailed by this approximation is approximately equal to the first term neglected; in this case x^3/6.  Plus or minus this doesn't matter.  We're only interested in the size of the error.

error[.1]

This is an absolute error.  You could also give a relative error, which would the size of the error relative to the approximate value:  x^3/x.  In this case that would be

(.1^3 / 6)/ .1

So the approximation is better than 99 % accurate.

Problem 4

You can likely do these by inspection, which is fine.  You don't have to compute any inverses to do this, just matrix multiplies.

A1 = {{1, 2I, 3+4I},{-2I, 5, 6-7I},{3-4I,6+7I,8}}
A2 = 1/Sqrt[2]{{1,1,0},{-1,1,0},{0,0,Sqrt[2]I}}
A3 = 1/3{{2,-2,1},{1,2,2},{2,1,-2}}
A4 = {{66,78,90},{78,93,108},{90,108,126}}
MatrixForm[A1]
MatrixForm[ConjugateTranspose[A1]]
MatrixForm[A2]
MatrixForm[ConjugateTranspose[A2] . A2]
MatrixForm[A3]
MatrixForm[Transpose[A3] . A3]
MatrixForm[A4]
MatrixForm[Transpose[A4]]

Problem 5

Some of you gave the angles in degrees.  That' s OK.  But in future, please use radians.

Print[z1 = 1 + I Sqrt[3]," ",Abs[z1]," ",Arg[z1]]
Print[z2 = (1-I)/(1+I), " ",Abs[z2], " ",Arg[z2]]
Print[z3 = I^5 + I^6," ", Abs[z3]," ",Arg[z3]]
Print[z4 = 1/(1-I)," ",Abs[z4]," ",Arg[z4]]

Problem 6

A = {{-1,2},{1,-2}};
h = {{1},{-1}};
MatrixForm[A]
MatrixForm[h]
Solve[A . {{x},{y}} == h]

n = NullSpace[A][[1]]

MatrixForm[n]

p = { {2y -1},{y}}
A  .  n
A  .  p
A . (p + a n)//FullSimplify

This shows that any vector of the form p + a n is a solution.  This is the most general statement.