Do you think that Maplesoft should invest a so much effort for typesetting?
Typesetting is a nice feature but I think that an excessive fine-tuning is too expensive.
I'd prefer these efforts be directed towards maths.
And anyway, Maple is not going to compete with LaTeX.
By the way, wouldn't it be more natural a LaTeX convention in the case of atomic variables?
It is not comfortable to use names like  `#msub(mi("x"),mn("1"))`  (in 1D notation)
(of course a macro() can be used here but even so...).

I also did not noticed the square. But Kitonum posted already a nice and complete solution.

P.S. It will be better in the future to post code instead of pictures.

@sajad 

@_Maxim_ 

I think you forgot index=1 in RootOf. Then it works as expected.
(It's only evalf which takes index=1 by default.)

@Mariusz Iwaniuk 

You cannot obtain this conclusion! The series could be divergent. for some c's.
As it is easy to check, the equation has at most 3 real roots,

@asa12 

Consider the row operation Eij of adding the row i to row j.
If  Eij . X = X  for each i,j (i<>j) then obviously X = O.

@asa12 

Your definition of "invariant matrix" is missing. For the standard definition, the only matrix is zero.

@_Maxim_ 

But if f is just absolutely integrable (in the Lebesgue sense) then the series could be divergent at any point! And anyway the side limits f(x0+0), f(x0-0) may not exist.

@sand15 

It works for me.

Have you loaded plots?
Or, use plots:-display

@tomleslie 

For Digits:=21 the imaginary part is still present.

@asa12 

You have successfully found what? The zero matrix or something else?

@asa12 

If you are considering row additions E, then a matrix X is invariant under them (i.e. E.X = X, for each E)
iff X = 0. Probably this is not what you need.

@asa12 

You should explain what you are trying to do. Code only (which does not work correctly) does not say much.

Are you looking for these?

https://en.wikipedia.org/wiki/Elementary_matrix