@Muhammad Usman

Pay attention to the Answer by Maxim: Do not solve for alpha! Instead, use eval to set the value of alpha in the system before it is solved. This makes the difference between a system of linear equations and and a system of nonlinear equations. It's a huge difference in computational effort at your value of Digits (which is implicit in Markiyan's Answer).

The purpose of my Answer was to address the proximate cause of the error message in the worksheet that you presented---that you were mixing lists and sets. This was a superficial syntax error that I could fix without executing the code, which is what I did because I was hurrying. Even though there was a deeper non-syntax-related problem with the code after that was fixed, the syntax still needed to be fixed, so my Answer is still valid.

I used k = 1/100 so that N = M = 100. That leads to a system of 29,700 equations. For Digits = 15, I obtained the solution in well under 1 minute. I have some concerns about the numerical stability of that solution that I haven't investigated. Regardless, such a system should probably be solved by sparse iterative matrix methods from the LinearAlgebra package. See ?LinearAlgebra,LinearSolve, ?LinearAlgebra,General,LinearSolveItr, and ?LinearAlgebra,GenerateMatrix.

@assma

If I copy-and-paste your code and run it exactly as you have it, except using a colon after end do, in Maple 2016, I get AVor = -4.633913767. If I use a second method (using add and evalhf), I get -4.63391167924315983, which is close enough to the first value. If I use a third method (compiling the code), I get -4.63391167924588299, which also is close enough. These are very different from the value that you got!

The code for the second method, which I'm nearly sure will run in Maple 18, is

f:= proc(x,y) option inline; sin(x)*cos(y) end proc:
CodeTools:-Usage(
   evalhf(add(add(f(.1*x, .1*y), x= 0..1000), y= 0..1000))
);

The code for the third method, which may not run in Maple 18 due to lack of a compiler, is

F:= proc()::float;
local i::integer, j::integer, AVor::float:= 0, x::float, y::float;
   for i from 0 to 1000 do
      x:= i*.1;     
      for j from 0 to 1000 do
         y:= j*.1;
         AVor:= AVor + sin(x)*cos(y)
      end do
   end do;
   AVor
end proc:

FC:= Compiler:-Compile(F):
FC();

"Extended" typesetting is not meant to be a correction for bugs present in Standard typesetting. It ought to display correctly in both.

Can you give more details about what you mean by "filter"? Do you mean something like removing outliers? For example, you could remove any datum that's more the 3 standard deviations from the mean.

By "code", Kitonum means that you should upload your worksheet file to this site. Use the green uparrow on the toolbar of the MaplePrimes editor.

Function application and index application ought to be listed in the chart at ?operators,precedence. In addition to being generaly useful, this would be a help for the people writing the typesetting code. Since there are no explicit infix symbols for these, people often fail to think of them as binary infix operators; but of course they are, as your example shows. I've mentioned this a few times over the years.

A slightly different way, using a few fewer lines of code, is Eigenvalues(F, V). For a floating-point problem this is more robust, but I doubt that there's much difference in computational effort for a symbolic problem such as this.

Acer, as far as I can tell, the OP's Reply immediately above is correct: The indexed occurrences of lam are being generated internally by a bug in int in the line beginning RRn:=. The OP doesn't appear to be at fault for this.

@tsunamiBTP Look up "catasthrophic cancellation" and especially the Wikipedia article "Loss of significance." The case at hand is an example of catasthrophic cancellation: the substraction of two quantities whose difference is many orders of magnitude smaller than the quantities.

@Math Pi Euler Try copy-and-pasting Tom's code. I think that you may have been entering it manually.

@Math Pi Euler Yeah, thanks, I figured that out by now. Anyway, I think that the Answer by Tom Leslie is sufficient for your purpose. If not, let me know.

What are the limits of summation?

Try typing your expressions here the same way as you would in Maple.

You say "verify the following formula" but I don't see any formula. You'll need to provide more information.

Update: The above comment applied to an earlier version of the Question.

@macmp91 So, we see that the minimal value of what you call w^2 eventually becomes negative. Why not use that as the criterion to stop your while loop? Don't take the sqrt in procedure Fn; take it afterwards.

@macmp91 Let's forget about the singular values for now. I don't know if they're appropriate for your problem. Why do you take the square root of the eigenvalues? Why not just stop your while loop when the minimal eigenvalue is negative?