@Alfred_F  What did you find difficult in my solution? It is based on direct calculation of triangle areas using coordinates of vertices of arbitrary parallelogram. Of course, you can solve it in another way, for example using commands of geometry package:

restart;
local D: 
with(geometry):
assume(a>0, c>0):
point(A,0,0): point(B,b,c): point(C,a+b,c): point(D,a,0): 
OnSegment(M,D,C,2/3): OnSegment(N,B,M,1): 
area_AND:=area(triangle(t,[A,N,D])):
area_BCM:=area(triangle(t1,[B,C,M])):
area_BCM/area_AND; 

@Ronan If you solve it in your head, it is easier to consider similar triangles  AFE and BCF  with a similarity coefficient of  1/2 . It follows that   BF = 2*FE . Also  S(BCE) = 1/2*S(ABCD) . So  S(EFC) = (1/2)*(1/3)*S(ABCD) = (1/6)*24 = 4  

@Carl Love Thanks for this. I didn't know this command before. Of course Iterator:-TopologicalSorts is the shortest way to generate permutations with restrictions.

@Susana30  You've already been shown one way. Here's another way, where we replace this region with a polygon for coloring:

restart;
f:=y->y^2-3: g:=y->y-y^2:
L1:=seq([g(y),y], y=-1..3/2, 0.1):
L2:=seq([f(y),y], y=3/2..-1, -0.1):
plots:-display(plot([L1], color=red, thickness=3),plot([L2], color=blue, thickness=3), plottools:-polygon([L1,L2], color="LightGreen"), scaling=constrained);

@minhthien2016  Why do you think this angle is 3*Pi/4?

Here is another solution to your problem if S(0,0,sqrt(2)/2). This solution uses the symmetry of this polyhedron about the ASC plane:

restart;
local D, O:
with(Student:-MultivariateCalculus):
A, B, C, D, S :=  [0,0,0], [1,0,0], [1,1,0], [0,1,0], [0,0,sqrt(2)/2]:
L:=Line(C,S);
O:=Projection(B, L);
v1:=convert(B-O,Vector);
v2:=convert(D-O,Vector);
Angle(v1,v2); # The answer

@Carl Love  Through the points M, M1, M2 (from my answer) we draw a plane intersecting the line of intersection of our half-planes at point O. Consider the quadrilateral OM1MM2. Obviously, the sum of the angles M1MM2 and M1OM2 is equal to Pi (we are looking for the angle M1OM2).

@JAMET  Insert  the line  s1:=polygon([A,M,N,P], color = "Pink", transparency = 0.7);  and add  s1  to the arguments of the  display  command.

Maple 2018.2 instantly returns a solution in terms of RootOf:

Download ode.mw

@JAMET  Your problem can be solved much more simply if you notice (it is not difficult to prove) that angles CHK and CDK are right angles. Therefore, the circle with diameter CK passes through points C and K. Therefore, O is the midpoint of segment СК.

@JAMET  Obviously ABCD is a rectangle and therefore the center of the circle described around it coincides with the center of this rectangle, that is  O(a/2, b/2, 0) . Maple is not needed here at all.

I looked at your code, but I didn't understand the meaning of what you are doing. In the title you write about the circumscribed circle of some triangle. What triangle do you mean?

Here is the solution to your example in Maple 2018.2:
 

Download no_warning_messages.mw

@JAMET  See code below. x, y, z are the barycentric coordinates of point M:

restart;
with(LinearAlgebra):
A:=<1,-3,0>: B:=<-2,1,1>: C:=<3,1,2>: M:=x*A+y*B+z*C:
{DotProduct(B-A,C-M)=0, DotProduct(C-A,B-M)=0, x+y+z = 1}:
solve({DotProduct(B-A,C-M)=0, DotProduct(C-A,B-M)=0, x+y+z = 1}); # barycentric coordinates of M
'M'=eval(M, %); # cartesian coordinates of M

@vv But  evala  works in other similar situations too. For example

restart;
M:=<a, -b; a/b, a*b>;
a:=2: b:=3:
evala(M);

@Susana30  For this example use the  surd  function:

A:=int((x^2)^(1/3), x);
surd(A^3, 3);