You have an error in the algorithm. The corrected iterate is:

But there is also a more subtle error.

XX:=x;

must be replaced by

XX:=Vector(x);

(or XX:=LinearAlgebra[Copy](x)). Otherwise x and XX will share the same memory, creating a mess in this case.

The program will work after that.

sort(collect(%, Nu));

Pascal := proc (n::posint)
local x,i,k,dd;
k:=length(binomial(n,floor(n/2)));
dd:=cat("%",2*k,"d");
for i from 0 to n do
  printf(cat(" "$(n-i)*k,dd$(i+1),"\n"), coeffs(expand((x+1)^i)));
end do end proc:
Pascal(12);

plot( [2,3,5]*~x^2, x=-2..2);

h:=(n,x)->1/2+1/2*tanh(n*x):
f:=proc(a,b,c,d,n)
   proc(x) c+ (d-c)*h(n,x-a)*h(n,b-x) end
   end:

The following function will look the same as the first of OP's function

plot( f(-80,40,11/9,19,50), -200..200);

Of course it is not constant:

f:=1+18*(sinh(9*x-9)-sinh(3*x-477))^2/(9*cosh(9*x-9)+cosh(3*x-477))^2;
limit(f,x=infinity),limit(f,x=-infinity);

eval(f,x=0);evalf(%);

     19.000000

To be able to plot, Digits must be increased.

Digits:=20;  # this is enough

plot(f,x=-200..200); # almost an "impulse"

You must use the proper syntax for Surface. E.g.

cone:= Surface(<r*cos(theta),r*sin(theta),2*a-r>,r=0..2*a, theta=0..2*Pi):

Flux( VectorField(<x, y, z>, cartesian[x, y, z]),  cone );

It should be not too difficult, following the next steps.

1. Find the 11 regions (connected components) R_1,...,R_11
and fix a point P_k in each of them
(each region is determined by 2, 3 or 4 inequalities of the form lines[i]<0 or lines[i]>0)
2.  OK:={}
3. For each S subset {P_1,...,P_11} having 4 elements do
    If S is admissible (i.e. in each halfspace there are exactly 2 points of S), then OK:=OK union {S}.

The set OK will contain all the desired configurations.

Here is a DirectSearch approch for finding a configuration. It is possible to find several configurations if we repeat the call.

lines := [-(1/2)*x+y, (1/2)*x+y, 1-x+(1/2)*y, 3-x-(1/2)*y]:
pl:=(i,j) -> eval(lines[i],[x=x[j],y=y[j]]):
cond1:=seq(seq(pl(i,j)^2>eps,i=1..4),j=1..4):
cond:= seq( op( [
      pl(i,1)*pl(i,2)*pl(i,3)*pl(i,4)>eps,
      min(pl(i,1),pl(i,2),pl(i,3),pl(i,4))<-eps,
      max(pl(i,1),pl(i,2),pl(i,3),pl(i,4))>eps  ])  ,
      i=1..4):
bounds:=seq(x[i]=-10..10,i=1..4), seq(y[i]=-10..10,i=1..4),  eps=0..10:
vars:=[seq(x[i],i=1..4),seq(y[i],i=1..4),eps]:
with(DirectSearch):
ds := Search(-eps, [cond, cond1,bounds], variables=vars):
pts:= [seq( [ds[2](i),ds[2](i+4)],i=1..4 )];

plot1:=plots[pointplot](pts,color=red,symbol=solidcircle,symbolsize=8):
plot2:=plots[implicitplot](lines,x=-10..10,y=-10..10):
plots[display](plot1,plot2,axes=none);

You will have to use IntegrationTools (or op()) for this.

J:=int( int( a(x)*b(y),x=-infinity..infinity), y=-infinity..infinity):

JJ:=expand(J);

with(IntegrationTools):

J1:=op(1,JJ); J2:=op(2,JJ):
int(int(GetIntegrand(J1)*GetIntegrand(J2),GetVariable(J1)=GetRange(J1)),GetVariable(J2)=GetRange(J2));

You should show how to reproduce this.

Otherwise, see my worksheet:

12x.mw

I get correct results.

Why should  int( fD(x)*fA(x), x=a..b ) be between 0 and 1? If you integrate the product of two densities, the result could be even infinity!

This is the first problem of the "Hundred-dollar, Hundred-digit Challenge problems":

https://en.wikipedia.org/wiki/Hundred-dollar,_Hundred-digit_Challenge_problems

The Maple solution was obtained by Robert Israel:

https://www.math.ubc.ca/~israel/challenge/challenge1.html

Edit: Mathematica seems to have better algorithms for computing oscillatory integrals.

Too many errors ...

neville:=proc(X::list,Y::list,x) # polynomial interpolation
local T,i,j,n;
n:=nops(Y)-1;
T:=Array(0..n, Y);
for j to n do
  for i from 0 to n-j do
  T[i] :=  normal( (T[i]*(X[j+i+1]-x) + T[i+1]*(x-X[i+1]) )/(X[j+i+1]-X[i+1]) )
od; od;
end:

Test:

f:=x->sin(3*x);
r:=neville([$1..10],[f(n)$n=1..10],x):
plot([f(x),r],x=1..10);

You were told that f is not a pdf unless d=0.

For d=0  f is a Negative binomial distribution which is implemented in Maple, so you can use it.

But you want d>0.

In this case, you probably want to mimic the moments defined for a pdf i.e.

M[k] = Sum( n^k * f(n), n=0..infinity);

But it can be shown that f(n) is equivalent (as n --> infinity) to K/n^2 where K is a nonzero constant.
Hence all the moments are infinite.