Problem:

Suppose you have a bunch of 2D data points which:

1. May include points with the same x-value but different y-values; and
2. May be unevenly-spaced with respect to the x-values.

How do you clean up the data so that, for instance, you are free to construct a connected data plot, or perform a Discrete Fourier Transform? Please note that Curve Fitting and the Lomb–Scargle Method, respectively, are more effective techniques for these particular applications. Let's start with a simple example for illustration. Consider this Matrix:

A := < 2, 5; 5, 8; 2, 1; 7, 8; 10, 10; 5, 7 >;

Consolidate:

First, sort the rows of the Matrix by the first column, and extract the sorted columns separately:

P := sort( A[..,1], output=permutation ); # permutation to sort rows by the values in the first column
U := A[P,1]; # sorted column 1
V := A[P,2]; # sorted column 2

We can regard the sorted bunches of distinct values in U as a step in a stair case, and the goal is replace each step with the average of the y-values in V located on each step.

Second, determine the indices for the first occurrences of values in U, by selecting the indices which give a jump in x-value:

m := numelems( U );
K := [ 1, op( select( i -> ( U[i] > U[i-1] ), [ seq( j, j=2..m ) ] ) ), m+1 ];
n := numelems( K );

The element m+1 is appended for later convenience. Here, we can quickly define the first column of the consolidated Matrix:

X1 := U[K[1..-2]];

Finally, to define the second column of the consolidated Matrix, we take the average of the values in each step, using the indices in K to tell us the ranges of values to consider:

Y1 := Vector[column]( n-1, i -> add( V[ K[i]..K[i+1]-1 ] ) / ( K[i+1] - K[i] ) );

Thus, the consolidated Matrix is given by:

B := < X1 | Y1 >;

Spread Evenly:

To spread-out the x-values, we can use a sequence with fixed step size:

X2 := evalf( Vector[column]( [ seq( X1[1]..X1[-1], (X1[-1]-X1[1])/(m-1) ) ] ) );

For the y-values, we will interpolate:

Y2 := CurveFitting:-ArrayInterpolation( X1, Y1, X2, method=linear );

This gives us a new Matrix, which has both evenly-spaced x-values and consolidated y-values:

C := < X2 | Y2 >;

Plot:

plots:-display( Array( [
        plots:-pointplot( A, view=[0..10,0..10], color=green, symbol=solidcircle, symbolsize=15, title="Original Data", font=[Verdana,15] ),
        plots:-pointplot( B, view=[0..10,0..10], color=red, symbol=solidcircle, symbolsize=15, title="Consolidated Data", font=[Verdana,15] ),
        plots:-pointplot( C, view=[0..10,0..10], color=blue, symbol=solidcircle, symbolsize=15, title="Spread-Out Data", font=[Verdana,15] )
] ) );

Sample Data with Noise:

For another example, let’s take data points from a logistic curve, and add some noise:

# Noise generators
f := 0.5 * rand( -1..1 ):
g := ( 100 - rand( -15..15 ) ) / 100:

# Actual x-values
X := [ seq( i/2, i=1..20 ) ];

# Actual y-values
Y := evalf( map( x -> 4 / ( 1 + 3 * exp(-x) ), X ) );

# Matrix of points with noise
A := Matrix( zip( (x,y) -> [x,y], map( x -> x + f(), X ), map( y -> g() * y, Y ) ) );

Using the method outlined above, and the general procedures defined below, define:

B := ConsolidatedMatrix( A );
C := EquallySpaced( B, 21, method=linear );

Visually:

plots:-display( Array( [
    plots:-pointplot( A, view=[0..10,0..5], symbol=solidcircle, symbolsize=15, color=green, title="Original Data", font=[Verdana,15] ),
    plots:-pointplot( B, view=[0..10,0..5], symbol=solidcircle, symbolsize=15, color=red, title="Consolidated Data", font=[Verdana,15]  ),
    plots:-pointplot( C, view=[0..10,0..5], symbol=solidcircle, symbolsize=15, color=blue, title="Spread-Out Data", font=[Verdana,15] )
] ) );

Generalization:

Below are more generalized custom procedures, which are used in the above example. These also account for special cases.

# Takes a matrix with two columns, and returns a new matrix where the new x-values are unique and sorted,
# and each new y-value is the average of the old y-values corresponding to the x-value.
ConsolidatedMatrix := proc( A :: 'Matrix'(..,2), $ )

        local i, j, K, m, n, P, r, U, V, X, Y:
  
        # The number of rows in the original matrix.
        r := LinearAlgebra:-RowDimension( A ):

        # Return the original matrix should it only have one row.
        if r = 1 then
               return A:
        end if:

        # Permutation to sort first column of A.
        P := sort( A[..,1], ':-output'=permutation ):       

        # Sorted first column of A.
        U := A[P,1]:

        # Corresponding new second column of A.
        V := A[P,2]:

        # Return the sorted matrix should all the x-values be distinct.
        if numelems( convert( U, ':-set' ) ) = r then
               return < U | V >:
        end if:

        # Indices of first occurrences for values in U. The element m+1 is appended for convenience.
        m := numelems( U ):
        K := [ 1, op( select( i -> ( U[i] > U[i-1] ), [ seq( j, j=2..m ) ] ) ), m+1 ]:
        n := numelems( K ):

        # Consolidated first column.
        X := U[K[1..-2]]:

        # Determine the consolidated second column, using the average y-value.
        Y := Vector[':-column']( n-1, i -> add( V[ K[i]..K[i+1]-1 ] ) / ( K[i+1] - K[i] ) ):

        return < X | Y >:

end proc:

# Procedure which takes a matrix with two columns, and returns a new matrix of specified number of rows
# with equally-spaced x-values, and interpolated y-values.
# It accepts options that can be passed to ArrayInterpolation().
EquallySpaced := proc( M :: 'Matrix'(..,2), m :: posint )

        local A, i, r, U, V, X, Y:

        # Consolidated matrix, the corresponding number of rows, and the columns.
        A := ConsolidatedMatrix( M ):
        r := LinearAlgebra:-RowDimension( A ):
        U, V := evalf( A[..,1] ), evalf( A[..,2] ):

        # If the consolidated matrix has only one row, return it.
        if r = 1 then
               return A:
        end if:

        # If m = 1, i.e. only one equally-spaced point is requested, then return a matrix of the averages.
        if m = 1 then
               return 1/r * Matrix( [ [ add( U ), add( V ) ] ] ):
        end if:

        # Equally-spaced x-values.
        X := Vector[':-column']( [ seq( U[1]..U[-1], (U[-1]-U[1])/(m-1), i=1..m ) ] ):

        # Interpolated y-values.
        Y := CurveFitting:-ArrayInterpolation( U, V, X, _rest ):    

        return < X | Y >:

end proc:

Worth Checking Out:

• The CurveFitting:-ArrayInterpolation() command
• The dataplot() command
• The plots:-pointplot() command
• The Statistics:-ScatterPlot() command
• The Interpolation package (new in Maple 2018)

Maple users frequently solve differential equations. If you want to use the results later in Maple, you need to deconstruct the solution, and then assign the functions -- something that isn't done automatically in Maple. We wrote a multi-purpose routine to help you out. For instance, suppose you solve a simple linear system of equations:

restart;

eqs := { x + y = 3, x - y = 1 };
soln := solve( eqs ); # { x = 2, y = 1 }
x, y; # plain x and y

To assign the values from the solution to the corresponding variables:

assign( soln );
x, y; # 2, 1

This won't work for solutions of differential equations:

restart;

sys := { D(x)(t) = y(t), D(y)(t) = -x(t), x(0) = 1, y(0) = 0 };
soln := dsolve( sys ); # { x(t) = cos(t), y(t) = -sin(t) }
assign( soln );
x(s), y(s); # plain x(s) and y(s)

To make this work, we wrote this multi-purpose routine:

restart;

# Type for a variable expression, e.g. x=5.
TypeTools:-AddType( 'varexpr', u -> type( u, 'And'('name','Non'('constant'))='algebraic' ) ):

# Type for a function expression, e.g. f(x)=x^2.
TypeTools:-AddType( 'funcexpr', u -> type( u, 'function'('And'('name','Non'('constant')))='algebraic' ) ):

# Procedure to assign variable and function expressions.
my_assign := proc( u :: {
        varexpr, 'list'(varexpr), 'rtable'(varexpr), 'set'(varexpr),
        funcexpr, 'list'(funcexpr), 'rtable'(funcexpr), 'set'(funcexpr)
}, $ )

        local F, L, R, V:       

        # Map the procedure if input is a data container, or apply regular assign(), where applicable.
        if not u :: {'varexpr','funcexpr'} then
               map( procname, u ):
               return NULL:
        elif u :: 'varexpr' then
               assign( u ):
               return NULL:
        end if:       

        L, R := lhs(u), rhs(u):
        F := op(0,L): 
        V := [ indets( L, 'And'( 'name', 'Non'('constant') ) )[] ]:    

        map( assign, F, unapply( R, V ) ):
        return NULL:

end proc:

# Example 1.

eqs := { x + y = 3, x - y = 1 };
my_assign( solve( eqs ) );
'x' = x, 'y' = y; # x=1, y=2

# Example 2.

unassign( 'x', 'y' ):
E := [ f(x,y) = x + y, g(x,y) = x - y ];
my_assign( E );
'f(u,v)' = f(u,v), 'g(u,v)' = g(u,v); # f(u,v)=u+v, g(u,v)=u-v

# Example 3.

sys := { D(x)(t) = y(t), D(y)(t) = -x(t), x(0) = 1, y(0) = 0 };
soln := dsolve( sys );
my_assign( soln ):
'x(s)' = x(s); # x(s)=cos(s)
'y(s)' = y(s); # y(s)=-sin(s)

Hi Primes Users,

We’re back with another tech support challenge that we wanted to share with you!

A customer had been having issues launching Maplets using the standard double-clicking method. This is a known issue that rarely occurs, but is being addressed for a future release. In the meantime, we were able to provide this person with a command-prompt-based way of opening the Maplet, and we thought it would be great to share in case you run into the same kind of problem.

After suggesting a few workarounds, our Team Lead was able to offer a command-prompt based way of solving the problem. Since command prompts are the target of batch scripts, which we had already used as a workaround for another issue, we just needed a way of programmatically creating scripts based on the command prompt code for each file.

Using various commands from the FileTools package (including a technique suggested by our Team Lead), we were able to put together code that takes all files in a particular folder (namely, a folder named “Maplets” on the Desktop), and creates a new batch script from a template that is based on the command prompt code (provided that the template is saved in the same Maplets folder under the file name “Maplet script”):

restart; with(FileTools): username := kernelopts(username):

directory := cat("C:\\Users\\", username, "\\Desktop\\Maplets"):

dir := ListDirectory(directory): dir := Basename~(dir):

main := cat(directory, "\\Maplet script.txt"): body := Import(main):


n := numelems(dir):

for i to n do

script := cat(directory, "\\launch ", dir[i], ".txt");

batch := cat(directory, "\\launch ", dir[i], ".bat");

newbody := StringTools:-Substitute(body, "name", dir[i]);

Export(script, newbody);

Rename(script, batch);

end do:


Script template:


if not "%minimized%"=="" goto :minimized

set minimized=true

start /min cmd /C "%~dpnx0"

goto :EOF

:minimized


"C:\Program Files\Maple 2018\bin.X86_64_WINDOWS\mapletviewer.exe" "C:\Users\%USERNAME%\Desktop\Maplets\name.maplet"

Before using the Maplet script:

After using the Maplet script:

If the appropriate executable is referenced, and the relevant file paths are adjusted accordingly, one should be able to adapt this process to other programs and their corresponding files.  In fact, any batch script that needs to be modified programmatically can be modified using these techniques.  Does anyone have other useful batch scripts that they’ve modified programmatically in Maple using similar techniques?

*(including dragging the files to the executable directly, which only seemed to work when the executable was in its original directory)

Hi MaplePrimes Users!

It’s your friendly, neighborhood tech support team; here to share some tips and tricks from issues we help users with on a daily basis.

A customer contacted us through a Help Page feedback form, asking how to add a row or column in a Matrix. The form came from the Row Operations help page, but the wording of the message suggested that the customer actually wanted to insert a new row or column altogether. Such manipulations can often be accomplished by a command in the ArrayTools package, but the only Insert command currently available is the one for Vectors and 1-D Arrays. Using the Concatenate command from that package, and various commands from the LinearAlgebra package (including the SubMatrix command), we were able to write two custom procedures to perform these manipulations:

InsertRow := proc (A::rtable, n::integer, v::Vector[row])
    local R, C, top, bottom;
    uses LinearAlgebra;
    R := RowDimension(A); C := ColumnDimension(A);
    top := SubMatrix(A, [1 .. n-1], [1 .. C]);
    bottom := SubMatrix(A, [n .. R], [1 .. C]);
    return ArrayTools:-Concatenate(1, top, v, bottom);
end proc:

InsertColumn := proc (A::rtable, n::integer, v::Vector[column])
    local R, C, left, right;
    uses LinearAlgebra;
    R := RowDimension(A); C := ColumnDimension(A);
    left := SubMatrix(A, [1 .. R], [1 .. n-1]);
    right := SubMatrix(A, [1 .. R], [n .. C]);
    return ArrayTools:-Concatenate(2, left, v, right)
end proc:

# test cases:

M := Matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]]):
v := Vector[row]([2, 2, 2]):
v2 := Vector[column]([2, 2, 2]):
seq(InsertRow(M, i, v), i = 1 .. 4);
seq(InsertColumn(M, i, v2), i = 1 .. 4);

We then reworked this problem using some handy indexing and construction notation that allows our previous procedures to save on the variable constructions and syntax:

InsertRow := proc( A :: rtable, V :: Vector[row], r :: posint )
    return < A[1..r-1,..]; V; A[r..-1,..] >:
end proc:

InsertColumn := proc( A :: rtable, V :: Vector[column], c :: posint )
    return < A[..,1..c-1] | V | A[..,c..-1] >:
end proc:

M := Matrix(3, 3, [seq(i, i = 1 .. 9)]);
A := convert(M, Array);
U := Vector[row]( [ a, b, c ] );
V := convert( U, 'Vector[column]' );
seq(InsertRow( A, U, i ), i=1..4);
seq(InsertColumn( A, V, i ), i=1..4);
seq(InsertRow( M, U, i ), i=1..4);
seq(InsertColumn( M, V, i ), i=1..4);