The Maple help page ?Matlab,setup describes what needs to be done before a link to the MATLAB engine can be established. I've pasted this page below.
Note that this statment is misleading: "If with(Matlab) returns an error, then you should follow the instructions in this document." Loading the Matlab package nolonger establishes the link to Matlab, as not all commands in this package need to call Matlab in order to compute. The first command that uses MATLAB will try to establish the link. The statement should instead say, "if Matlab[openlink]() returns an error, ...".
Here are the setup instructions:
__Configuring a Computer for Matlab__
The Maple-MATLAB link works with MATLAB version 7 (R14) on most platforms that
have Maple and MATLAB versions. Versions of Maple running on Macintosh, Compaq
True64, SGI IRIX, and IBM-AIX use MATLAB version 6.5 (R13). To determine if the
link works on a platform, use the following Maple command.
`Matlab/valid_os`();
For the Maple-MATLAB link to work properly, you may have to update your
configuration files. If the command with(Matlab) returns a list of function
names, then your setup is correct and no additional configuration is necessary.
If *Matlab[openlink]()* returns an error, then you should follow the instructions in this document.
Since setup is platform-dependent, follow the instructions for your machine
architecture.
**Windows 98 and ME**
The following instructions describe how to set the path and update the Windows
Registry. These instructions assume that MATLAB is installed on the C: drive in
a directory called MATLAB. Substitute the appropriate drive letter and
directory name if necessary.
1. Add the following line to the end of your Autoexec.bat file to specify
the correct version of MATLAB.
SET PATH=%PATH%;C:\MATLAB7;
2. To restart the computer, click the Start button, then Shutdown, and then
Restart.
3. Start Maple and enter the command with(Matlab). If this command returns a
list of function names, then the Maple-MATLAB link is working correctly. If
this command returns an error, proceed to the next step.
4. To open a DOS window, click the Start button, then Programs, then
Accessories, and then Command Prompt.
5. To change to the bin directory, enter the following command at the prompt
in the DOS window.
c: cd \MATLAB7\BIN\WIN32
6. To update the registry, enter the following command.
matlab /regserver
7. Start Maple and enter the command with(Matlab). If this command does not
return a list of function names, review the first six steps. If problems
persist, contact Maple Technical Support.
**Windows NT, 2000, and XP**
The following instructions describe how to set the path and update the Windows
Registry. These instructions assume that you are logged on as an administrator
and that MATLAB is installed on the C: drive in a directory called MATLAB.
Substitute the appropriate drive letter and directory name if necessary.
1. To open the System Window in the Control Panel, click the Start button,
then Settings, and then Control Panel.
2. Double-click System.
3. Specify user variables:
- For Windows NT, in the System Properties dialog box, select the
Environment tab.
- For Windows 2000, select the Advanced tab and click the Environment
Variables button. Look at the User Variables section.
- For Windows XP, select Start>Control Panel>System. Click the Advanced tab.
Double-click the Environment Variables button. Double-click Path.
You must specify the correct version of MATLAB for the PATH.
If there is an entry with Value PATH, then click that entry. Append the text
c:\matlab7\bin\win32 to the Value field and click OK.
Otherwise, add the text PATH to the Variable field, add the text
c:\matlab7\bin\win32 to the Value field and click OK.
4. To restart the computer, click the Start button, then Shutdown, and then
Restart.
5. Start Maple and enter the command with(Matlab). If this command returns a
list of function names, then the Maple-MATLAB link is working correctly. If
this command results in an error, proceed to the next step.
6. To open a DOS window, click the Start button, then Programs, then
Accessories, and then Command Prompt.
7. To change to the bin directory, enter the following command at the prompt
in the DOS window.
c: cd \MATLAB7\BIN\WIN32
8. To update the registry, enter the following command.
matlab /regserver
9. Start Maple and enter the command with(Matlab). If this command does not
return a list of function names, review the first six steps. If problems
persist, contact Maple Technical Support.
**Macintosh**
Location of MATLAB Installation
The location of the MATLAB installation must be set in the Info.plist file,
inside the Maple.app package.
1. Control-click the application icon (Maple 10.app) and select Show Package
Contents from the context menu.
2. In the packages contents, open the Contents/Info.plist folder.
3. Find the lines (ll. 123-124):
matlab.basedir
4. Between and , enter the base install directory of your
MATLAB installation. For example, if MATLAB is installed in
/Applications/MATLAB701, then the second line must be changed to
/Applications/MATLAB701.
5. Save the file and restart Maple.
Standard Worksheet
- Execute the Matlab_setup script found in
/Library/Frameworks/Maple.framework/Versions/10/bin.
Command-Line Maple
Some environment variables must be defined in the Maple script. The following
instructions describe how to edit this script. These instructions assume that
MATLAB is installed in /Applications/Matlab. Substitute the appropriate path if
necessary.
1. Remove the comment characters (#) from the lines that look like the
following example.
MATLAB=/Applications/matlab
export MATLAB
2. Locate your system type in the lines that follow the text in step 1.
Remove the comment characters (#) from the appropriate lines for your system.
The string "$MATLAB/extern/lib/$SYS" (where $SYS is your Macintosh platform)
must resolve to a valid directory where the MATLAB shared libraries are
located. The string "$MATLAB/bin" must resolve to a valid directory where the
MATLAB binaries are located.
**UNIX and Linux**
Some environment variables must be defined in the Maple script
$MAPLE/bin/maple. The following instructions describe how to edit this script.
Use your favorite text editor that preserves line breaks. These instructions
assume that MATLAB is installed in /usr/local/matlab. Substitute the
appropriate path if necessary.
1. Remove the comment characters (#) from all lines mentioning MATLAB for a
given platform, for example:
MATLAB=/usr/local/matlab
export MATLAB
2. Locate your system type in the lines that follow the text in step 1.
Remove the comment characters (#) from the appropriate lines for your system.
- The string "$MATLAB/extern/lib/$SYS" (where $SYS is your UNIX platform) must
resolve to a valid directory where the MATLAB shared libraries are located.
To locate the string, simply search for "$MATLAB/extern/lib".
- The string "$MATLAB/bin" must resolve to a valid directory where the MATLAB
binaries are located.
- Remove the comment characters (#) from the lines LD_LIBRARY_PATH = ... and
PATH = ... for your system type.

This bit of code aught to work in general, but runs into the problem that gcd(D,D/(1+D)) balks at the fact that D/(1+D) is not a polynomial in your sample matrix. Note that this code can be greatly simplified if it expected a Matrix instead of matrix.
ngcd := proc()
if nargs = 1 then
args[1];
elif type(nargs,'odd') then
ngcd(seq(gcd(args[i],args[i+1]),i=1..trunc(nargs/2),2),args[nargs]);
else
ngcd(seq(gcd(args[i],args[i+1]),i=1..trunc(nargs/2),2));
end if;
end proc:
p := proc( M )
local m, n, B, i, j, row, v;
(m,n) := op(map(rhs,[op(2,eval(M))]));
B := array(1..m, 1..n);
for i from 1 to m do
row := seq(normal(M[i,j]),j=1..n);
v := lcm(row)/ngcd(row);
for j from 1 to n do
B[i,j] := v * M[i,j];
end do;
end do;
return eval(B);
end proc:
M := array( [[ D+D^2, D^2, D], [1, (D+D^2)/(1+D), D/(1+D)] ] );
B := p(M);
#error non-polynomial in gcd
M := array( [[ D+D^2, D^2, D], [1, (D+D^2)/(1+D), D] ] );
B := p(M); #ok

Could you provide the example that is causing the memory problem? How much memory is being used?
The following example seems ok:
M := rtable(1..100,1..100,(i,j)->i*x+j*y*I);
map(Re,M);

There is no command line argument or api command that will specifically disable the OpenMaple splash screen.