Script Interface

The purpose of this project is to provide a scripting object that allows external scripting languages to be incorporated in VEE programs. As with the Matlab Object, the Scripting object accepts arbitrary inputs and outputs and provides an editor for scripts. Unlike the Matlab object this can accept scripts in any supported language, as selected by a leading shebang.

Script

Why is this desired?

There are various ways in which this may be accomplished.

Toolkit << maybe low effort but clumsy and perhaps fragile
VEE Handler << moderate effort and very low risk but poor performance
External Handler << moderate effort and very low risk but maybe most flexible and is backward and sideways compatible
Matlab spoofing << high effort and high risk but most elegant

It seems very likely that this project will come down to choosing between options 3 and 4, which would be a choice between elegance and (maybe) functionality. Or perhaps both methods can be implemented since there will be a major overlap.

A suggested architecture would allow for utilization in other environments.

Communicating between VEE and Python
In any scenario it is necessary to pass data back and forth between VEE and Python

What version of Python should we use?

Packaging the embeddable server

Choose a Protocol

Why do we need a scripting interface?

It is perfectly fair and expected to ask "WHY?" do we need such a thing in the first place. The short answer is to extend the capabilities of VEE by providing a link to the modern world.

The math libraries in VEE date back to the late 1980's. While they were leading-edge at that time they have not been updated since are are showing their age. The existing libraries are 32-bit. Only a few accept complex arguments. Today there are a vast array of pure math, scientific, and engineering libraries available to be used- but no suitable link exists in VEE.

As it stands, VEE allows the import of libraries written in VEE or that have straight C interfaces via the Import Library object.

importing libraries

This capability allows the end user to transparently use the imported functions along side all the built-in functions. They show up in the Function Browser and can be used as desired in any expression, such as in the formula object

formula object

function browser

However this capability is limited to function written in VEE or whose import is available through a straight C interface. While straight C was common in the past, most modern libraries and tools do not provide such an interface. In some cases dotNet is available and can be used, but it tends to be clumsy and poorly documented.

To address this basic limitation and provide access to a broad range of functionality the MATLAB Object was introduced.

Matlab Object

The Matlab Object provides a clean link to Matlab and hence to the broad array of Matlab libraries. In function it behaves just like a Formula object except that the expression (script) is interpreted by Matlab rather than the internal VEE engine.

Since the modern world has largely moved from compiled tools (C#, etc) to interpreted tools ( Python, etc) it becomes highly desirable to develop a more general purpose link to scripting, but while maintaining the ease of use of the Matlab Script object.

Toolkit (All VEE)

Implement any needed interface directly in VEE. For each scripting environment create a Toolkit, perhaps similar to the Excel or Database Toolkits.

scripting Toolkit menu

The end user would string together the needed tools for each supported scripting language. It seems likely that in each case there would be similar needed functions such as

Initialization - load needed libraries, set variables, etc.
send variable
send script
execute
get data
quit

This would almost certainly succeed since all that is needed is to implement whatever API each scripting language has available. Like addons such as VEEOS, this Toolkit could be added to the menus. Since this method requires no changes to VEE itself it can be implemented by a third-party. An example is the Excel tookit.

excel toolkit menu

However, like these addons are a programming aid and the end user faces the significant task of understanding the needed details which are likely different for each supported language. The usage model would be very different from Matlab Object wherein no VEE programming is needed- just specify the script and variables.

This is likely the lowest risk and easiest to implement from a VEE perspective, but also the clumsiest in that it leaves many programming details to the end user.

Disadvantages

VEE is quite general purpose but does not support modern API's so it may be significant work interfacing to some engines
End user has significant programming task

VEE Handler

Create an intermediary function or set of functions within VEE that communicates with scripting languages.

The underlying functionality is provided by a function that hides the scripting Toolkit created for the Toolkit method. The end user sees this function instead of having to deal with the various pieces of the Toolkit.
underlying scripting function

The end user could call this function directly or it could placed behind a user object that includes a simple script editor. The script editor could be as simple as a text constant or as useful as a "real" editor included as a dotnet object.
scripting object icon view

NOTE that since the function has exactly one input and one output for variables, if multiple variables are needed (probably most cases) they would need to be a single data structure. To do this it would make sense to use a record and have the variables be identified as field names with the accompanying values be of course the field values.

Record example- define two variables "A" and "B"
(Record
 (schema
  (numFields 2)
  (fieldName "A"
   (type Text)
  )
  (fieldName "B"
   (type Real64)
  )
 )
 (data 
  (record
   ( "A" "Text field")
   ( "B" 1.25)
  )
)
)

Similarly, return variables would be part of the same input data structure, with dummy values as appropriate.

INPUT #################
(Record
 (schema
  (numFields 3)
  (fieldName "A"
   (type Text)
  )
  (fieldName "B"
   (type Real64)
  )
  (fieldName "X"
   (type Text)
  )
 )
 (data 
  (record
   ( "A" "Text field")
   ( "B" 1.25)
   ( "X" "dummy value")
  )
)
)

OUTPUT ##################
(Record
 ...
   ( "X" "calculated value")
  )
)
)

This could be implemented with any version of VEE and with no internal VEE knowledge. Hence it is very low risk and mostly requires only knowledge of whatever language in which the handler is written. The handler function would examine the input record for variable names and values and then call the required Toolkit functions for the appropriate scripting language.

UPDATE: 20210220 -- initial version built using File IO and portable Python is successful. This is the least sophisticated handler and is slow but it works and should be fairly robust.

Only Python is implemented but the structure is such that any language could be added
The initial (shebang) line is read as intended and used to determine which language and executable to utilize. This way if python is installed then the end user can simply use #!python since the python executable will already be in the default path. If multiple pythons are available such as portable versions then the full path can be used, as in #!//server/share/apps/python3.8/python to select python independently for each script.
Variables are identified by the incoming record field names and created using the field values.
Variables are passed to Python by prepending lines to the given script.
Variables are retrieved by appending file.write(str(variable)) lines to the script, where each variable is written in ASCII format to a file whose name is that of the variable. Then in VEE these files are read in the same type as the original values. these value are used to update the arguments record and that record is outputted.
```
define variables
a = 45
B = 16

actual script
B= a + 2 * B

write variables to files
f = open("a", "w")
f.write(str(a))
f.close()
f = open("B", "w")
f.write(str(B))
f.close()

quit python
exit
```
After the augmented script is written to a file, python is called in order to run that script.
After retrieving all values python is closed and all the temporary files are deleted

python script handler

Profiling shows that, not surprisingly, the time sinks are writing files and especially instantiating and running Python. BUT at the same time this method works and shows feasibility.

The next step in complexity is to maintain the same structure but instead of using file IO, look at embedded Python. The advantage here will be performance since one should be able to instantiate Python and leave it running, calling functions as needed. Odds are a C wrapper would be involved to create a DLL that can be referenced as a Compiled Library.

Advantages

No need to read/write container format since using VEE directly
Relatively quick and easy

Disadvantages

clumsy since user has access to exactly one input so must create a single piece of data. This since the needed function is fixed and therefore the programmer cannot change it. There would be no problem if only a predefined schema was needed, but that is not the case.
Each new engine will have a unique interface that needs to be developed within VEE, mostly from scratch.

Create External Handler

Write an external handler that communicates with scripting languages and with VEE. VEE data and scripts are sent as VEE containers. The handler parses the containers, formats the script and data as needed, a executes the appropriate scripting language, then passes data back to VEE. By choosing an external language that is well supported and widely used, the task of incorporating new scripting engines could be made fairly simple.

The underlying functionality is provided by a function.
underlying scripting function

This could be implemented in multiple ways, including

A VEE function that communicates with the handler via sockets. the handler would be started and would remain running as needed as an external process.
A compiled function that is part of the handler and is imported. The handler would be launched and run as needed and as an imported function would be part of the running VEE program.
A compiled function that acts as intermediary to the handler. This could be a small C wrapper to a handler written in Python

The handler is the key here. It would communicate with both VEE and whatever scripting language is chosen. In the general case a standard shebang would be added as the first line of any script. This would identify which scripting language is used by the script. The handler would use this line to determine which of the supported scripting languages would be called. It would seem logical to create the handler in Python, since Python is very well supported and can interface to most anything.

The handler itself would need to read and write VEE container format, but that is stable and well defined structured ASCII so should be reasonably simple to implement. Standard parsing tools would likely be very close to working with minor modifications. Since any VEE data can be placed in container format this would be very general purpose. Examples of VEE containers include:

Scalar complex ##########################
(Complex
 (data  (34, -56))
)

Real64 array ############################
(Real64
 (numDims 1)
 (size 3)
 (data 
   [ 0.001251220703125 0.563568115234375 0.19329833984375 ]
 )
)

NOTE that since the function has exactly one input and one output, if multiple variables are needed (probably most cases) they would need to be a single data structure. To do this it would make sense to have the variables be identified as field names with the accompanying values be of course the field values.

Record example- define two variables "A" and "B"
(Record
 (schema
  (numFields 2)
  (fieldName "A"
   (type Text)
  )
  (fieldName "B"
   (type Real64)
  )
 )
 (data 
  (record
   ( "A" "Text field")
   ( "B" 1.25)
  )
)
)

Similarly, return variables would be part of the same input data structure, with dummy values as appropriate.

INPUT #################
(Record
 (schema
  (numFields 3)
  (fieldName "A"
   (type Text)
  )
  (fieldName "B"
   (type Real64)
  )
  (fieldName "X"
   (type Text)
  )
 )
 (data 
  (record
   ( "A" "Text field")
   ( "B" 1.25)
   ( "X" "dummy value")
  )
)
)

OUTPUT ##################
(Record
 ...
   ( "X" "calculated value")
  )
)
)

clumsy since user has access to exactly one input so must create a single piece of data. This since the needed function is fixed and therefore the programmer cannot change it. There would be no problem if only a predefined schema was needed, but that is not the case.

NOTE that if one had sufficient knowledge of VEE data structures it should be possible to pass data by reference instead of container format. However the reverse-engineering needed is significant and may not be worth the trouble compared to using container format.

The primary advantage of this method would be in the great support and widespread usage of Python. Once the data is in python ( variables and the script to run) there are likely tools already developed to link with Octave, Matlab, Spice and all sorts of other tools.

Conceptual Implementation:

Interface to VEE via
- C DLL linked to VEE as a Compiled Function library
  - This could be very simple and highly portable
  - if we have a small C DLL with a single function then the def file becomes trivial, looking something like
    char* runScript(char* VEErecord)
  - On the VEE side one can create a single record to pass with "script" field being the script along with the various variable fields. Converting that to a container and concatenating yields a single string that can be passed by reference with one call to runScript(). The C DLL processes as needed (probably just pass through to Python) then returns a single string.
  - the resulting DLL and def file can be embedded directly in a VEE function so no external dependencies.
  - The entire VEE function is a handful of parts that looks something like
  - this would then be usable outside of VEEOS, though inside veeos the DLL could be loaded by default
- Dotnet library written directly in Python - less portable and maybe convoluted
In either case the interface to VEE could consist of a very small number of functions (maybe just ONE) that would be very similar to the existing Matlab interface
- Initialize - instantiates the handler process
- putContainer - transmits both the arguments and the script in the form of a VEE Record container. One field for each argument and one field ("script") with text array for the script itself. The handler creates variables as needed and also reads the script.
- execute - instructs the handler to proceed. This may not be necessary at all- it could proceed as soon as the above container is sent. The handler gets data back from the chosen engine and creates an output in VEE record container format
- getContainer - gets arguments back as VEE record
The interface could perhaps even be as small as a single straight-C function executeScript() which is loaded as a DLL and does all the work, returning the result.
If that is the case then all that's needed on the VEE side is a DLL import that can be done in veerc, as in
clib mlib VEEROOT/veeos/lib/mLib.dll VEEROOT/veeos/lib/mLib.h
There would be a minimal VEE code (a UF and a UO wrapper) around this since all the real work would be done by the handler.

NOTE that the script is already text and the variables become text also when written as a record, so what likely makes sense for this method is to combine both into a single record, as in

(Record
(schema
(numFields 3)
(fieldName "script"
(type Text)
(numDims 1)
(size 8)
)
(fieldName "A"
(type Real32)
)
(fieldName "B"
(type Real32)
)
)
(data 
(record
( "script"
[ "#! Python" "prices={\'apple\':A,\'banana\':B}" "my_purchase={" " \'apple\':1," " \'banana\':6}" "grocery_bill = sum (prices[fruit]*my_purchase[fruit]" " for fruit in my_purchase)" "print(\'I owe\'first+\" \"+last+\'$%.2f % grocery_bill)" ]
)
( "A" 0.4000000059604645)
( "B" 0.6000000238418579)
)
)
)

AND while this multi-line form is the default and is nicely human readable, it likely makes more sense to remove the EOL's and make this a single string. That's likely easier for parsing and is also appropriate as a single function argument. On the VEE side either form works fine.

(Record (schema (numFields 3) (fieldName "script" (type Text) (numDims 1) (size 8) ) (fieldName "A" (type Real32) ) (fieldName "B" (type Real32) ) ) (data (record ( "script" [ "#! Python" "prices={\'apple\':A,\'banana\':B}" "my_purchase={" " \'apple\':1," " \'banana\':6}" "grocery_bill = sum (prices[fruit]*my_purchase[fruit]" " for fruit in my_purchase)" "print(\'I owe\'first+\" \"+last+\'$%.2f % grocery_bill)" ] ) ( "A" 0.4000000059604645) ( "B" 0.6000000238418579) )))

Note that the record container format is very regular, but will ant to look at all the cases except (probably) waveform and spectrum. Accordingly a simple VEE program is provided that writes a record with all datatypes to a file, but with (human readable) and without (more easily parsed?) EOL. These examples have Scalar,1D and 2D arrays.

NOTE that the ideal here may be to pass the entire record with both script and variables as a single string, but return just the variables. Probably best to do all the real work in python and have the C code do little more than just pass strings. It is likely feasible to write the DLL in Python also but since that needs to include the Python engine it would likely be a very large DLL compared to a few lines of C code. As noted earlier a very small DLL is desirable since it can easily be placed inside of VEE.

WHEN WRITING the container format will want to be sure and place the needed white space - primarily between words and before/after brackets. Extra whitespace is OK.

A possible follow-on would be to integrate a simple script-savvy text editor into VEE and present that on the face of a runScript UO.

ADVANTAGES of this method include

This would be completely in parallel with the existing Matlab scripting
No limitations on datatypes
Runtime script modification the same as for formula objects, unlike the Matlab script object where the script is fixed at program time
backward compatible to all versions of VEE if a simple C-like DLL us used

Matlab Spoofing and the Matlab Object

A handler provides a spoofed version of the Matlab interface and VEE preferences are set to use this interface. In VEE the Matlab Object becomes a general Script object but is otherwise exactly the same. For the end user this becomes extremely clean and simple since it works exactly like the native Matlab object. The user becomes free to add any inputs and outputs desired. For backward compatibility also need to be able to call both the OEM Matlab and full Matlab from the spoof and Matlab should be default engine.

Script

With VEE 7 the interface is a simple set of COM functions that are imported when the Matlab object is run. Note that when VEE is first installed this library is not registered, but it is registered when the Matlab object is first run.

VEE 7 references

VEE 7 methods

There would still need to be a handler that determines which script engine to run and handles communications.

What is needed along with a handler is the spoofed interface

Determine the exact functionality of the Matlab interface functions (looks very simple and is probably documented by Mathworks)
Create library that spoofs Matlab
Determine appropriate install. Could replace the built-in Matlab, or maybe better be placed in the registry so that VEE can choose to use it in default preferences.

With VEE 9 this becomes much more involved.

It is quickly seen that there is NOT an obvious load of external libraries as with VEE 7. Digging further shows that even a full install of Matlab does not register any useful-looking libraries. However the link is established somehow and is almost certainly a function library of some sort.

Firing up the internal Matlab that comes with VEE and looking at the attached DLL's (procexp) shows quite a few


awt.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\awt.dll
boost_date_time-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_date_time-vc90-mt-1_40.dll
boost_filesystem-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_filesystem-vc90-mt-1_40.dll
boost_iostreams-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_iostreams-vc90-mt-1_40.dll
boost_regex-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_regex-vc90-mt-1_40.dll
boost_signals-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_signals-vc90-mt-1_40.dll
boost_system-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_system-vc90-mt-1_40.dll
boost_thread-vc90-mt-1_40.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\boost_thread-vc90-mt-1_40.dll
comcli.dll	comcli	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\comcli.dll
ctfarchiver.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\ctfarchiver.dll
ctfrt.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\ctfrt.dll
ctfrtcrypto.dll	ctfrtcrypto	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\ctfrtcrypto.dll
dservices.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\dservices.dll
fontmanager.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\fontmanager.dll
graphics_util.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\graphics_util.dll
hg.dll	hg	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\hg.dll
hgbuiltins.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\hgbuiltins.dll
hgdatatypes.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\hgdatatypes.dll
hgutils.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\hgutils.dll
hpi.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\hpi.dll
icudt42.dll	ICU Data DLL	IBM Corporation and others	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\icudt42.dll
icuin42.dll	IBM ICU I18N DLL	IBM Corporation and others	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\icuin42.dll
icuio42.dll	IBM ICU I/O DLL	IBM Corporation and others	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\icuio42.dll
icuuc42.dll	IBM ICU Common DLL	IBM Corporation and others	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\icuuc42.dll
iqm.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\iqm.dll
ir_xfmr.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\ir_xfmr.dll
java.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\java.dll
jmi.dll	jmi	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\jmi.dll
jproxy.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\jproxy.dll
jvm.dll	Java HotSpot(TM) Client VM	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\client\jvm.dll
libexpat.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libexpat.dll
libhdf5.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libhdf5.dll
libifcoremd.dll	Intel(r) Fortran Compiler RTL (thread-safe) 	Intel Corporation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libifcoremd.dll
libmat.dll	libmat	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmat.dll
libmex.dll	libmex	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmex.dll
libmmd.dll	Math Library for Intel(r) Compilers (thread-safe)	Intel Corporation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmmd.dll
libmwamd.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwamd.dll
libmwbinder.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwbinder.dll
libmwblas.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwblas.dll
libmwbridge.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwbridge.dll
libmwbuiltins.dll	libmwbuiltins	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwbuiltins.dll
libmwcholmod.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwcholmod.dll
libmwcolamd.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwcolamd.dll
libmwcsparse.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwcsparse.dll
libmwfl.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwfl.dll
libmwgui.dll	gui	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwgui.dll
libmwhardcopy.dll	hardcopy	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwhardcopy.dll
libmwi18n.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwi18n.dll
libmwlapack.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwlapack.dll
libmwma57.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwma57.dll
libmwmathcore.dll	mathcore	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathcore.dll
libmwmathelem.dll	mathelem	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathelem.dll
libmwmathlinalg.dll	mathlinalg	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathlinalg.dll
libmwmathrng.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathrng.dll
libmwmathutil.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathutil.dll
libmwmathxps.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwmathxps.dll
libmwMATLAB_res.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwMATLAB_res.dll
libmwompwrapper.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwompwrapper.dll
libmwresource_core.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwresource_core.dll
libmwrookfastbp.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwrookfastbp.dll
libmwservices.dll	libmwservices	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwservices.dll
libmwspmatrix.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwspmatrix.dll
libmwspqr.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwspqr.dll
libmwumfpack.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmwumfpack.dll
libmx.dll	libmx	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libmx.dll
libuij.dll	libuij	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libuij.dll
libut.dll	libut	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\libut.dll
m_dispatcher.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_dispatcher.dll
m_interpreter.dll	m_interpreter	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_interpreter.dll
m_ir.dll	m_ir	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_ir.dll
m_parser.dll	m_parser	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_parser.dll
m_pcodegen.dll	m_pcodegen	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_pcodegen.dll
m_pcodeio.dll	m_pcodeio	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\m_pcodeio.dll
mclbase.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mclbase.dll
mclmcr.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mclmcr.dll
mclmcrrt7_15.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\runtime\win32\mclmcrrt7_15.dll
mcos.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mcos.dll
mcr.dll	mcr	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mcr.dll
mlautoregister.dll	mlautoregister	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mlautoregister.dll
mlint.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mlint.dll
mlutil.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mlutil.dll
mpath.dll	mpath	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mpath.dll
msvcr71.dll	Microsoft® C Runtime Library	Microsoft Corporation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\msvcr71.dll
mtok.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mtok.dll
mvm.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mvm.dll
mwoles05.dll	MATLAB Automation Server	The MathWorks	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\mwoles05.dll
nativehg.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\nativehg.dll
nativejava.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\nativejava.dll
nativejava_services.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\nativejava_services.dll
nativejmi.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\nativejmi.dll
nativeservices.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\nativeservices.dll
net.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\net.dll
nio.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\nio.dll
profiler.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\profiler.dll
tbb.dll	Threading Building Blocks library	Intel Corporation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\tbb.dll
tbbmalloc.dll	Scalable Allocator library	Intel Corporation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\tbbmalloc.dll
udd.dll	udd	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\udd.dll
udd_mi.dll	udd_mi	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\udd_mi.dll
uinone.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\uinone.dll
uiw.dll	uiw	The MathWorks Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\uiw.dll
verify.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\verify.dll
xerces-c_2_7.dll	Shared Library for Xerces-C Version 2.7.0	Apache Software Foundation	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\xerces-c_2_7.dll
xmlcore.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\xmlcore.dll
xsd_binder.dll			C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\xsd_binder.dll
zip.dll	Java(TM) Platform SE binary	Sun Microsystems, Inc.	C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\sys\java\jre\win32\jre\bin\zip.dll
zlib1.dll	zlib data compression library		C:\Program Files (x86)\Keysight\VEE Pro 9.3\matlab\bin\win32\zlib1.dll

A full Matlab install (NOTE that one needs 32-bit version) includes a large number of DLL's


acml.dll
activation.dll
asynciocore.dll
asyncioimpl.dll
boost_date_time-vc90-mt-1_40.dll
boost_filesystem-vc90-mt-1_40.dll
boost_graph-vc90-mt-1_40.dll
boost_iostreams-vc90-mt-1_40.dll
boost_math_c99-vc90-mt-1_40.dll
boost_math_c99f-vc90-mt-1_40.dll
boost_math_c99l-vc90-mt-1_40.dll
boost_math_tr1-vc90-mt-1_40.dll
boost_math_tr1f-vc90-mt-1_40.dll
boost_math_tr1l-vc90-mt-1_40.dll
boost_prg_exec_monitor-vc90-mt-1_40.dll
boost_program_options-vc90-mt-1_40.dll
boost_regex-vc90-mt-1_40.dll
boost_serialization-vc90-mt-1_40.dll
boost_signals-vc90-mt-1_40.dll
boost_system-vc90-mt-1_40.dll
boost_thread-vc90-mt-1_40.dll
boost_unit_test_framework-vc90-mt-1_40.dll
boost_wave-vc90-mt-1_40.dll
boost_wserialization-vc90-mt-1_40.dll
comcli.dll
dastudio.dll
dastudio_hg.dll
dirmonitor.dll
dotnetcli.dll
dotnetcli_loader.dll
DXEnumerator.dll
EasyHook32.dll
emf.dll
glee.dll
glee2.dll
glee2_mi.dll
glee2_qt.dll
glee_util.dll
glren.dll
gluegen-rt.dll
gmp.dll
graphics_util.dll
gsdll32.dll
handle_graphics.dll
hd424m.dll
hg.dll
hgbuiltins.dll
hgdatatypes.dll
hgutils.dll
hm424m.dll
icudt42.dll
icuin42.dll
icuio42.dll
icule42.dll
icuuc42.dll
instutil.dll
iqm.dll
ir_interp.dll
ir_xfmr.dll
JavaAccessBridge.dll
JAWTAccessBridge.dll
jmi.dll
jogl.dll
jogl_awt.dll
Jp2Adapter.dll
kdu_a63R.dll
kdu_v63R.dll
libaudiodevice.dll
libcdf.dll
libeng.dll
libexpat.dll
libfftw3.dll
libfftw3f.dll
libfftw3i.dll
libGctp.dll
libhdf5.dll
libhdf5_hl.dll
libhdfeos.dll
libifcoremd.dll
libifportmd.dll
libiomp5md.dll
libmat.dll
libmex.dll
libmmd.dll
libmwamd.dll
libmwarpack.dll
libmwbinder.dll
libmwblas.dll
libmwblascompat32.dll
libmwbridge.dll
libmwbuiltins.dll
libmwchart.dll
libmwcholmod.dll
libmwcli.dll
libmwcolamd.dll
libmwcsparse.dll
libmwdastudio_res.dll
libmwfftw.dll
libmwfl.dll
libmwglee_res.dll
libmwglue2_res.dll
libmwgui.dll
libmwhardcopy.dll
libmwhghardcopy.dll
libmwi18n.dll
libmwlapack.dll
libmwma57.dll
libmwmathcgeo.dll
libmwmathcore.dll
libmwmathdsp.dll
libmwmathelem.dll
libmwmathlinalg.dll
libmwmathrng.dll
libmwmathspec.dll
libmwmathutil.dll
libmwmathxps.dll
libmwMATLAB_res.dll
libmwompwrapper.dll
libmwqhull.dll
libmwresource_core.dll
libmwrookfastbp.dll
libmwsavevars.dll
libmwservices.dll
libmwSimulinkTypes_res.dll
libmwSL_SERVICES_res.dll
libmwspmatrix.dll
libmwspqr.dll
libmwumfpack.dll
libmx.dll
libnetcdf.dll
libpng12.dll
libportaudio.dll
libtiff.dll
libuij.dll
libut.dll
m3i.dll
m3i_mi.dll
mcos.dll
mcos_factory.dll
mcr.dll
mkl.dll
mlautoregister.dll
mlib.dll
mlint.dll
mllapack.dll
mlutil.dll
MMCodecChooser.dll
mmreadercore.dll
mmreaderudd.dll
mpath.dll
mpfr.dll
msvcp71.dll
msvcr71.dll
mtok.dll
multimedia.dll
mvm.dll
mwdx.dll
mwoles05.dll
m_dispatcher.dll
m_interpreter.dll
m_ir.dll
m_parser.dll
m_pcodegen.dll
m_pcodeio.dll
nativecmdwin.dll
nativedirmonitor.dll
nativehg.dll
nativei18n.dll
nativejava.dll
nativejava_services.dll
nativejmi.dll
nativelex.dll
nativelmgr.dll
nativemat.dll
nativemlint.dll
nativeservices.dll
nativewindows.dll
osg65-osg.dll
osg65-osgDB.dll
osg65-osgFX.dll
osg65-osgGA.dll
osg65-osgManipulator.dll
osg65-osgParticle.dll
osg65-osgShadow.dll
osg65-osgSim.dll
osg65-osgTerrain.dll
osg65-osgText.dll
osg65-osgUtil.dll
osg65-osgViewer.dll
osgserver.dll
ot11-OpenThreads.dll
PreviewWindow.dll
profiler.dll
Qt3Support4.dll
QtCore4.dll
QtGui4.dll
QtNetwork4.dll
QtOpenGL4.dll
QtSql4.dll
QtSvg4.dll
QtWebKit4.dll
QtXml4.dll
QtXmlPatterns4.dll
refblas.dll
reflapackpt.dll
rxtxSerial.dll
servicesproxy.dll
sl_services.dll
sl_types.dll
sl_utility.dll
tamimframe.dll
tamimframemex.dll
tammex.dll
tamobjsys.dll
tamutil.dll
tbb.dll
tbbmalloc.dll
udd.dll
udd_mi.dll
uinone.dll
uiw.dll
VideoDeviceChooser.dll
VideoFormatInfo.dll
WindowsAccessBridge.dll
xerces-c_2_7.dll
xmlcore.dll
xmlxrcsc.dll
ziparchiver.dll
zlib1.dll

but looking at what VEE attaches when it runs a script shows a relatively small number of DLL's. these would seem a good starting point seeing what is necessary for the spoof.


boost_date_time-vc90-mt-1_40.dll			C:\apps\MATLAB\R2011a\bin\win32\boost_date_time-vc90-mt-1_40.dll
boost_filesystem-vc90-mt-1_40.dll			C:\apps\MATLAB\R2011a\bin\win32\boost_filesystem-vc90-mt-1_40.dll
boost_signals-vc90-mt-1_40.dll			C:\apps\MATLAB\R2011a\bin\win32\boost_signals-vc90-mt-1_40.dll
boost_system-vc90-mt-1_40.dll			C:\apps\MATLAB\R2011a\bin\win32\boost_system-vc90-mt-1_40.dll
boost_thread-vc90-mt-1_40.dll			C:\apps\MATLAB\R2011a\bin\win32\boost_thread-vc90-mt-1_40.dll
icudt42.dll	ICU Data DLL	IBM Corporation and others	C:\apps\MATLAB\R2011a\bin\win32\icudt42.dll
icuin42.dll	IBM ICU I18N DLL	IBM Corporation and others	C:\apps\MATLAB\R2011a\bin\win32\icuin42.dll
icuio42.dll	IBM ICU I/O DLL	IBM Corporation and others	C:\apps\MATLAB\R2011a\bin\win32\icuio42.dll
icuuc42.dll	IBM ICU Common DLL	IBM Corporation and others	C:\apps\MATLAB\R2011a\bin\win32\icuuc42.dll
libeng.dll	libeng	The MathWorks Inc.	C:\apps\MATLAB\R2011a\bin\win32\libeng.dll
libexpat.dll			C:\apps\MATLAB\R2011a\bin\win32\libexpat.dll
libmwfl.dll			C:\apps\MATLAB\R2011a\bin\win32\libmwfl.dll
libmwi18n.dll			C:\apps\MATLAB\R2011a\bin\win32\libmwi18n.dll
libmwMATLAB_res.dll			C:\apps\MATLAB\R2011a\bin\win32\libmwMATLAB_res.dll
libmwresource_core.dll			C:\apps\MATLAB\R2011a\bin\win32\libmwresource_core.dll
libmx.dll	libmx	The MathWorks Inc.	C:\apps\MATLAB\R2011a\bin\win32\libmx.dll
libut.dll	libut	The MathWorks Inc.	C:\apps\MATLAB\R2011a\bin\win32\libut.dll
zlib1.dll	zlib data compression library		C:\apps\MATLAB\R2011a\bin\win32\zlib1.dll234

A quick experiment shows that a real Matlab install is registered in
HKLM\SOFTWARE\WOW6432Node\MathWorks\MATLAB\7.12
and has a single key key
MATLABROOT pointing to install directory C:\apps\MATLAB\R2011a

RegistryEntries

VEE's Default Preferences looks at these registry entries to determine choices.

add spoof choice

Hence the "spoof" install could be as simple as placing the bits and creating a single registry entry.


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Mathworks]

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Mathworks\Matlab]

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Mathworks\Matlab\spoof]
"MATLABROOT"="c:\\apps\\scriptInterface"

VEE would then recognize and link to it. Note that an empty directory will not work, but all that is needed for the Default Preferences choice is a file <location>\bin\win32\MATLAB.exe. Note that this can be a dummy file. What's left is to look at what resources "MATLAB.exe" needs to have in order for VEE to link to it. Note that the real "MATLAB.exe" is a small file so almost certainly essentially a header. The real work is probably a script engine incorporating files such as mcr.dll. This will need to be determined for a successful spoof.

What would be best would be to examine VEE source for the above link and for the actual interfacing to Matlab, though there may also be good information in Mathworks' External API documentation for R2011A (aka 7.12) as is used inside VEE. Mathworks website has versions back to 2015b. VEE ships with 2011a, but a simple test would tell us if it links to 2015b or later.

Since the spoof needs to also handle Matlab we will probably want to use the same HKLM hive to determine which Matlab to use and will want to provide a mechanism for the end user to select Matlab. Perhaps just use the directive for this, as in

        #! Matlab - for the default internal Matlab
        #! <path to Matlab install> for installed Matlab - equivalent to the MATLABROOT key. typical example #!C:\apps\Matlab\R2015b

With this approach the handler does not need to look at internal VEE data structures since the Matlab Object will format them to be Matlab compatible. The handler needs to understand Matlab data structures but fortunately those are well documented by Mathworks.

I would expect that the handler would be written in Python with perhaps a C wrapper for the interface to VEE. After dealing with Python scripting the first priority would be Matlab. But unlike the native Matlab Object, any version of Matlab could be linked as selected by the shebang. If architected well, hooks should be left so that virtually any script-driven tool could be incorporated. Python interfaces to many tools already exist so such extensions would be relatively easy to add.

        #! Matlab 7.13
        #! Octave
        #! Python 3.9
        #! SciPy
        #! SciLab
        #! Spice 3F
        etc.

This is the more difficult approach but by far the cleanest and most elegant since the user only sees native VEE.

A perusal of Mathworks documentation shows that as long as Agilent stayed with a standard Matlab interface (likely since VEE will link with COTS Matlab) then the reverse engineering may be largely documented. In particular, extracting the relevant section of Mathworks' documentation shows that the interface could be as simple as a DLL with a handful of functions:


C and Fortran Engine Library Functions - 2015b
engOpen (C and Fortran) 	Start MATLAB engine session
engOpenSingleUse (C) 		Start MATLAB engine session for single, nonshared use
engClose (C and Fortran) 	Quit MATLAB engine session
Engine (C) 			Type for MATLAB engine
engEvalString (C and Fortran) 	Evaluate expression in string
engGetVariable (C and Fortran) 	Copy variable from MATLAB engine workspace
engPutVariable (C and Fortran) 	Put variable into MATLAB engine workspace
engGetVisible (C) 		Determine visibility of MATLAB engine session
engSetVisible (C) 		Show or hide MATLAB engine session
engOutputBuffer (C and Fortran) Specify buffer for MATLAB output

Fortunately Mathworks provides some nice examples to go along with this documentation. These examples are from R2011a which is the same vintage as VEE 9.3's OEM Matlab. If they match the 2015b documentation then that documentation should be valid for our work. An extract at one of the examples (endemo.c) indeed shows exactly the above functions.


if (!(ep = engOpen("\0"))) {
		fprintf(stderr, "\nCan't start MATLAB engine\n");
		return EXIT_FAILURE;
	}

	/*
	 * PART I
	 *
	 * For the first half of this demonstration, we will send data
	 * to MATLAB, analyze the data, and plot the result.
	 */

	/* 
	 * Create a variable for our data
	 */
	T = mxCreateDoubleMatrix(1, 10, mxREAL);
	memcpy((void *)mxGetPr(T), (void *)time, sizeof(time));
	/*
	 * Place the variable T into the MATLAB workspace
	 */
	engPutVariable(ep, "T", T);

	/*
	 * Evaluate a function of time, distance = (1/2)g.*t.^2
	 * (g is the acceleration due to gravity)
	 */
	engEvalString(ep, "D = .5.*(-9.8).*T.^2;");

	/*
	 * Plot the result
	 */
	engEvalString(ep, "plot(T,D);");
	engEvalString(ep, "title('Position vs. Time for a falling object');");
	engEvalString(ep, "xlabel('Time (seconds)');");
	engEvalString(ep, "ylabel('Position (meters)');");

If this is indeed the case then the interface is very similar to what was used back in VEE 7, albeit with different functions. The Mathworks documentation gives a decent description of these functions so implementing a Python emulation may not be all that tough. It would of course be great to examine VEE source, but that is probably not possible, so experimentation will be needed, but at least the docs should greatly limit that exercise. The apparent process is exactly as with Matlab 7:

start the Matlab engine
create variables as needed - this may not be that simple since the Matlab interface maps data to Matlab data types and we want Python data
execute the script line by line
get variables as needed
quit

Going the other way- controlling Matlab from Python- appears fully documented so should be much easier.

ADVANTAGES of this method include

The existing Matlab interface will become outdated at some future release of Matlab. With a spoof we could link to any future API
aside from the directive line this should be transparent to the end user

DISADVANTAGES other than the difficulty

VEE data inputs are limited to whatever the existing interface supports. VEE Data types are converted to Matlab equivalents which may be an issue for Python or other languages.
The Matlab Script object sends one line of code at a time to Matlab. This makes it likely that sending the script every time is the biggest time sink. If we do our own handler we can send the script once and execute it multiple times, perhaps sending new data in between runs.
32-bit only - this looks to be a significant issue already and will only get worse.

Suggested Architecture

As with any project like this it is important to start with a design rather than starting with the details. In this case after perusing the various needs we will want to settle on an architecture before digging in very far. From a very high level this is simple enough

Get data and script from VEE
Execute the script
Return data to VEE

But looking even slightly deeper shows needed architectural details such as:

The "proof of concept" code creates a final script that includes the original script, lines to create the needed data, and more lines to create the needed outputs. Then the full script is passed to Python and run as a batch job.

Conversely the Matlab engine method is much more serial in nature. The Matlab engine is started then one line at a time is passed to it, creating data, running the desired script, and then bringing data back. But everything is a single line of code at a time (engEvalString)

Our code will want to be able to accommodate either batch or serial mode on the backend (running external engines) and we will want to decide on a transfer mode when we are talking to VEE. In one scenario we would send all the data and script as a VEE container and the Python handler would go from there:

Start the handler if it is not already running
VEE Container sent with everything for Python to process
1. Read the rec.script[0] line to determine which engine to use
2. If Python
  1. create the needed data from the record fields
  2. run the script
  3. write the output record
3. If Not python process the data and script as needed for whatever execution engine is desired
  1. send data to engine
  2. execute the script in serial or batch mode as appropriate
  3. bring data back
Send container back to VEE
Quit the handler or leave it running depending upon how we choose to make this work. Leave it running for performance reasons seems likely.

Given a structure something like this, some fairly obvious modules come to mind:

readVEEContainer - parses a VEE container and creates variables in Python corresponding to the filed names and values - medium effort, existing parser should be fairly close
writeVEEContainer - creates a VEE container given an arbitrary set of named values - medium effort, may need to write from scratch
determineEngine - parse the directive line from the script and determine which engine to use - should be very simple
runPythonScript - executes an arbitrary python script - should be simple since we are already inside python
plugin modules for each supported language - existing integrations probably already exist and are useful with minimal mods
- runMatlabScript
- runOctaveScript
- runSpicedeck
- leave a hook for arbitrary new languages
readFromVEE - highly dependent upon how we choose to proceed, but this brings script and variables from VEE. Communications method needs to be selected. High effort if Matlab spoofing chosen, simple if bring VEE container across
writeToVEE - highly dependent upon how we choose to proceed, but this sends variables back to VEE. Communications method needs to be selected. High effort if Matlab spoofing chosen, simple if bring VEE container across

Given a multi-person team it seems reasonable to choose and IDE, define schemas, write up some external docs and then assign the various pieces to team members.

QUESTION: once we have the emulated Matlab can we call that directly as opposed to via the Matlab script object? if so then maybe can build UF which behaves like the script object but is programmable. Of course if that's our goal then why do the spoofing?

Use minimal Python engine for our server and treat full Python as a client. (9MAY2021)

This may seem convoluted but at the same time it solves multiple issues and sets up a simple way to expand in the future and to support network computation such as the image processing that Richard would like to do.

This approach stems from initial investigation by Faisal. In looking at the socket server he notes that since we allow the end user to specify arbitrary scripts to execute, there's nothing stopping those scripts from loading libraries or doing most anything else to our python install. We can't simply lock things down with permissions since the end user may fundamentally need to load new libraries or make other changes. It would seem that the core problem here is that in this scenario Python is acting as both server AND client. The server needs to be stable and controlled but the client needs to be open- and we really can't have both.

My suggested solution is to have a small dedicated python install (as in directly from python.org, adding only necessary libraries and removing anything not needed) for the server that is called by VEE. This server talks to VEE via sockets. Any "client" including python is controlled by this small server.

VEE   <>  Python server <> python
                        <> octave
                        <> SPICE
                        <> any other client which we can add as a plugin to our server

By doing this we have complicated things somewhat but have effectively separated the SERVER and CLIENT rolls. As for implementation details I expect that a small piece of any network computing Python library would do the trick just fine, then treat localhost as a network client. A real plus to using the network computing library would be that it would be very simple to place the end python on a server somewhere instead of on the user's machine. A side benefit is that since our server uses a minimal python engine it should start very quickly.

One such network computing choice might be RAY. It is overkill by quite a bit BUT at the same time it would certainly set us up well to support servers such as with Richard's image processing work.

Disadvantages

Complexity - adds level of communications (python <> python)

Advantages

stability - server is self contained and does not execute anything so can't be corrupted by end user
Can settle on any single distribution for server
supportability - only need to support exactly one distribution
expandable - lays foundation for network computation, cross platform

Communicating Between VEE and Python

One fundamental piece of this project is a communications channel between VEE and Python. There are many options to be considered.

DLL interface - write a simple C DLL that can be imported and used - this would be universal
Disadvantages:
- our bugs become VEE bugs
- 32 bit only since VEE is 32 bit. This means one needs to use 32-bit python or deal with the complexities of bridging the 64 bit Windows with the 32-bit subsystem
- possible security issue since compiled code
Advantages
- simplicity, universally available to any version of VEE or for that matter most any program
EVO - create an EVO for the interface - recent VEE only and no function though could put the EVO inside a function
Disadvantages:
- only available for VEE 9.x, can be slow to start since dotNet
- 32-bit only
Advantages:
- supported and customizable, very clean to end user
Sockets - in this scenario, we create a program in Python or C or ?? that runs continuously. It has an open port and a simple command language (perhaps VCTP or something). On the VEE side we open a socket and send/receive data as needed. To illustrate the concept there is a very simple server and client written in VEE. NOTE that this example sends data and script separately so multiple runs of the given script using different data involves very little overhead. We will need to define a protocol which I will label VEE Script Interface Protocol (VSIP) for the moment.
Disadvantages:
- can be resource hog if not done carefully since it polls continuously
- can be fragile due to syntax issues
- opens door to DNS attacks
Advantages:
- networkable and cross-platform - hence can have server most anywhere
- universally available to any version of VEE or for that matter most any program
- AND while sockets are ancient, they are also how most modern communications on the internet are done anyhow (HTTP, SMTP, FTP, etc) so are very well supported and will be around for a very long time
- no compiled code so minimal security issues
- easy path to expanding into other directions than Python
- simple and flexible- very low level so can define more or less any protocol.
- Can have a single server being used by multiple clients
- if security measures are needed, sockets can be controlled by most any firewall
Matlab Spoof - uses existing interface defined by Mathworks. In this interface, Matlab (or our spoof) provides a set of functions that are exercised by VEE's Matlab Script object. These are presumably dotNet or maybe ActiveX, but include creation of variables and execution of scripts one-line-at-a time.
Disadvantages
- Requires reverse engineering
Advantages
- transparent to end user
DotNet interface - methods that can be imported and used written using dotnet code
- Disadvantages
  - Complexity
  - Windows only
  - Dependencies
- Advantages

What Version of Python Should We Use?

This is a relatively small issue but one which we should carefully consider from the end user's point of view. It seems reasonable to adopt a similar use model as is presented with the VEE Matlab Script object. In that case a core set of Matlab (basically an engine plus signal processing library) is shipped with VEE. This is the default Matlab that is exercised by the Matlab Script object. This is transparent to the end user aside from the time lag instantiating the Matlab engine. This does not provide everything that Matlab can do but is a good solid subset and it is incorporated into the Function browser. However the end user can also install a full "real" Matlab and instruct the Matlab Script engine to instead use this installed Matlab. This way the Matlab Script object can exercise the full power of Matlab.

With Python we could do something similar fairly easily. The basic idea of the shebang line is that the end user can specify a script engine, so one way we could proceed would be

#! Python - uses the default version of Python that we ship the script object with

#! PythonXXX - allows the end user to specify which Python- perhaps full path or whatever makes sense

HOWEVER using a socket server this gets slightly more involved as there are now 2 engines involved: 1 for our python server, and one for the actual execution engine. These could be the same python but may well not be. The engine is specified in the shebang line, but we also need to specify the server. By default it probably makes sense to have both be the same, but then allow the end user to easily (probably in an RC file) specify a different choice. Since VEEOS already deals with config via rc files, that seems a likely way to go- perhaps something like

gvar scriptServer Text VEEROOT/veeos/python/server.py              <<< probably want full executable path here
gvar pythonEnginePath Text VEEROOT/veeos/python/python.exe         <<< probably want full executable path here
gvar pythonEngineHost Text LOCALHOST
gvar pythonEnginePort Int32 5001

Either or both of these could be easily changed by the end user. NOTE that if we use "scriptServer" and specify the executable it would be easy to specify any server desired, such as a server written in Julia or some other language. (NOTE- we will want to document the server itself well!)

As for what we ship, something with minimal install hassles would seem the best. A portable version of python would seem an ideal candidate since we could install it along with the rest of VEEOS by just copying it into place (presumably in VEEROOT/veeos/python if the version is small enough and there are no issues with spaces in the path or c:/apps/python or similar otherwise). While the original "Portable Python" is small and nicely functional, it is no longer maintained. However other portable versions exist such as WinPython. If we elect to go with a non-portable version then we will need to include a default and a simple method for the end user to specify python's location. It also may make perfect sense for our default install to have bare Python (python.org) with selected libraries but no IDE. This since many/most users will not be doing any actual Python programming but will rather be copying scripts into place. And without the IDE python is much smaller.

NOTE- we might want to NOT install python into VEE's install since it seems quite possible that scripts will have a problem with the "Program File (x86)" path where VEE is installed. A suggested alternative is to use C:\apps\python\veeosPy or similar.

One could argue that we could have user install Python and "register" it with Windows so that all we need to do is to provide the server script and execute it, letting Windows choose which Python. However this would be problematic if user has multiple python's or if one wanted to do network computing. Hence we could consider this as a default but would still need to enable end user to specify things.

A significant advantage to a portable version that can't be overlooked is the portability. The bits could be moved anywhere by just a simple copy, precluding the need to install them on end user's machines.

Embeddable Python would seem a natural candidate also. This is VERY small and is direct from python.org so also very mainstream. Adding modules is not supported but appears simple enough. NOTE that one downside of this approach is that it doesn't support XP since the executables involved require WIN7 at least.

Packaging the Embeddable Server

22JUN2021 - it has tentatively been decided that the server will be run from a minimal python install packaged as part of the VEEOS library. Now let's look at the install needed for this, with a goal of making it as small as practical.

No development environment is needed- this should be considered as a "run-only" python. As such it makes sense to download directly from python.org. And since we are are by necessity running the server on Windows, the embeddable code also makes sense. NOTE that for remote operation on a Windows server we could use the same package but on a linux box we would need something different.
1. Core Python - go to python.org and download the Windows embeddable 64-bit package - as of this writing the current version is 3.9.5
2. unzip this package and have fully portable python
Now grab the server code and the parser - server.py and VEEparser.py from Github . Place these in the python install
Next we need to install numpy since that is used by the parser
1. Install PIP to manage installs:
  1. From https://bootstrap.pypa.io/get-pip.py grab get-pip.py and place in the python install
  2. run this script
2. configure the path
  1. uncomment the import site line in python39._pth
  2. create sitecustomize.py with two lines:
    import sys
    sys.path.insert(0, '')
3. now use pip to install numpy
  cd Scripts
  pip install numpy

When all is said and done, the entire install is about 90MB. Roughly 15MB of this is python itself with the rest being numpy. Experimentation shows that perhaps 50MB of this can be stripped since it is not needed but it may be more supportable to just leave it intact.

05JUL2021 - this looks pretty good but decide to add a basic engine install along with the server install. By uninstalling PIP after putting numpy in place one can keep the overall size down to about the same as without the extra install.
NOTE that it would be nice to support Windows 7 as well as Windows 10- and that looks very feasible by using Python 3.8 instead of 3.9. The latest 3.8 code that has embeddable package is 3.8.10 so that's a good target.
15AUG2021 - for the octave interface oct2py is needed and it in turn has lots of dependencies so will just need to accept the size.

Embeddable python can be used for both the server and engine
1. Download python from python.org - grab 3.8.10 embeddable 64-bit (local copy)
2. unpack to python/server and to python/engine (each ~15MB) - use same bits for both (unpacked ~18MB)
Now upgrade the server install by adding PIP then numpy
1. From https://bootstrap.pypa.io/get-pip.py grab get-pip.py and place in the python install
2. run this script (now ~31MB)
3. uncomment the import site line in python38._pth
4. create sitecustomize.py with two lines:
  import sys sys.path.insert(0, '')
5. now use pip to install numpy (now ~90MB)
  cd Scripts pip install numpy
  This installs numpy-1.21.1
6. and now oct2py since needed for octave interface (now ~285MB)
  cd Scripts pip install oct2py
  Note that this installs lots of dependencies: backcall-0.2.0 colorama-0.4.4 debugpy-1.4.1 decorator-5.0.9 ipykernel-6.1.0 ipython-7.26.0 ipython-genutils-0.2.0 jedi-0.18.0 jupyter-client-6.1.12 jupyter-core-4.7.1 matplotlib-inline-0.1.2 metakernel-0.27.5 oct2py-5.2.0 octave-kernel-0.32.0 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.19 ptyprocess-0.7.0 pygments-2.10.0 python-dateutil-2.8.2 pywin32-301 pyzmq-22.2.1 scipy-1.7.1 six-1.16.0 tornado-6.1 traitlets-5.0.5 wcwidth-0.2.5
And now strip PIP since it is not needed. (now ~271MB)
1. python -m pip uninstall pip setuptools wheel

At this point the entire python install (server + engine) runs just about 90MB so is very manageable. Next step is to add the server's python code (server.py, execute.py, VEEParser.py and bridgeScripts.py)

HOWEVER - it is hard to come up with a good reason not to use the "server" install for the default engine also so maybe no need for second python??

06JUL2021 - for the moment let's go with a single python install, but with PIP stripped to minimize the size. Since this will be used for the engine also we can leave all the standard install intact. That gives the engine full standard libraries plus all of numpy.

Choose a Protocol

20210701: have proposed server solution on VEE side and on Python side. All the functionality appears to work so now time to firm up the communications protocol. ASSUME that the protocol commands themselves are plain text and that all other messages are in VEE container format. For status and handshakes one standard is the aviation procedure words

VEE proposed	Functionality
STATUS	return server status
DATA	read data handshake ("ROGER" or equiv)
SCRIPT	read script handshake ("ROGER" or equiv)
EXECUTE	handshake ("WILCO" or equiv) run the script
getDATA	return data
QUIT	handshake ("OUT" or equiv) shut down the server

PYTHON proposed	Functionality
INPUT
OUTPUT
SCRIPT
GET
HALT	close server connection, leave server running
EXECUTE
STATUS	return server status
QUIT	shut down the server
returnToVEE

Chosen Protocol	Functionality
STATUS	return server status (VEE container)
DATA	read data (VEE container) handshake ("OK")
SCRIPT	read script (VEE container) handshake ("OK")
EXECUTE	handshake ("OK") run the script
GETRESULT	return data (VEE container)
QUIT	handshake ("OK") shut down the server
HALT	handshake ("OK") close server connection