Interplatform (interprocess, interlanguage) communication

»X« below is another language than Java, for example,
VBA, C#, or C.

When an X process and a Java process have to exchange
information on the same computer, what possibilites are
there? The Java process should act as a client, sending
commands to the X process and also wants to read answers
from the X process. So, the X process is a kind of server.

My criteria are: reliability and it should not be extremely
slow (say exchanging a string should not take more than
about 10 ms). The main criterion is reliability.

»Reliability« means little risk of creating problems, little
risk of failure at run-time. (It might help when the client
[=Java process] can reset the communication to a known and
sane start state in case of problems detected at run-time.)

The host OS is Windows, but a portable solution won't hurt.

A list of possibilities I am aware of now:

Pipes

I have no experience with this. I heard one can establish
a new process »proc« with »exec« and then use

BufferedWriter out = new BufferedWriter(new
OutputStreamWriter(proc.getOutputStream()));
BufferedReader in = new BufferedReader(new
InputStreamReader(proc.getInputStream()));

Files

One process writes to the end of a file, the other reads
from the end of the file? - I never tried this, don't know
if it is guaranteed to work that one process can detect and
read, whether the other has just appended something to a file.

What if the processes run very long and the files get too
large? But OTOH this is very transparent, which makes it easy
to debug, since one can open the files and directly inspect
them, or even append commands manually with »copy con file«.

Sockets

This is slightly less transparent than files, but has the
advantage that it becomes very easy to have the two
processes running on different computers later, if this
should ever be required. Debugging should be possible
by a man-in-the-middle proxy that prints all information
it sees or by connecting to the server with a terminal.

JNI

JNI might be used to access code written in C or
ABI-compatible languages. This should be fast, but I heard
that it is error prone to write JNI code and needs some
learning (code less maintainable)?

On 02/03/2012 08:52 PM, Stefan Ram wrote:
> »X« below is another language than Java, for example,
> VBA, C#, or C.
>
> When an X process and a Java process have to exchange
> information on the same computer, what possibilites are
> there? The Java process should act as a client, sending
> commands to the X process and also wants to read answers
> from the X process. So, the X process is a kind of server.
>
> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
>
> »Reliability« means little risk of creating problems, little
> risk of failure at run-time. (It might help when the client
> [=Java process] can reset the communication to a known and
> sane start state in case of problems detected at run-time.)
>
> The host OS is Windows, but a portable solution won't hurt.
>
> A list of possibilities I am aware of now:
>
> Pipes
>
> I have no experience with this. I heard one can establish
> a new process »proc« with »exec« and then use
>
> BufferedWriter out = new BufferedWriter(new
> OutputStreamWriter(proc.getOutputStream()));
> BufferedReader in = new BufferedReader(new
> InputStreamReader(proc.getInputStream()));

A pipes is just 1:1 communication and only in 1 direction.

> Files
>
> One process writes to the end of a file, the other reads
> from the end of the file? - I never tried this, don't know
> if it is guaranteed to work that one process can detect and
> read, whether the other has just appended something to a file.

You can, but what do you do with the ever increasing file? This is not
reliable since the filesystem will fill up at some point.

> What if the processes run very long and the files get too
> large? But OTOH this is very transparent, which makes it easy
> to debug, since one can open the files and directly inspect
> them, or even append commands manually with »copy con file«.
>
> Sockets
>
> This is slightly less transparent than files, but has the
> advantage that it becomes very easy to have the two
> processes running on different computers later, if this
> should ever be required. Debugging should be possible
> by a man-in-the-middle proxy that prints all information
> it sees or by connecting to the server with a terminal.

You can as well use a packet sniffer (Wireshark for example). If you
use a standard protocol you'll typically have encoding functionality in
the tool.

> JNI
>
> JNI might be used to access code written in C or
> ABI-compatible languages. This should be fast, but I heard
> that it is error prone to write JNI code and needs some
> learning (code less maintainable)?

That would be a clumsy approach IMHO.

I'd pick a higher level protocol such as

- SOAP (XML based, ubiquitous)
- CORBA (a little out of fashion but quite efficient in terms of network
transport)

Advantage: you can focus on definition of the API and need not take care
of all the nifty details. Choice should also depend on the availability
for language X, of course.

Kind regards

robert

On 2/3/2012 2:52 PM, Stefan Ram wrote:
> »X« below is another language than Java, for example,
> VBA, C#, or C.
>
> When an X process and a Java process have to exchange
> information on the same computer, what possibilites are
> there? The Java process should act as a client, sending
> commands to the X process and also wants to read answers
> from the X process. So, the X process is a kind of server.
>
> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
>
> »Reliability« means little risk of creating problems, little
> risk of failure at run-time. (It might help when the client
> [=Java process] can reset the communication to a known and
> sane start state in case of problems detected at run-time.)
>
> The host OS is Windows, but a portable solution won't hurt.
>
> A list of possibilities I am aware of now:
>
> Pipes
>
> I have no experience with this. I heard one can establish
> a new process »proc« with »exec« and then use
>
> BufferedWriter out = new BufferedWriter(new
> OutputStreamWriter(proc.getOutputStream()));
> BufferedReader in = new BufferedReader(new
> InputStreamReader(proc.getInputStream()));

That would require the client to start the server.

Does not look as a good solution.

> Files
>
> One process writes to the end of a file, the other reads
> from the end of the file? - I never tried this, don't know
> if it is guaranteed to work that one process can detect and
> read, whether the other has just appended something to a file.
>
> What if the processes run very long and the files get too
> large? But OTOH this is very transparent, which makes it easy
> to debug, since one can open the files and directly inspect
> them, or even append commands manually with »copy con file«.

It should work, but it will be slow.

> Sockets
>
> This is slightly less transparent than files, but has the
> advantage that it becomes very easy to have the two
> processes running on different computers later, if this
> should ever be required. Debugging should be possible
> by a man-in-the-middle proxy that prints all information
> it sees or by connecting to the server with a terminal.

That would be my choice.


> JNI
>
> JNI might be used to access code written in C or
> ABI-compatible languages. This should be fast, but I heard
> that it is error prone to write JNI code and needs some
> learning (code less maintainable)?


JNI would mean single process.

It does fit with your problem description.

JNI is a bit tricky, but it is not more difficult than
many other things. But since Java programmers very rarely
use JNI, then most Java programmers never learn JNI properly
with the expected result. You could learn JNI if you need to.

Arne

On 2/3/2012 4:44 PM, Robert Klemme wrote:
> On 02/03/2012 08:52 PM, Stefan Ram wrote:
>> »X« below is another language than Java, for example,
>> VBA, C#, or C.
>>
>> When an X process and a Java process have to exchange
>> information on the same computer, what possibilites are
>> there? The Java process should act as a client, sending
>> commands to the X process and also wants to read answers
>> from the X process. So, the X process is a kind of server.
>>
>> My criteria are: reliability and it should not be extremely
>> slow (say exchanging a string should not take more than
>> about 10 ms). The main criterion is reliability.
>>
>> »Reliability« means little risk of creating problems, little
>> risk of failure at run-time. (It might help when the client
>> [=Java process] can reset the communication to a known and
>> sane start state in case of problems detected at run-time.)
>>
>> The host OS is Windows, but a portable solution won't hurt.
>>
>> A list of possibilities I am aware of now:
>>
>> Pipes
>>
>> I have no experience with this. I heard one can establish
>> a new process »proc« with »exec« and then use
>>
>> BufferedWriter out = new BufferedWriter(new
>> OutputStreamWriter(proc.getOutputStream()));
>> BufferedReader in = new BufferedReader(new
>> InputStreamReader(proc.getInputStream()));
>
> A pipes is just 1:1 communication and only in 1 direction.

That type of pipe is bidirectional.

And Windows named pipes are bidirectional as well.

>> Files
>>
>> One process writes to the end of a file, the other reads
>> from the end of the file? - I never tried this, don't know
>> if it is guaranteed to work that one process can detect and
>> read, whether the other has just appended something to a file.
>
> You can, but what do you do with the ever increasing file? This is not
> reliable since the filesystem will fill up at some point.

It would be possible to switchover to a new file and
delete the old file if he really wanted to go this route.

> I'd pick a higher level protocol such as
>
> - SOAP (XML based, ubiquitous)
> - CORBA (a little out of fashion but quite efficient in terms of network
> transport)
>
> Advantage: you can focus on definition of the API and need not take care
> of all the nifty details. Choice should also depend on the availability
> for language X, of course.

They will use socket as transport.

But if the X language has a good SOAP toolkit, then it would
certainly make things a lot easier.

Arne

On 12-02-03 03:52 PM, Stefan Ram wrote:
> »X« below is another language than Java, for example,
> VBA, C#, or C.
>
> When an X process and a Java process have to exchange
> information on the same computer, what possibilites are
> there? The Java process should act as a client, sending
> commands to the X process and also wants to read answers
> from the X process. So, the X process is a kind of server.
>
> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
[ SNIP ]
>
> Files
>
> One process writes to the end of a file, the other reads
> from the end of the file? - I never tried this, don't know
> if it is guaranteed to work that one process can detect and
> read, whether the other has just appended something to a file.
>
> What if the processes run very long and the files get too
> large? But OTOH this is very transparent, which makes it easy
> to debug, since one can open the files and directly inspect
> them, or even append commands manually with »copy con file«.
[ SNIP ]

A logical subset of files for IPC is database tables.

AHS
--
....wherever the people are well informed they can be trusted with their
own government...
-- Thomas Jefferson, 1789

On 02/03/2012 02:52 PM, Stefan Ram wrote:
> »X« below is another language than Java, for example,
> VBA, C#, or C.
>
> When an X process and a Java process have to exchange
> information on the same computer, what possibilites are
> there? The Java process should act as a client, sending
> commands to the X process and also wants to read answers
> from the X process. So, the X process is a kind of server.
>
> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
>
> »Reliability« means little risk of creating problems, little
> risk of failure at run-time. (It might help when the client
> [=Java process] can reset the communication to a known and
> sane start state in case of problems detected at run-time.)
>
> The host OS is Windows, but a portable solution won't hurt.
>
For Windows platform:
<http://msdn.microsoft.com/en-us/library/windows/desktop/aa365574%28v=vs.85%29.aspx>
Prune for Java/X support, prune again for your choice of protocol.

snip

ram@zedat.fu-berlin.de (Stefan Ram) writes:
>Sockets

Thanks for the answers so far! So I might go for sockets,
because I like that fact that I do not need any additional
libraries under C (where one can use winsocks) and under
Java (where they are part of Java SE).

Stefan Ram <ram@zedat.fu-berlin.de> wrote:

> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
>
> »Reliability« means little risk of creating problems, little
> risk of failure at run-time. (It might help when the client
> [=Java process] can reset the communication to a known and
> sane start state in case of problems detected at run-time.)

Other options:

* Use a message broker such as CORBA or MQ. Perhaps the cleanest
solution code-wise, but requires more infrastructure and gives you a
more complicated installation.

* Communicate through a shared database. Fiddly, but can be convenient
if the programs share a database anyway.

* REST -- have an HTTP servlet running in one program and make RESTful
calls to it from the other. (Roughly the same as a SOAP approach,
but if the communication is mostly command and control messages, it
might be more convenient.)

* JNI wrapper around shared memory / memory mapped files. Very fiddly,
very system specific, but potentially very high performance.
(There's a discussion on using memory mapped files with java here:
http://tinyurl.com/6oa3wej )

* The presence of trigger / lock files in a directory. Quick and easy,
but limited.

* Use a shared web resource / whiteboard. Quick and convenient if your
programs do HTTP calls anyway, but requires the shared web resource
to be in place and there's performance and latency to take into
account.

--
Leif Roar Moldskred

I would add to the list:

Shared Memory

Stefan Ram schrieb:
> »X« below is another language than Java, for example,
> VBA, C#, or C.
>
> When an X process and a Java process have to exchange
> information on the same computer, what possibilites are
> there? The Java process should act as a client, sending
> commands to the X process and also wants to read answers
> from the X process. So, the X process is a kind of server.
>
> My criteria are: reliability and it should not be extremely
> slow (say exchanging a string should not take more than
> about 10 ms). The main criterion is reliability.
>
> »Reliability« means little risk of creating problems, little
> risk of failure at run-time. (It might help when the client
> [=Java process] can reset the communication to a known and
> sane start state in case of problems detected at run-time.)
>
> The host OS is Windows, but a portable solution won't hurt.
>
> A list of possibilities I am aware of now:
>
> Pipes
>
> I have no experience with this. I heard one can establish
> a new process »proc« with »exec« and then use
>
> BufferedWriter out = new BufferedWriter(new
> OutputStreamWriter(proc.getOutputStream()));
> BufferedReader in = new BufferedReader(new
> InputStreamReader(proc.getInputStream()));
>
> Files
>
> One process writes to the end of a file, the other reads
> from the end of the file? - I never tried this, don't know
> if it is guaranteed to work that one process can detect and
> read, whether the other has just appended something to a file.
>
> What if the processes run very long and the files get too
> large? But OTOH this is very transparent, which makes it easy
> to debug, since one can open the files and directly inspect
> them, or even append commands manually with »copy con file«.
>
> Sockets
>
> This is slightly less transparent than files, but has the
> advantage that it becomes very easy to have the two
> processes running on different computers later, if this
> should ever be required. Debugging should be possible
> by a man-in-the-middle proxy that prints all information
> it sees or by connecting to the server with a terminal.
>
> JNI
>
> JNI might be used to access code written in C or
> ABI-compatible languages. This should be fast, but I heard
> that it is error prone to write JNI code and needs some
> learning (code less maintainable)?
>

On 03.02.2012 23:56, Arne Vajhøj wrote:
> On 2/3/2012 4:44 PM, Robert Klemme wrote:
>> On 02/03/2012 08:52 PM, Stefan Ram wrote:
>>> »X« below is another language than Java, for example,
>>> VBA, C#, or C.
>>>
>>> When an X process and a Java process have to exchange
>>> information on the same computer, what possibilites are
>>> there? The Java process should act as a client, sending
>>> commands to the X process and also wants to read answers
>>> from the X process. So, the X process is a kind of server.
>>>
>>> My criteria are: reliability and it should not be extremely
>>> slow (say exchanging a string should not take more than
>>> about 10 ms). The main criterion is reliability.
>>>
>>> »Reliability« means little risk of creating problems, little
>>> risk of failure at run-time. (It might help when the client
>>> [=Java process] can reset the communication to a known and
>>> sane start state in case of problems detected at run-time.)
>>>
>>> The host OS is Windows, but a portable solution won't hurt.
>>>
>>> A list of possibilities I am aware of now:
>>>
>>> Pipes
>>>
>>> I have no experience with this. I heard one can establish
>>> a new process »proc« with »exec« and then use
>>>
>>> BufferedWriter out = new BufferedWriter(new
>>> OutputStreamWriter(proc.getOutputStream()));
>>> BufferedReader in = new BufferedReader(new
>>> InputStreamReader(proc.getInputStream()));
>>
>> A pipes is just 1:1 communication and only in 1 direction.
>
> That type of pipe is bidirectional.

Well, that are actually two pipes aren't they? Or it's a socketpair,
depending on platform. Also, this approach only works if the Java
process always starts the other process. Alternatively the other
process would start the Java process this way and we can read from
System.in and write to System.out.

> And Windows named pipes are bidirectional as well.

Oh, I didn't knew that. Learn something new every day. Thanks!

>>> Files
>>>
>>> One process writes to the end of a file, the other reads
>>> from the end of the file? - I never tried this, don't know
>>> if it is guaranteed to work that one process can detect and
>>> read, whether the other has just appended something to a file.
>>
>> You can, but what do you do with the ever increasing file? This is not
>> reliable since the filesystem will fill up at some point.
>
> It would be possible to switchover to a new file and
> delete the old file if he really wanted to go this route.

Well, yes, but that soon gets nasty because of file locking etc.

>> I'd pick a higher level protocol such as
>>
>> - SOAP (XML based, ubiquitous)
>> - CORBA (a little out of fashion but quite efficient in terms of network
>> transport)
>>
>> Advantage: you can focus on definition of the API and need not take care
>> of all the nifty details. Choice should also depend on the availability
>> for language X, of course.
>
> They will use socket as transport.
>
> But if the X language has a good SOAP toolkit, then it would
> certainly make things a lot easier.

Exactly.

Cheers

robert

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

On 2/4/2012 4:57 AM, Jan Burse wrote:
> I would add to the list:
>
> Shared Memory
>


What Java API do you use for that?

markspace schrieb:
> On 2/4/2012 4:57 AM, Jan Burse wrote:
>> I would add to the list:
>>
>> Shared Memory
>>
>
>
> What Java API do you use for that?
>


One solution would be to port MemoryFiles from
Android to Java SE. The API of MemoryFiles is
seen here:

http://developer.android.com/referen...emoryFile.html

You can also find the source code of the classes.
But suggesting the above has more to do with my
obsession for memory files (just joking).

But the following stack overflow entry lists 5 (five)
alternative ways do deal with shared memory in Java:

http://stackoverflow.com/questions/1...memory-in-java

Bye

Jan Burse schrieb:
> But the following stack overflow entry lists 5 (five)
> alternative ways do deal with shared memory in Java:
>
> http://stackoverflow.com/questions/1...memory-in-java
>

Oops, 4 solutions and 1 finger-wagging.

Roedy Green schrieb:
> Let's say you used a simple RandomAccessFile. How could you implement
> a busy lock field in the file to indicate the file was busy being
> updated? or busy being read? In RAM you have test and set locks to
> check a value and set the value in one atomic operation. How could
> you simulate that without test and set hardware on the SSD? You can't
> very well share a RAM lock between separate jobs.

What do you want, a write lock or a read lock?
Here is a write lock:

Obtain the lock:
raf = new RandomAccessFile(file, "rw");

fo = new FileOutputStream(raf.getFD());
fo.getChannel().lock(0, Long.MAX_VALUE, false);

Release the lock:
fo.close();

raf.close();

Maybe it can be done even simpler, but the above
works for me over process / jvm boundaries. Can
be also used to synchronize jvm with non-jvm code.

Similar code I use to obtain a read lock, via an
FileInputStream and the lock() methods third
argument =true. Currently seems also to work on
Android, but did not yet thoroughly test...

Bye

(*)
http://docs.oracle.com/javase/1.4.2/...,%20boolean%29

On 2/4/2012 7:59 AM, Robert Klemme wrote:
> On 03.02.2012 23:56, Arne Vajhøj wrote:
>> On 2/3/2012 4:44 PM, Robert Klemme wrote:
>>> On 02/03/2012 08:52 PM, Stefan Ram wrote:
>>>> »X« below is another language than Java, for example,
>>>> VBA, C#, or C.
>>>>
>>>> When an X process and a Java process have to exchange
>>>> information on the same computer, what possibilites are
>>>> there? The Java process should act as a client, sending
>>>> commands to the X process and also wants to read answers
>>>> from the X process. So, the X process is a kind of server.
>>>>
>>>> My criteria are: reliability and it should not be extremely
>>>> slow (say exchanging a string should not take more than
>>>> about 10 ms). The main criterion is reliability.
>>>>
>>>> »Reliability« means little risk of creating problems, little
>>>> risk of failure at run-time. (It might help when the client
>>>> [=Java process] can reset the communication to a known and
>>>> sane start state in case of problems detected at run-time.)
>>>>
>>>> The host OS is Windows, but a portable solution won't hurt.
>>>>
>>>> A list of possibilities I am aware of now:
>>>>
>>>> Pipes
>>>>
>>>> I have no experience with this. I heard one can establish
>>>> a new process »proc« with »exec« and then use
>>>>
>>>> BufferedWriter out = new BufferedWriter(new
>>>> OutputStreamWriter(proc.getOutputStream()));
>>>> BufferedReader in = new BufferedReader(new
>>>> InputStreamReader(proc.getInputStream()));
>>>
>>> A pipes is just 1:1 communication and only in 1 direction.
>>
>> That type of pipe is bidirectional.
>
> Well, that are actually two pipes aren't they? Or it's a socketpair,
> depending on platform.

The Java Process supports in and out.

Whether the OS does it via single bidirectional or two unidirectional
does not change the Java code.

Thinking of it then two sounds more likely as Java also need to
separate err and out - that would be a lot easier with two.

> Also, this approach only works if the Java
> process always starts the other process.

Yep.

>>>> One process writes to the end of a file, the other reads
>>>> from the end of the file? - I never tried this, don't know
>>>> if it is guaranteed to work that one process can detect and
>>>> read, whether the other has just appended something to a file.
>>>
>>> You can, but what do you do with the ever increasing file? This is not
>>> reliable since the filesystem will fill up at some point.
>>
>> It would be possible to switchover to a new file and
>> delete the old file if he really wanted to go this route.
>
> Well, yes, but that soon gets nasty because of file locking etc.

Some coding required.

Arne