tcl file handles on Linux and Windows - advice requested

I am porting a Tcl/Tk/C application from Linux to Windows. It uses Tcl/
Tk to manage files (selection, open, etc.). In one use, a file is
opened by Tcl, and the file descriptor is set in a variable accessible
from C, and the C code can happily read()/write() to/from this file. I
use code like this to get the file descriptor on Linux:

set FileDesc [string trimleft $FileHandle file]

This results in FileDesc being the Linux file descriptor for the file
$FileHandle, which came from an earlier call to Tcl's open.

On Windows, the situation is different. The Tcl open call does not
return a file descriptor as part of the return value. I am guessing it
is, instead, a FILE *, as Windows does not support the C open() call.
I can still use the code above to get the FILE *, but the C code that
would use it can no longer use read/write.

My questions are:

- Is it the case that the Tcl open call on Windows does an fopen(),
whereas on Linux it does an open()? The Tcl open man page makes
reference to fopen() in such a way that one would think that is what
is used. However, the value returned by open is not from fdopen. Does
the Tcl open call use fileno() to get the file descriptor?

- If this is the case, C code that wants to access the open file must
use fread/fwrite on Windows. and read/write on Linux?

- Am I missing some fancy, perhaps even obvious, technique to get
around this? For example, if the Linux Tcl open is really using fopen
on all platforms, and using fileno() in the return value on non-
Windows platforms, can I get the FILE * that Tcl has?

I should point out that I am not mixing read/write between Tcl and C.
I am just letting Tcl open and configure the file.

One interesting thing is that I compile my C library for windows with
MinGW. So I do have a read/write interface. But as the Windows version
of Tcl/tk was not compiled the same way, it does not have open/read/
write/close. Does that sound reasonable? If I compiled Tcl/Tk with
MinGW, would the open/read/write/close interface be used in Tcl? This
is a hypothetical question, as I do not want to do that. I am just
curious.

On Jul 8, 10:10*am, Roger O <roger.oberholt...@gmail.com> wrote:
> I am porting a Tcl/Tk/C application from Linux to Windows. It uses Tcl/
> Tk to manage files (selection, open, etc.). In one use, a file is
> opened by Tcl, and the file descriptor is set in a variable accessible
> from C, and the C code can happily read()/write() to/from this file.

... snipped ...

> On Windows, the situation is different. The Tcl open call does not
> return a file descriptor as part of the return value. I am guessing it
> is, instead, a FILE *, as Windows does not support the C open() call.
> I can still use the code above to get the FILE *, but the C code that
> would use it can no longer use read/write.
>

You might want to try using the Tcl_GetChannelHandle function to
get the OS-specific file handle from the channel name, rather than
trying to infer it yourself from the channel name.

I'm not a Windows programmer, so I'm not too sure how you turn a
Windows file handle into a C standard library style file descriptor
(especially with MinGW thrown in)!

But I believe there is an _open_osfhandle in the MS C run time
library, so perhaps you need something like this: (untested!)

int writeToChannel (Tcl_Object *channelNameObj) {

char *channelName;
Tcl_Channel channel;
intptr_t os_file_handle;
int fd;

channelName = Tcl_GetStringFromObj(channelNameObj, NULL);
channel = Tcl_GetChannel(interp, channelName, NULL);
if (channel == NULL) {
Tcl_AppendResult(interp,
"could not find channel ",
channelName, (char *) NULL);
return TCL_ERROR;
}

if (Tcl_GetChannelHandle(channel, TCL_WRITABLE,
(ClientData) &os_file_handle) == TCL_ERROR) {
Tcl_AppendResult(interp,
"could not get writable OS device handle for channel ",
channelName, (char *) NULL);
return TCL_ERROR;
}

char *data = "hello\n";
size_t nbytes = strlen(data);

#ifdef WINDOWS
fd = _open_osfhandle (os_file_handle, 0);
#else
fd = (int) os_file_handle;
#endif

/* NB: I've seen some Windows code use _write()
* instead of write() for some reason? */
ssize_t bytes_written = write(fd, data, nbytes);

if (bytes_written != nbytes) {
Tcl_AppendResult(interp,
"failed to write to channel ",
channelName, (char *) NULL);
return TCL_ERROR;
}

return TCL_OK;
}

At Wed, 8 Jul 2009 02:10:37 -0700 (PDT) Roger O <roger.oberholtzer@gmail.com> wrote:

>
> I am porting a Tcl/Tk/C application from Linux to Windows. It uses Tcl/
> Tk to manage files (selection, open, etc.). In one use, a file is
> opened by Tcl, and the file descriptor is set in a variable accessible
> from C, and the C code can happily read()/write() to/from this file. I
> use code like this to get the file descriptor on Linux:
>
> set FileDesc [string trimleft $FileHandle file]
>
> This results in FileDesc being the Linux file descriptor for the file
> $FileHandle, which came from an earlier call to Tcl's open.
>
> On Windows, the situation is different. The Tcl open call does not
> return a file descriptor as part of the return value. I am guessing it
> is, instead, a FILE *, as Windows does not support the C open() call.
> I can still use the code above to get the FILE *, but the C code that
> would use it can no longer use read/write.
>
> My questions are:
>
> - Is it the case that the Tcl open call on Windows does an fopen(),
> whereas on Linux it does an open()? The Tcl open man page makes
> reference to fopen() in such a way that one would think that is what
> is used. However, the value returned by open is not from fdopen. Does
> the Tcl open call use fileno() to get the file descriptor?
>
> - If this is the case, C code that wants to access the open file must
> use fread/fwrite on Windows. and read/write on Linux?
>
> - Am I missing some fancy, perhaps even obvious, technique to get
> around this? For example, if the Linux Tcl open is really using fopen
> on all platforms, and using fileno() in the return value on non-
> Windows platforms, can I get the FILE * that Tcl has?
>
> I should point out that I am not mixing read/write between Tcl and C.
> I am just letting Tcl open and configure the file.
>
> One interesting thing is that I compile my C library for windows with
> MinGW. So I do have a read/write interface. But as the Windows version
> of Tcl/tk was not compiled the same way, it does not have open/read/
> write/close. Does that sound reasonable? If I compiled Tcl/Tk with
> MinGW, would the open/read/write/close interface be used in Tcl? This
> is a hypothetical question, as I do not want to do that. I am just
> curious.


You really should NOT be mixing C I/O library calls and Tcl I/O calls,
even if the Tcl code is only doing the open. If Tcl opens file file,
the C code should be using the Tcl I/O functions (Tcl_Read, Tcl_Write,
etc.). Otherwise the Tcl code should just pass the [file nativename ...]
string to the C code and let the C code use its own I/O functions.
Mixing things and trying to get the channel number from the file handle
string is not a good idea.

>
>
>
>

--
Robert Heller -- 978-544-6933
Deepwoods Software -- Download the Model Railroad System
http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows
heller@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/

On Jul 8, 1:46*pm, Robert Heller <hel...@deepsoft.com> wrote:

> You really should NOT be mixing C I/O library calls and Tcl I/O calls,
> even if the Tcl code is only doing the open. *If Tcl opens file file,
> the C code should be using the Tcl I/O functions (Tcl_Read, Tcl_Write,
> etc.). Otherwise the Tcl code should just pass the [file nativename ...]
> string to the C code and let the C code use its own I/O functions.
> Mixing things and trying to get the channel number from the file handle
> string is not a good idea. *

I know all the arguments. But it really should not be a problem if I
am not mixing the calls in a controlled fashion. As I wrote, I never
mix reading between Tcl and C. That would surely fail. Let's all agree
that I know the possible consequences of my behavior. I just want to
know how to get the OS handle, knowing that it is bastard programming
for which I take full responsibility.

Roger O wrote:
...
> Let's all agree
> that I know the possible consequences of my behavior. I just want to
> know how to get the OS handle, knowing that it is bastard programming
> for which I take full responsibility.

Ok, write a Tcl extension that contains Tcl_GetChannelHandle and
executes your C code (which I'll assume is an executable).

On Windows, you'll get a HANDLE rather than a FILE*, though, so use of
fgets()/fwrite() isn't valid for good xplat'ability. I do like Kieran's
advice for use of _open_osfhandle().

What would be excellent in your situation would be to re-use your C-code
in a different manner and create a Tcl extension with it. Change all
your fgets() to be Tcl_Gets, etc. Then you won't have to derive what
the channel represents, and you can operate on the channel directly
using the channel API.

That's the Tcl way of being X-plat.

--



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpUmcYACgkQlZadkQh/RmG01wCfQeh20rahGNTvGBMcAL2ezIR2
j3YAoKVTcGDwz8uLHgdRpHYLIvjnii4O
=bTBM
-----END PGP SIGNATURE-----

On Jul 8, 3:06*pm, David Gravereaux <davyg...@pobox.com> wrote:
> Roger O wrote:
>
> ...
>
> > Let's all agree
> > that I know the possible consequences of my behavior. I just want to
> > know how to get the OS handle, knowing that it is bastard programming
> > for which I take full responsibility.
>
> Ok, write a Tcl extension that contains Tcl_GetChannelHandle and
> executes your C code (which I'll assume is an executable).
>
> On Windows, you'll get a HANDLE rather than a FILE*, though, so use of
> fgets()/fwrite() isn't valid for good xplat'ability. *I do like Kieran's
> advice for use of _open_osfhandle().
>
> What would be excellent in your situation would be to re-use your C-code
> in a different manner and create a Tcl extension with it. *Change all
> your fgets() to be Tcl_Gets, etc. *Then you won't have to derive what
> the channel represents, and you can operate on the channel directly
> using the channel API.
>
> That's the Tcl way of being X-plat.
>
> --
>
> *signature.asc
> < 1KViewDownload

As an added benefit of changing direct C I/O to the Tcl I/O calls is
that you'll get support for VFS for free. This allows your code to
read inside Tclkits for example. Transforming the C code in this way
is fairly straightforward and definitely worth the little effort.

Mark

On Jul 8, 3:06*pm, David Gravereaux <davyg...@pobox.com> wrote:

> Ok, write a Tcl extension that contains Tcl_GetChannelHandle and
> executes your C code (which I'll assume is an executable).

The code is part of an image processing library that runs in numerous
places. This is only one. Sometimes the code runs on numerous
computers in a cluster, and these file descriptors refer to sockets.
Sometimes the code runs as part of a Tcl/Tk GUI all on a local
machine, and these very same file descriptors refer to local files. In
both cases, the C code is dynamically loaded. There is Tcl/Tk GUI in
both uses. This works because in the Unix/Linux Tcl, both the open and
the socket command return the file descriptor as part of the return
value. As long as the GUI always ran on Linux, I was a happy camper.

On Windows, I do not care if it is a HANDLE reference that is
returned. I can access that as well.

> On Windows, you'll get a HANDLE rather than a FILE*, though, so use of
> fgets()/fwrite() isn't valid for good xplat'ability. *I do like Kieran's
> advice for use of _open_osfhandle().

If Windows returns a HANDLE, and Unix/Linux an integer file descriptor
(which it does and which works with read/write), than I would imagine
on no platforms does Tcl return a FILE *. I don't mind which is
returned. As long as I know what it is. I can handle the x-platform in
my image processing library. Even if Tcl offers an alternative. The
problem is that the alternative may be too obtrusive. So, I am
checking my options.

At Wed, 8 Jul 2009 07:34:47 -0700 (PDT) Roger O <roger.oberholtzer@gmail.com> wrote:

>
> On Jul 8, 3:06=A0pm, David Gravereaux <davyg...@pobox.com> wrote:
>
> > Ok, write a Tcl extension that contains Tcl_GetChannelHandle and
> > executes your C code (which I'll assume is an executable).
>
> The code is part of an image processing library that runs in numerous
> places. This is only one. Sometimes the code runs on numerous
> computers in a cluster, and these file descriptors refer to sockets.

To what extent is the *I/O* interspersed with the *image processing*?
From my own experience with image processing (I worked at the UMass
Image Understanding Laboratory for over 20 years), it was often the case
that an image processing program would look like this psuedo code:

get parameters;
read images from disk in to memory;
process images loaded in memory;
write output images (or other data) out to disk;
display statisics or other information;

If your image processing library can be separated into two
sub-libraries, an image I/O library (reads and writes image data between
some external source and memory) and an image processing library (which
only processes in-memory images), then it makes sense to have two
versions of the image I/O library (possibly built from a single code
source using #ifdefs -- which you will need anyway), one version using
'native' I/O (eg open/read/write/close under UNIX and whatever
MS-Windows uses instead of open/read/write/close) and one version using
Tcl's I/O functions
(Tcl_OpenChannel/Tcl_Read/Tcl_Write/Tcl_CloseChannel). Both library act
on some commonly defined in-memory image structure object which gets
passed to/from the image processing library.

> Sometimes the code runs as part of a Tcl/Tk GUI all on a local
> machine, and these very same file descriptors refer to local files. In
> both cases, the C code is dynamically loaded. There is Tcl/Tk GUI in
> both uses. This works because in the Unix/Linux Tcl, both the open and
> the socket command return the file descriptor as part of the return
> value. As long as the GUI always ran on Linux, I was a happy camper.
>
> On Windows, I do not care if it is a HANDLE reference that is
> returned. I can access that as well.
>
> > On Windows, you'll get a HANDLE rather than a FILE*, though, so use of
> > fgets()/fwrite() isn't valid for good xplat'ability. =A0I do like Kieran'=
> s
> > advice for use of _open_osfhandle().
>
> If Windows returns a HANDLE, and Unix/Linux an integer file descriptor
> (which it does and which works with read/write), than I would imagine
> on no platforms does Tcl return a FILE *. I don't mind which is
> returned. As long as I know what it is. I can handle the x-platform in
> my image processing library. Even if Tcl offers an alternative. The
> problem is that the alternative may be too obtrusive. So, I am
> checking my options.
>

--
Robert Heller -- 978-544-6933
Deepwoods Software -- Download the Model Railroad System
http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows
heller@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/

Roger O wrote:
> I am porting a Tcl/Tk/C application from Linux to Windows. It uses Tcl/
> Tk to manage files (selection, open, etc.). In one use, a file is
> opened by Tcl, and the file descriptor is set in a variable accessible
> from C, and the C code can happily read()/write() to/from this file. I
> use code like this to get the file descriptor on Linux:
>
> set FileDesc [string trimleft $FileHandle file]
>
> This results in FileDesc being the Linux file descriptor for the file
> $FileHandle, which came from an earlier call to Tcl's open.

That is not supported and is not portable. Please use the Tcl C API calls
to translate from a Tcl file handle to a C file descriptor.


--
+------------------------------------------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+

Hi Roger,
to me the result of [open] (e.g. 'file1202ed0') looks a lot like it
contains a hexadecimal address.
If your lucky this is the address of the corresponding Tcl_Channel
struct, which surely contains something useful to you.

Of course I would never publicly recommend such an approach.
Good luck
Helmut Giese

On Jul 8, 7:00*pm, "Gerald W. Lester" <Gerald.Les...@cox.net> wrote:

> That is not supported and is not portable. *Please use the Tcl C API calls
> to translate from a Tcl file handle to a C file descriptor.

I am fully aware that this is not supported (despite the method on
Linux being described in the first Tcl book by Tcl's inventor - I did
not originate it). Sometimes one must hack. I strive not to do so. But
this is a case where I think I will continue trying to sort this out.

I am surprised that the Linux and Windows versions of Tcl do not both
return a channel in the open call. I would be more inclined to sort
out an official Tcl approach if Tcl itself was consistent in what the
open call returned. I suspect that the Linux/Unix open call value was
not changed when channels were added because there is lots of code
doing what I am doing. Even if it is not official. And I am not sure
the Windows version is returning a Channel. I am guessing it is either
a Handle or a FILE *. I will have to check the source. I say this
because I think Tcl's open was implemented on Windows before channels.

Roger O wrote:
> On Jul 8, 7:00 pm, "Gerald W. Lester" <Gerald.Les...@cox.net> wrote:
>
>> That is not supported and is not portable. Please use the Tcl C API calls
>> to translate from a Tcl file handle to a C file descriptor.
>
> I am fully aware that this is not supported (despite the method on
> Linux being described in the first Tcl book by Tcl's inventor - I did
> not originate it). Sometimes one must hack. I strive not to do so. But
> this is a case where I think I will continue trying to sort this out.
>
> I am surprised that the Linux and Windows versions of Tcl do not both
> return a channel in the open call. I would be more inclined to sort
> out an official Tcl approach if Tcl itself was consistent in what the
> open call returned. I suspect that the Linux/Unix open call value was
> not changed when channels were added because there is lots of code
> doing what I am doing. Even if it is not official. And I am not sure
> the Windows version is returning a Channel. I am guessing it is either
> a Handle or a FILE *. I will have to check the source. I say this
> because I think Tcl's open was implemented on Windows before channels.

Actually, channels came about before the windows port. The thing is,
Tcl is consistent.. channels all the way through. The difference you
see regards open/write/read. There are no such Win32 API calls. The
native ones on windows are OpenFile/WriteFile/ReadFile, IIRC. I think
open/write/read on windows is managed by the runtime.

--



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpVt30ACgkQlZadkQh/RmH1pACfSS5iG6cOC4rVVL/Ox/wuETSH
TLcAoKdetidi1MPDMje57kI27PjDjKep
=Ommy
-----END PGP SIGNATURE-----

Roger O wrote:
> On Jul 8, 7:00 pm, "Gerald W. Lester" <Gerald.Les...@cox.net> wrote:
>
>> That is not supported and is not portable. Please use the Tcl C API calls
>> to translate from a Tcl file handle to a C file descriptor.
>
> I am fully aware that this is not supported (despite the method on
> Linux being described in the first Tcl book by Tcl's inventor - I did
> not originate it). Sometimes one must hack. I strive not to do so. But
> this is a case where I think I will continue trying to sort this out.

Tcl is consistent -- the value is an opaque handle to be used by Tcl's
internals. You need to use the API call to convert the opaque handle to a C
file descriptor or handle.

Please note, that at *any* release of Tcl -- even a x.y.z bug fix release --
the appearance of that opaque handle may change and your hacks into it will
cease to work. You really need to use the C API.

Suggestion: create a very tiny extension that exposes the
Tcl_GetChannelHandle and Tcl_GetOpenFile in a meaningful way to you.

--
+------------------------------------------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+

Roger O wrote:

> On Jul 8, 6:08*pm, Robert Heller <hel...@deepsoft.com> wrote:
>...
> involve 50 kilometers of images (continuous, with 1.6mm per pixel
> resolution). ... individual JPEG2000 images ... are chunked into images of
a
> couple of meters in length, which are analyzed, ...

Wow!
-------------
While there is very good advice given in this thread, there is one more
thing you could look into:

Make sure your filehandle does not start with an "e":
Your
> set FileDesc [string trimleft $FileHandle file]
makes an filee12abcdef into 12abcdef
It is much better to use
set FileDesc [string range $FileHandle 4 end]

man n string
--
Mit freundlichen Grüßen (best regards)

Albrecht Mucha
Software Entwicklung
----------------------------------------------------------------------
MediTec GmbH
IT-Lösungen für das Gesundheitswesen
Griesbergstr. 1b-c eMail : [string map {X {}}
a.Xmucha@Xmeditec-gmbh.com]
31162 Bad Salzdetfurth Internet : www.meditec-gmbh.com

On Jul 9, 6:01*pm, am <a...@medi.xx> wrote:

> Make sure your filehandle does not start with an "e":
> Your> set FileDesc [string trimleft $FileHandle file]
>
> makes an filee12abcdef into 12abcdef
> It is much better to use
> set FileDesc [string range $FileHandle 4 end]

This is a good point. On Linux/Unix, the values following the 'file'
were always '0'-'9'. Windows changed that to be a HEX value. Thanks
for pointing that out.