unformatted I/O

Hi

consider the following example.

....
real(wp), dimension(10,10) :: nt

open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='write')
write(55) nt
close(55)

open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='read')
read(55) nt
close(55)

Question: if the write is done on a 32-bit Linux, then nt.unformatted is
copied to a 64-bit Linux and the read is done there, will nt contain the
same values? What if it is copied to 32-bit Windows and read there?

Thanks a lot!

Daniel

On 22 feb, 14:40, Daniel H <no.spam.addr...@gmx.de> wrote:
> Hi
>
> consider the following example.
>
> ...
> real(wp), dimension(10,10) :: nt
>
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='write')
> write(55) nt
> close(55)
>
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='read')
> read(55) nt
> close(55)
>
> Question: if the write is done on a 32-bit Linux, then nt.unformatted is
> copied to a 64-bit Linux and the read is done there, will nt contain the
> same values? What if it is copied to 32-bit Windows and read there?
>
> Thanks a lot!
>
> Daniel

One thing that may be a problem is that on 64-bits systems you or
at the very least other people may want the ability to read and write
very large arrays, that is arrays whose size exceeds 2 or 4 GB.
The most common structure for unformatted files on 32-bits OSes is
one where 4 bytes at the start and the end of each record indicate
how many bytes the record contains. So that limits the record size
to 2 or 4 GB (depending on whether signed or unsigned integers
are being used).

I know from past discussions that this has been an issue on 64-bits
platforms.

If you want to be sure the files are useable on any platform (for
small enough values of "any"), use "access='stream'" instead.
Then you do not have this hidden record structure to worry about.

Regards,

Arjen

On 2/22/2012 7:40 AM, Daniel H wrote:
....

> Question: if the write is done on a 32-bit Linux, then nt.unformatted is
> copied to a 64-bit Linux and the read is done there, will nt contain the
> same values? What if it is copied to 32-bit Windows and read there?
....

Specifically, it's compiler-dependent.

Odds are the 64-bit compiler will have a switch to control the size of
the record markers and you can manage it, but not guaranteed.

32-bit compilers tend to be consistent but again, the details of
implementation are left to the discretion of the compiler writer.

--

On 02/22/2012 02:40 PM, Daniel H wrote:
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='write')
> write(55) nt
> close(55)
>
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='read')
> read(55) nt
> close(55)
>
> Question: if the write is done on a 32-bit Linux, then nt.unformatted is
> copied to a 64-bit Linux and the read is done there, will nt contain the
> same values? What if it is copied to 32-bit Windows and read there?

I think it should work most of the time.

Possible issues:

a) Difference between little and big endian. That's architecture
dependent and not a 32 vs 64 bit issue, but 32 and 64 bit Linux is vague
enough that I thought I should mention it. Compilers have various
compile-time flags, environment variables, and vendor-specific OPEN
keywords which allow one to handle this.

b) Record format: In principle every compiler can write records in their
own way. In practice, most compiler have chosen the same method, thus it
works most of the time.

One special case, mentioned by Arjen, is how the compilers deal with
files exceeding 2 or 4 GB. Nowadays, the common solution is to use a
tricky format which uses the old 32bit record marker for smaller files
(remaining backwards compatible) and only for larger records to switch
to a 64bit marker. Thus, with very few exceptions (one is GCC's gfortran
but only the versions 4.0.x and 4.1.x), there should be no issue.

* * *

However, you can simply try and, to make sure that it works, you could
also always embed a dummy value as first entry and use it to check
whether the value is correct. That can be also used to detect endian
issues. For instance, the following program prints with gfortran
-fconvert=native
FEFF = 65279
and with -fconvert=swap
FFFE0000 = -131072

! Run once to write and once to read
! as -fconvert= applies to reading and writing
implicit none
integer :: i
open(99, form='unformatted')
!write(99) int(z'FEFF')
read(99) i
write(*,'(z8," = ",i8)') i, i
end

Tobias

On 2012-02-22, Tobias Burnus <burnus@net-b.de> wrote:
> One special case, mentioned by Arjen, is how the compilers deal with
> files exceeding 2 or 4 GB. Nowadays, the common solution is to use a
> tricky format which uses the old 32bit record marker for smaller files
> (remaining backwards compatible) and only for larger records to switch
> to a 64bit marker. Thus, with very few exceptions (one is GCC's gfortran
> but only the versions 4.0.x and 4.1.x), there should be no issue.

FWIW, the tricky format that GFortran (and at least the Intel compiler
as well, maybe others) uses does not switch to 64-bit markers, but
rather uses the sign of the 32-bit marker to designate a subrecord,
thus allowing arbitrary sized records to be split into a series of
subrecords < 2 GB in size.

IIRC a similar scheme is also used by the XDR data serialization
format.


--
JB

JB <foo@bar.invalid> wrote:

> On 2012-02-22, Tobias Burnus <burnus@net-b.de> wrote:
> > One special case, mentioned by Arjen, is how the compilers deal with
> > files exceeding 2 or 4 GB. Nowadays, the common solution is to use a
> > tricky format which uses the old 32bit record marker for smaller files
> > (remaining backwards compatible) and only for larger records to switch
> > to a 64bit marker. Thus, with very few exceptions (one is GCC's gfortran
> > but only the versions 4.0.x and 4.1.x), there should be no issue.
>
> FWIW, the tricky format that GFortran (and at least the Intel compiler
> as well, maybe others) uses does not switch to 64-bit markers, but
> rather uses the sign of the 32-bit marker to designate a subrecord,
> thus allowing arbitrary sized records to be split into a series of
> subrecords < 2 GB in size.
>
> IIRC a similar scheme is also used by the XDR data serialization
> format.

And I've seen simillar schemes for other systems at least as far back as
1984. I might have used some earlier than that, but I'm hazy on the
details of some earlier systems that I used. I distinctly recall that
the Elxsi we got in 1984 used a scheme much like that. Record headers
could be 8, 16, or 32 bits (I don't think I recall 64-bit ones). Any of
the record header sizes could support arbitrary-sized records.

--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain

On Feb 23, 12:40*am, Daniel H <no.spam.addr...@gmx.de> wrote:
> Hi
>
> consider the following example.
>
> ...
> real(wp), dimension(10,10) :: nt
>
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='write')
> write(55) nt
> close(55)
>
> open(55,file='nt.unformatted',form='unformatted',i ostat=err,action='read')
> read(55) nt
> close(55)
>
> Question: if the write is done on a 32-bit Linux, then nt.unformatted is
> copied to a 64-bit Linux and the read is done there, will nt contain the
> same values? What if it is copied to 32-bit Windows and read there?

That kind of I/O can be expected to work only on the same
operating system on the same or compatible hardware.

First, best to write out identical values on both systems
and compare the outputs.

baf wrote:
> On 02/23/2012 10:23 AM, Paul van Delst wrote:
>> In 12+ years I've never encountered problems with the portability of
>> unformatted, sequential access files. I guess I
>> have been extremely lucky.
>
> You have been lucky. I have not. gfortran and Intel Fortran produce
> incompatible files under windows.
>

Ah, windows. Well, in 25+ years I have been additionally lucky in that I have never needed to use a windows box for
anything other than using word or powerpoint.

Given that gfortran and Intel produce compatible files under just about every other combination of platform and compiler
version I've used, I know what I would change.... ) I've just never grokked windows as a work platform.

cheers,

paulv

baf <baf@nowhere.com> wrote:

> On 02/23/2012 10:23 AM, Paul van Delst wrote:

> > In 12+ years I've never encountered problems with the portability of
> > unformatted, sequential access files. I guess I have been extremely
> > lucky.
>
> You have been lucky. I have not. gfortran and Intel Fortran produce
> incompatible files under windows.

And 12 years isn't all that long. :-) In my 43 years of doing Fortran,
I've seen lots of unformatted sequential access files that weren't even
close to compatible. Sometimes, when getting a new file was going to be
impractical for various reasons, it was quite a lot of work to fix,
particularly with tapes, where it could sometimes be hard to convince
the destination system to read the darned tape at all.

Today's systems do tend to be a lot better about compatibility for
unformatted files. Not perfect, as baf notes, but sure a lot better than
in decades gone by. But I'd recommend taking the lessons of the past
instead of ignoring them as old stuff that no longer matters. New ways
for things to be incompatible can always come up again.

And the various compatibility switches only work for simple cases, I
might add. Sometimes there's no way for the compiler to know how to
handle byte swapping if, for example, you have something comparable to a
tagged variant record, where you don't know the size or alignment of
some of the data until after you have looked at earlier data. True, most
cases aren't like that. If you never have to deal with such things, then
I'd agree you are lucky.

I'd second Arjen's suggestion of using stream. It can of course, also
have incompatibilities, but it makes your odds a lot better in that you
no longer have to worry about things like record headers at all.

--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain

On 2/23/2012 2:33 PM, baf wrote:
> You have been lucky. I have not. gfortran and Intel Fortran produce
> incompatible files under windows.

I would find that, um, astonishing, as long as you were using a
reasonably current version of gfortran and not any options to either
compiler that change the unformatted file structure. To the best of my
knowledge, those two compilers share the same default on-disk structure
for unformatted sequential files of any record length.

--
Steve Lionel
Developer Products Division
Intel Corporation
Merrimack, NH

For email address, replace "invalid" with "com"

User communities for Intel Software Development Products
http://software.intel.com/en-us/forums/
Intel Software Development Products Support
http://software.intel.com/sites/support/
My Fortran blog
http://www.intel.com/software/drfortran

Refer to http://software.intel.com/en-us/arti...ization-notice
for more information regarding performance and optimization choices in
Intel software products.

baf <baf@nowhere.com> schrieb:

> You have been lucky. I have not. gfortran and Intel Fortran produce
> incompatible files under windows.

This would surprise me. I modified the gfortran code to be
compatible with Intel a few years ago (also using the nice Intel
scheme for records > 2GB).

What is the hexdump of the fort.10 file generated by the program

write (10) 1, 'a'
end

for both compilers on your system, and which version of gfortran
are you using?

baf <baf@nowhere.com> wrote:

> I have no idea what file format Intel uses when form="binary" is set,
> but it is nonstandard so I guess anything goes. I believe it was an
> option in the original Digital Visual Fortran, and it is listed in my
> 1999 version of the Compaq Fortran Language Reference Manual. For my
> purposes, I wish I could trick gfortran into producing that same output
> stream.

I believe that form='binary" to be a nonstandard spelling for what
amounts to the same thing as unformatted stream.

--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain

On 2/24/2012 1:33 AM, Richard Maine wrote:

> I believe that form='binary" to be a nonstandard spelling for what
> amounts to the same thing as unformatted stream.

Correct. FORM='UNFORMATTED', ACCESS='STREAM' to be specific. No record
structure at all, just a stream of bytes. Standard in F2003.

--
Steve Lionel
Developer Products Division
Intel Corporation
Merrimack, NH

For email address, replace "invalid" with "com"

User communities for Intel Software Development Products
http://software.intel.com/en-us/forums/
Intel Software Development Products Support
http://software.intel.com/sites/support/
My Fortran blog
http://www.intel.com/software/drfortran

Refer to http://software.intel.com/en-us/arti...ization-notice
for more information regarding performance and optimization choices in
Intel software products.

On 2/24/2012 8:37 AM, Steve Lionel wrote:
> On 2/24/2012 1:33 AM, Richard Maine wrote:
>
>> I believe that form='binary" to be a nonstandard spelling for what
>> amounts to the same thing as unformatted stream.
>
> Correct. FORM='UNFORMATTED', ACCESS='STREAM' to be specific. No record
> structure at all, just a stream of bytes. Standard in F2003.
>

That is very useful information. Thanks!

On 2/24/2012 8:37 AM, Steve Lionel wrote:
> On 2/24/2012 1:33 AM, Richard Maine wrote:
>
>> I believe that form='binary" to be a nonstandard spelling for what
>> amounts to the same thing as unformatted stream.
>
> Correct. FORM='UNFORMATTED', ACCESS='STREAM' to be specific. No record
> structure at all, just a stream of bytes. Standard in F2003.
>

Based on the information provided by Steve Lionel, I can confirm that
form='unformatted', access='stream' in gfortran produces the same output
as Intel's form='binary' on 32-bit windows. A several year old problem
solved. Thanks!