Bug 3274050 and 4.1.1 beta

Bug 3274050 seems to have ground to a halt... fixed, but pending.

It's not fixed in my tests of the 4.1.1 beta, and since this is the
only one of my bugs which persists, it's the fly in my ointment.

One consequence of this is a security exposure for me. I have a log
file which has to be a+w so all processes can add log entries.
However, it also has to be a+r as well, which means that processes can
read each others log entries.

I try to keep sensitive information out of the log, but it cannot be
guaranteed...

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

Swifty <steve.j.swift@gmail.com> wrote:

> Bug 3274050 seems to have ground to a halt... fixed, but pending.
>
> It's not fixed in my tests of the 4.1.1 beta, and since this is the only
> one of my bugs which persists, it's the fly in my ointment.
>
> One consequence of this is a security exposure for me. I have a log file
> which has to be a+w so all processes can add log entries. However, it also
> has to be a+r as well, which means that processes can read each others log
> entries.

Wouldn't it be better to have a log per process, with each process only able
to update its own log?

And separately to run some sort of log aggregator that produces a merged log
somewhere for your use?

--
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".

On Fri, 24 Feb 2012 14:36:44 +0000, Jeremy Nicoll - news posts
<jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:

>Wouldn't it be better to have a log per process,

I'm thinking of things like my outbound email log, my error log and
other such "global" logs that can potentially be appended to by any
process running code that I've written. The logs are not peculiar to
the process, but the functions that the process uses.

But anyway, the bug is that I cannot write to a file over which I have
write authority, and that is what should get fixed. Then everything in
my garden would be rosy.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

Swifty <steve.j.swift@gmail.com> wrote:

>On Fri, 24 Feb 2012 14:36:44 +0000, Jeremy Nicoll - news posts
><jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:
>
>>Wouldn't it be better to have a log per process,
>
> I'm thinking of things like my outbound email log, my error log and other
> such "global" logs that can potentially be appended to by any process
> running code that I've written. The logs are not peculiar to the process,
> but the functions that the process uses.

I'd still consider having the processes write what are in essence
transaction requests (add this text to a log) somewhere process-specific,
and have some other process pick those up and update the actual log. Apart
from anything else you can circumvent the problems of having multiple
processes simultaneously trying to append data to a single log file that
way.

It's probably be overkill but there may be a message-queue handling
app/daemon you could use for that.


> But anyway, the bug is that I cannot write to a file over which I have
> write authority, and that is what should get fixed. Then everything in my
> garden would be rosy.

Escept for the multiple simultaneous access issue. IIRC (having seen your
common routines code) you don't have your log write code do multiple
retries, nor have a fallback if none of those work.

I did do on some of the TSO/ISPF applications I wrote long ago; I also made
sure that if someone chose to view such a log then the app would make a
temporary copy of it (in VIO usually) and let them browse that, so that no
viewing process would hold an ENQ on the real log for more than a tiny
snippet of time.

--
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".

Jeremy Nicoll - news posts wrote:
> Swifty <steve.j.swift@gmail.com> wrote:
>
>> On Fri, 24 Feb 2012 14:36:44 +0000, Jeremy Nicoll - news posts
>> <jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:
>>
>>> Wouldn't it be better to have a log per process,
>> I'm thinking of things like my outbound email log, my error log and other
>> such "global" logs that can potentially be appended to by any process
>> running code that I've written. The logs are not peculiar to the process,
>> but the functions that the process uses.
>
> I'd still consider having the processes write what are in essence
> transaction requests (add this text to a log) somewhere process-specific,
> and have some other process pick those up and update the actual log. Apart
> from anything else you can circumvent the problems of having multiple
> processes simultaneously trying to append data to a single log file that
> way.
>
> It's probably be overkill but there may be a message-queue handling
> app/daemon you could use for that.
>
>
>> But anyway, the bug is that I cannot write to a file over which I have
>> write authority, and that is what should get fixed. Then everything in my
>> garden would be rosy.
>
> Escept for the multiple simultaneous access issue. IIRC (having seen your
> common routines code) you don't have your log write code do multiple
> retries, nor have a fallback if none of those work.
>
> I did do on some of the TSO/ISPF applications I wrote long ago; I also made
> sure that if someone chose to view such a log then the app would make a
> temporary copy of it (in VIO usually) and let them browse that, so that no
> viewing process would hold an ENQ on the real log for more than a tiny
> snippet of time.
>
Logs are tricky things. A lot depends on what you're doing, how it's done and
what you expect the log to show you.

Sometimes you simply MUST have a single log file to track the exact sequence of
events across multiple pieces of code, in my case, THE macros on Windows. The
timestamp on Windows isn't fine enough to merge logs into the proper sequence of
events.

--

Les (Change Arabic to Roman to email me)

On 2012-02-24 18:25, Jeremy Nicoll - news posts wrote:
> Swifty<steve.j.swift@gmail.com> wrote:
>
>> On Fri, 24 Feb 2012 14:36:44 +0000, Jeremy Nicoll - news posts
>> <jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:
>>
>>> Wouldn't it be better to have a log per process,
>>
>> I'm thinking of things like my outbound email log, my error log and other
>> such "global" logs that can potentially be appended to by any process
>> running code that I've written. The logs are not peculiar to the process,
>> but the functions that the process uses.
>
> I'd still consider having the processes write what are in essence
> transaction requests (add this text to a log) somewhere process-specific,
> and have some other process pick those up and update the actual log. Apart
> from anything else you can circumvent the problems of having multiple
> processes simultaneously trying to append data to a single log file that
> way.
>
> It's probably be overkill but there may be a message-queue handling
> app/daemon you could use for that.
>
>
>> But anyway, the bug is that I cannot write to a file over which I have
>> write authority, and that is what should get fixed. Then everything in my
>> garden would be rosy.
>
> Escept for the multiple simultaneous access issue. IIRC (having seen your
> common routines code) you don't have your log write code do multiple
> retries, nor have a fallback if none of those work.
>
> I did do on some of the TSO/ISPF applications I wrote long ago; I also made
> sure that if someone chose to view such a log then the app would make a
> temporary copy of it (in VIO usually) and let them browse that, so that no
> viewing process would hold an ENQ on the real log for more than a tiny
> snippet of time.
>
Hi!
It is not only log files involved. All files created by a Rexx process have the same bl--dy problem.
I have been forced to either create the files outside rexx, or use a linux command to alter the permissions.

Swifty is in the right. It is a bug!

/dg

On Fri, 24 Feb 2012 17:25:42 +0000, Jeremy Nicoll - news posts
<jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:

>Except for the multiple simultaneous access issue. IIRC (having seen your
>common routines code) you don't have your log write code do multiple
>retries, nor have a fallback if none of those work.

"Call stream log,'c','open write append'" and a simple:
"call lineout log,date() time() text"

.... are all you need, at least in the Linux environment. Linux assumes
that the programmer knows what they want, so there are no interlocks
to prevent concurrent write from multiple processes.

But, like you, I was leery of problems. So I wrote a test program.
You passed it an integer, N.
It started N "threads".
Each thread wrote 10,000 lines to the log file, numbered 1-10,000.

When the threads had all ended, the main routine scanned the log file
to ensure that it had N entries for every number from 1-10,000

I ran it many times, with values of N from 1-100. It never failed. I
was right to trust Linux to do what I wanted, and not what it thought
should happen. :-)

As it happens, I do also check the results of OPEN and LINEOUT, and
log failures in the common "Things you need to worry about" log (yes,
another of those global log files I'm so enamoured of).

Of course, it's conceivable that there is an error in the "Things you
need to worry about" log code, but this is getting recursive.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

Swifty <steve.j.swift@gmail.com> wrote:

>On Fri, 24 Feb 2012 17:25:42 +0000, Jeremy Nicoll - news posts
><jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:
>
>>Except for the multiple simultaneous access issue. IIRC (having seen your
>>common routines code) you don't have your log write code do multiple
>>retries, nor have a fallback if none of those work.
>
>"Call stream log,'c','open write append'" and a simple:
>"call lineout log,date() time() text"
>
>... are all you need, at least in the Linux environment.

Is that because you're only writing a single line? If so it'd imply that
the writing of a line is an atomic process in the file system and possibly
the OS. I don't see why it should be though - I think I thought unix files
were just character streams.

What stops a line being written by one process from being mixed up with one
from another?


> Linux assumes that the programmer knows what they want, so there are no
> interlocks to prevent concurrent write from multiple processes.

So why did your testing never produce mixed-up output?


> But, like you, I was leery of problems. So I wrote a test program. You
> passed it an integer, N. It started N "threads". Each thread wrote 10,000
> lines to the log file, numbered 1-10,000.

Did those threads run under separate processes?

>When the threads had all ended, the main routine scanned the log file
>to ensure that it had N entries for every number from 1-10,000
>
>I ran it many times, with values of N from 1-100. It never failed. I
>was right to trust Linux to do what I wanted, and not what it thought
>should happen. :-)

But why? I would wan to understand why output hadn't been mixed up; that
it hadn't suggests to me that the test was in some way inadequate.

>As it happens, I do also check the results of OPEN and LINEOUT, and
>log failures in the common "Things you need to worry about" log (yes,
>another of those global log files I'm so enamoured of).

If the OS doesn't protect the file, I'm not sure that testing the return
code from a lineout would tell you anything. A lineout would have caused a
number of characters to be sent to a file ok. But if something else is also
sending chars to the file that wouldn't affect lineout.

>Of course, it's conceivable that there is an error in the "Things you
>need to worry about" log code, but this is getting recursive.


--
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".

danfan46 <danfan46@hotmail.com> wrote:

> Swifty is in the right. It is a bug!

I wasn't disagreeing that the bug is a bug.

--
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".

LesK <5mre20@tampabay.rr.com> wrote:

> Logs are tricky things. A lot depends on what you're doing, how it's done
> and what you expect the log to show you.
>
> Sometimes you simply MUST have a single log file to track the exact
> sequence of events across multiple pieces of code, in my case, THE macros
> on Windows. The timestamp on Windows isn't fine enough to merge logs into
> the proper sequence of events.

I know about the timestamp problems... But Windows does at least allow a
single process to own a file, so you'd expect not to get log lines from one
writer mixed up with those of another.

I don't know that you can guarantee that sets of lines that several
processes wish to append to a common file will be written in the right order
though. Surely the best you can do is ensure that while one process's
output is being written, the other processes wait for an opportunity to do
their writes. You'd have to implement that wait/retry yourself I think, and
that'd probably mean you had no control over the order in which the tasks
actually got access to the file.

I know of no way in rexx to queue a desire to open a file so that each task
can later be given access to the log in the right order (I mean, without
designing such a thing, maybe using a shared data structure). Even if that
werre possible, is there a mechanism that does this at the OS level so that
apps written in other languages can also share the same log file?

--
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".

On 2/24/2012 4:26 AM, Swifty wrote:
> Bug 3274050 seems to have ground to a halt... fixed, but pending.
>
> It's not fixed in my tests of the 4.1.1 beta, and since this is the
> only one of my bugs which persists, it's the fly in my ointment.
>
> One consequence of this is a security exposure for me. I have a log
> file which has to be a+w so all processes can add log entries.
> However, it also has to be a+r as well, which means that processes can
> read each others log entries.
>
> I try to keep sensitive information out of the log, but it cannot be
> guaranteed...
>
The fix for this bug was accidentally omitted from the beta. We'll spin
a second beta version soon with the fix included.

Rick

On Feb 25, 5:11*am, Rick McGuire <object.r...@gmail.com> wrote:
> On 2/24/2012 4:26 AM, Swifty wrote:> Bug 3274050 seems to have ground to a halt... fixed, but pending.
>
> > It's not fixed in my tests of the 4.1.1 beta, and since this is the
> > only one of my bugs which persists, it's the fly in my ointment.
> > ...
>
> The fix for this bug was accidentally omitted from the beta. *We'll spin
> a second beta version soon with the fix included.


Yeah, sorry about that Steve. I'm going to build new Debian packages
this today.

--
Mark Miesfeld

On Sat, 25 Feb 2012 08:00:31 +0000, Jeremy Nicoll - news posts
<jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:

>Surely the best you can do is ensure that while one process's
>output is being written, the other processes wait for an opportunity to do
>their writes.

Again, my test program sheds light on this, at least under Linux. The
log entries written also included the PIDs of the "threads". In all
cases, the output from individual PIDs were in the correct order in
the log.

It is possible that the absolute order of the log entries may not
agree 100% with the precise instants when competing PID's attempted to
write to the log, but this didn't concern me (and still does not) as I
have no processes which depend on the real-time sequence of events
across processes.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

On Sat, 25 Feb 2012 07:48:24 +0000, Jeremy Nicoll - news posts
<jn.nntp.scrap007@wingsandbeaks.org.uk> wrote:

>Is that because you're only writing a single line? If so it'd imply that
>the writing of a line is an atomic process in the file system and possibly
>the OS. I don't see why it should be though - I think I thought unix files
>were just character streams.

Each call to my logging code results in a single LINEOUT, or in some
cases a single CHAROUT, but I'll avoid saying that I'm writing a
single line as the string passed to the logging code may contain
embedded linends (I'm ignoring record-based file systems). The
resultant entries are never interspersed, so these are indeed atomic
filesystem actions.

One very important aspect is that it is crucial to open the output
with the "APPEND" option. This seems to force REXX to use a single
buffer, but I have no idea how this effect is achieved internally.
Without the APPEND option, competing threads may lose lines of output.

This was also one of the reasons for writing my test program. I knew
that an implicit open, or a simple "open write" would lead to loss of
entries, and someone suggested the APPEND option.
The documentation on the APPEND option implied that it would do what I
needed, but I like to be sure that it works in practice. Over a period
of about 10 years, I've seen no evidence of any failures.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

On Sat, 25 Feb 2012 09:08:52 -0800 (PST), Mark Miesfeld
<miesfeld@gmail.com> wrote:

>Yeah, sorry about that Steve. I'm going to build new Debian packages
>this today.

No need for apologies, Mark; I'm eternally grateful for the work
you're all doing on Open Object REXX.

I've had my current job for over 35 years, and apart from a brief
period at the beginning I've been coding in REXX ever since. Given the
rate of change in the IT industry, I consider myself very lucky to
have been doing essentially the same job for that length of time.(*)

With the advances introduced in IBM Object REXX, and then Open Object
REXX, I've managed to become more productive with time, despite my
mental faculties starting to slip.

------------------------------------------------------------------------
(*) The trick of staying employed writing in REXX was to find a
situation where my code ran on a server, and it doesn't matter one
iota what language the code is written in. My code runs CGI scripts
inside an Apache webserver.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk