Re: SAS memory usage (was How long can a single line of sas code

On 12/18/05, Paul M. Dorfman <sashole@bellsouth.net> wrote:
> Puddin',
>
> I ran the test using SAS 9.1.3 DM under AIX.

I should have guessed, as you've presented AIX results
numerous times ...

> I know not precisely how fast the
> six processors are there or how much RAM the whole thing has. I know that I have
> had no problem sticking all the dimensions of a large DW (summarily worth
> perhaps 3-4 gig) into hash tables in a single DATA step, so I reckon RAM was not
> a problem in our context.
>
> As far as the memory usage per variable is concerned, that is what the log told
> me, so I sold it for what I bought it...

Well, 'twas what SAS reported. What else was there to present?

The ballpark 60% increase in mem usage over W2k (and I presume
XP) remains a mystery to po' me. The principal components of
var-related usage (i.e. storage of formats, informats, labels,
etc) are fixed across OS's? Spread across 1,000,000 array
elements, I doubt that a difference in basic DATA step startup
would explain very much of the difference.

Suffice to say I am glad neither you nor I are paying for
physical mem for AIX boxes out-of-pocket.

Salut,
Puddin'

************************************************** ***************
*** Puddin' Man PuddingDotMan at GmailDotCom **
************************************************** ***************;

> > -----Original Message-----
> > From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On
> > Behalf Of Pudding Man
> > Sent: Saturday, December 17, 2005 3:46 PM
> > To: SAS-L@LISTSERV.UGA.EDU
> > Subject: SAS memory usage (was How long can a single line of
> > sas code be?)
> >
> > On 12/16/05, Dorfman, Paul <paul.dorfman@fcso.com> wrote:
> > > On Fri, 16 Dec 2005 04:30:28 -0800, RolandRB
> > <rolandberry@HOTMAIL.COM>
> > > wrote:
> > >
> > > >What is the maximum length for a single line of sas code? I am
> > > >thinking of a massive retain statement that lists all the
> > variables
> > > >in a dataset with perhaps hundreds of variables. Come to
> > think of it,
> > > >what is the maximum number of variables for a dataset?
> > >
> > > Roland,
> > >
> > > The limit of 2**15-1=32767 in a data set (but NOT in the PDV) had
> > > existed in pre-V9. Nowadays it is only memory-bound at the
> > step compilation time.
> > > With a mere 1G of RAM, a SAS data set can easily swallow 1m+
> > > variables, although it takes the compiler a relatively long time to
> > > arrange everything properly in the symbol table (plus I reckon it
> > > takes much longer than that to write all the 1m rows by 18
> > variables
> > > junk to DICTIONARY.COLUMNS and update the rest of the
> > system metadata):
> > >
> > > 2 data n ; array nn [1000000] 8 ; run ;
> > > NOTE: The data set USER.N has 1 observations and 1000000 variables.
> > > NOTE: DATA statement used (Total process time):
> > > real time 38.93 seconds
> > > user cpu time 28.78 seconds
> > > Memory 364033k
> > >
> > > 3 data c ; array cc [1000000] $ 8 ; run ;
> > > NOTE: The data set USER.C has 1 observations and 1000000 variables.
> > > NOTE: DATA statement used (Total process time):
> > > real time 36.07 seconds
> > > user cpu time 26.42 seconds
> > > Memory 395265k
> > >
> > > As you see, as a rule of thumb, it takes approximately 400 bytes of
> > > RAM per variable to compile.
> >
> > ...
> >
> > Paul,
> >
> > Good post. Covered numerous relevant bases.
> >
> > However, having followed SAS mem usage to some extent over
> > the years, I found "400 bytes of RAM per variable to compile"
> > a bit hi.
> >
> > Consider:
> >
> > 148 options fullstimer;
> > 149 data aa;
> > 150 array x(200000);
> > 151 run;
> >
> > NOTE: The data set WORK.AA has 1 observations and 200000 variables.
> > NOTE: DATA statement used (Total process time):
> > real time 10.16 seconds
> > user cpu time 8.23 seconds
> > system cpu time 0.48 seconds
> > Memory 49822k
> >
> >
> > 152
> > 153 data aa;
> > 154 array x(1000000);
> > 155 run;
> >
> > NOTE: The data set WORK.AA has 1 observations and 1000000 variables.
> > NOTE: DATA statement used (Total process time):
> > real time 5:30.86
> > user cpu time 1:17.16
> > system cpu time 11.00 seconds
> > Memory 247993k
> >
> > As you can see from above, looks like I get mem usage of
> > about 250 b./var. Yours is >50% higher.
> >
> > You have faster machine with more mem. I have heard that
> > certain PROCs will use more mem when available: not at all
> > certain that such should be expected with a DATA step.
> >
> > I was running W2kP/9.1.3. You were running XPP/<same>?
> > Any obvious/not-so-obvious explanation for sizable diff.
> > in reported mem usage?
> >
> > Prost,
> > Puddin'
> >
> > ************************************************** ***************
> > *** Puddin' Man PuddingDotMan at GmailDotCom **
> > ************************************************** ***************;
> >
> > "Well, mama don't allow no Boogie-Woogie round here ...
> > We don't care what mama don't allow
> > We're gonna Boogie-Woogie anyhow.
> > - Traditional
> >
>