proc processdata |

Version 2.33 Jun'06
Scripts
| |
|
Manual page for proc_processdata(PL)
proc processdata performs various types of manipulations on the
current data set.
Here are some of the things this proc can do:
-
-
- accumulate
-
-
- break processing
-
-
- count/summate on instances
-
-
- joins
-
-
- row numbering
-
-
- compute percents
-
-
- reverse row order
-
-
- rotation of row/column matrix
-
-
- find mean, sd, min, max
-
-
- compute totals
The action attribute controls the type of processing to be done.
The result is usually a new data set (but sometimes the result is simply
the setting of some variables, as with action: totals).
When a new data set is created, it will be kept in memory along with the original data by default.
To discard the new data set and return to the original one, use
proc usedata.
For large data sets you can cause the result to overwrite the
original data set in memory by setting (stack: no).
In either case, after proc processdata is finished with an operation that creates a new data set
the result will automatically become the "current data set".
You can write the results to a file (outfile) for processing by
other programs, or to read back in and process further using proc getdata
filter.
For additional processing capabilities see also
proc getdata
filter.
Example
A Gallery example where this is used is
stock
where the available data is in reverse chronological order.
Since lineplot must work from left to right, proc processdata
is used to reverse the record order.
Variables that are set by processdata
NRECORDS = Number of rows in the data result.
NFIELDS = Number of fields per row in the data result.
TOTALS = If totals, percents, or accumulation are being done,
this variable will be set to hold the field total(s). If more than
one field is being operated on, this will be a comma-delimited
list of totals; individual totals may be accessed in your script
using something like the following, which would access the first
total in the list: #set T = $nmember(1,@TOTALS)
BREAKFIELD1 .. n = Current contents of break fields with action: breaks
action: stats sets a number of other variables, described in the stats section below.
The action attribute
The action attribute controls the type of operation that will be done.
Often the fields attribute is used to indicate which field(s) are involved.
-
-
action: accumulate
Rewrite field as a cumulative series (accumulation).
The field(s) to operate on must be given in the fields attribute.
For example, the data set on the left would be transformed to the one
on the right (fields: 2):
A21 3 A21 3
A22 5 --> A22 8
A23 2 A23 10
A24 1 A24 11
-
-
action: breaks
Perform break processing (the act of passing through an ordered
data set and taking some action when a key field or fields change).
Break processing is significantly more efficient than scanning a data
set multiple times with a select statement.
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified using the fields attribute.
The BREAKFIELD1 .. n variable(s) will be set to the current contents
of the break field(s). Your script can detect when the entire data set has
been processed by checking
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
For more discussion of break processing
see below.
-
-
action: breakreset
Reset the "current row" to the beginning of the data set,
for the occasional time when more than one pass through a data set will be done
using action: breaks .
This is automatically done if a new data set is read by proc getdata.
-
-
action: count
Collapse data by counting the number of instances of a key field.
This action can also summate some other field based on a key
field. Input data must be sorted (or at least grouped) on the key field.
Resulting data set will always have two fields.
One or two fields must be specified using the fields attribute.
If one field is specified, the result fields will be 1) key field, 2) count.
For example (fields: 1):
062698 062698 2
062698 --> 062898 1
062898 070198 3
070198 070498 1
070198
070198
070498
-
-
If two fields are specified, the result fields will be 1) key field, 2) sum
of the numeric contents found in the second specified field.
For example (fields: 1 2):
062698 4 062698 10
062698 6 --> 062898 3
062898 3 070198 9
070198 2 070498 2
070198 4
070198 3
070498 2
See also action: segment and action: segmentb described below, which are
similar.
A gallery example that uses action: count is
hitcount
-
-
action: join
Perform a relational join operation. (2.30+)
All records come from the current data set; left side records are selected using the leftselect
attribute and right side records are selected using rightselect.
The records are joined on one or more fields specified in the fields attribute.
The data must be ordered on these fields (integer fields should be ordered numerically; alphanumeric
fields should be ordered alphabetically).
One (and only one) left side record and one (and only one) right side record must exist in order to be joined
(but see leftjoin and rightjoin below).
For example, suppose we want a scatterplot where the X component and Y component
are represented in different sets of records within the data set:
001 X 4.3 001 X 4.3 001 Y 5.2
001 Y 5.2 --> 002 X 3.2 002 Y 2.9
002 X 3.2
002 Y 2.9
In order to do a scatterplot of X vs. Y we need to do a join so that all data for case 001 are
on the same row, and all data for case 002 are on the same row. We can do this using:
#proc processdata
showresults: yes
action: join
fields: 1
leftselect: @@2 = X
rightselect: @@2 = Y
Also available are action: leftjoin and action: rightjoin.
With a left join, if a left side record exists with no right side companion, the right side
is filled in with a missing data code (= by default; can be specified in missingdatacode
attribute). A right join is the same thing but in reverse.
-
-
action: numberrows
Prepend a row number field to each row. For example:
A AA AAA 0 1 A AA AAA 0
B BB BBB 0 ---> 2 B BB BBB 0
C CC CCC 1 3 C CC CCC 1
D DD DDD 1 4 D DD DDD 1
E EE EEE 0 5 E EE EEE 0
F FF FFF 1 6 F FF FFF 1
-
-
action: percent
Rewrite one or more fields as percentages of its field (column) total.
The field(s) to operate on must be given in the fields attribute.
For example (fields: 1):
8 40
4 --> 20
3 15
5 25
-
-
action: reverse
The last record becomes the first one; the
record order is reversed. For example (fields: 2):
AXB 34 DIF 14
BYA 22 --> CES 52
CES 52 BYA 22
DIF 14 AXB 34
-
-
action: rotate
First row becomes 1st field, 2nd row
becomes 2nd field, and so on. This may be useful
in that most of the plotting procs work from data fields,
but sometimes data is given (or is more intuitive) in rows.
For example:
A 2 4 6 8 10 --> A B
B 3 6 9 12 15 2 3
4 6
6 9
8 12
10 15
action: segment
Similar to action: count, except that instead of a count result, a range is produced,
useful for plotting bar segments, etc.
Resulting data set will always have three fields: 1) key field, 2) start position, 3) end position.
If one field is specified, the start and end position will be record numbers of the first
and last records in the range (where first record is 1). For example (fields: 1):
062698 062698 1 3
062698 ---> 070198 4 6
062698 070498 7 7
070198
070198
070198
070498
If two fields are specified, the start and end position will be the contents of the second
specified field, for the first and last records in the range.
For example (fields: 1 2):
062698 A 062698 A C
062698 B ---> 070198 D F
062698 C 070498 G G
070198 D
070198 E
070198 F
070498 G
action: segmentb
Same as action: segment, except that groups will butt up against one another.
For example (fields: 1 2):
062698 A 062698 A D
062698 B ---> 070198 D G
062698 C 070498 G G
070198 D
070198 E
070198 F
070498 G
-
-
action: stats
Find the mean, SD, min, max, etc. for data field(s), and put mean, SD, min, max (etc) into ploticus variables.
No new data set is created. Versions 2.30+.
The data field(s) to operate on should specified in the fields attribute.
Sets the following ploticus variables:
MEAN = mean of all numeric values found
SD = standard deviation
N = number of numeric values found
MIN = the lowest numeric value found
MIN_ID = contents of identifier/abscissa field in data row where minima was found
MAX = the highest numeric value found
MAX_ID = contents of identifier/abscissa field in data row where maxima was found
NMISSING = number of non-numeric observations
TOTAL = sum of all numeric values in the field
(The following is a change in functionality starting in 2.31:)
This action can operate on one data field or several.
The data field(s) to operate on should specified in the fields attribute.
If more than one data field is specified, the data values from all the fields will be examined as
a group and the result will still be a single mean, SD, etc.
tagfield may be specified (see below) to indicate an identifier or abscissa field
which will be used to identify min and max cases (MIN_ID and MAX_ID) and may be useful in annotating
the min and/or max with a later invocation of #proc annotate.
-
-
action: total
Compute field total(s) only and place total(s) into the variable
TOTALS (see above). No new data set is created.
Field(s) to be totalled are specified in the fields attribute.
The decimal format of the total(s) is controlled by the resultformat
attribute. If total(s) are to be written in presentable notation
(a spacer for thousands, etc.) the resultformat attribute may be
preceded by a n, e.g. n%7.0f.
Other attributes
fields
dfield
list
-
-
The field(s) to be operated on. Required for any action that involves
data fields.
Example: fields: 2 5 6 7
keepfields
dfield
list
-
-
If specified, only the listed fields in the original data set
will be kept. The others will be rejected.
action may be anything.
Example: keepfields: 4 5 6
rejectfields
dfield
list
-
-
If specified, the listed fields in the original data set
will not be kept. The others will be.
action may be anything.
Example: rejectfields: 1 2 3 4
tagfield
dfield
-
-
Used with action: stats to specify an identifier or abscissa data field,
which will be used to identify min and max cases.
fieldnames
namelist
-
-
If specified, result data fields will be assigned the given names.
These names can later be used in any plotting proc to identify data fields.
namelist is a space- or comma- delimited list of names.
Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.
Note that if field names are specified in proc getdata
and then proc processdata is used to alter the order of fields or
delete fields, then this fieldnames attribute must be
used in order to redefine the field names properly.
Example: fieldnames: date group n
resultformat
printf-spec
-
-
Controls the decimal format of rewritten percents, accumulations, totals.
Default is %g.
Example: resultformat: %f
Example for action: totals (see above): resultformat: n%g
keepall
yes | no
-
-
If yes, original fields are preserved when doing accumulate, percent, or total.
Thus the result will have more fields than the original data set. Default is no.
select
selection expression
-
-
Used when action is select in order to specify the selection
condition. Beginning in version 2.20 select can be used along with
any action.
Example: select: @@4 = A
stack
yes | no
-
-
The default is yes which causes
the result data set to be "stacked", and the original data set is preserved in memory.
If no, the result data set replaces the original data set in memory (this might
be desired for larger data sets).
showresults
yes | no
-
-
If yes the data are shown after processing, as a diagnostic aid.
outfile
filename
-
-
If specified, results are written to filename, in tab-delimited format,
for processing by other programs, or to read back in and process further using
proc getdata filter.
filename can be a pathname or stdout or stderr.
Use of this attribute implies stack: no
leftselect
select expression
rightselect
select expression
-
-
Used with action: join (and leftjoin, rightjoin).
Specifies selection conditions for left side of join and right side of join.
missingdatacode
string
-
-
Used with action: leftjoin (and rightjoin) to specify a missing data code to use for
filling in missing fields.
More on break processing
action: breaks performs break processing.
In data processing terminology, "break processing" is the act of passing
through a sorted data set and doing something special when a change is encountered in key field(s).
For example, if we were processing a list of charges ordered by paying budget
number, we could use a break processing strategy to pause and generate
a statement for one budget number, when we reached the point in the
data set where the budget numbers changed. Then we would continue on.
action: breaks allows a similar thing to be done with plotting.
Break processing is significantly more efficient than scanning the entire data
set multiple times with a select statement, especially with larger data sets.
The data set must be sorted such that key fields are grouped.
The key field(s) must be specified in the fields attribute.
proc processdata is generally called within a #loop. When proc processdata
finishes, the
current data set
will be the block of data from the previous break to the current break.
proc usedata must be used at the bottom of the loop, to set the
current data set
back to the larger data set that we are passing thru.
Subsequent invocations of proc processdata then continue
from the most recent break location.
Your script can access the current contents of the break field(s) via the
BREAKFIELD1 .. n variable(s).
Your script can detect when the entire data set has been processed by checking
the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).
The following is an example:
#loop
#proc processdata
action: breaks
fields: 1 2 3
#proc endproc
#if @NRECORDS = 0
#break
#endif
#proc page
title: Account @BREAKFIELD1
#proc bars
...
#proc usedata
original: yes
#endloop
Limits: up to 5 break fields may be used.
Comparisons for equality are limited to the first 50 characters.
|
 data display engine
Copyright Steve Grubb
|