proc processdata


Version 2.33 Jun'06

Scripts


Manual page for proc_processdata(PL)

proc processdata performs various types of manipulations on the current data set. Here are some of the things this proc can do:

- accumulate
- break processing
- count/summate on instances
- joins
- row numbering
- compute percents
- reverse row order
- rotation of row/column matrix
- find mean, sd, min, max
- compute totals

The action attribute controls the type of processing to be done. The result is usually a new data set (but sometimes the result is simply the setting of some variables, as with action: totals).

When a new data set is created, it will be kept in memory along with the original data by default. To discard the new data set and return to the original one, use proc usedata. For large data sets you can cause the result to overwrite the original data set in memory by setting (stack: no). In either case, after proc processdata is finished with an operation that creates a new data set the result will automatically become the "current data set".

You can write the results to a file (outfile) for processing by other programs, or to read back in and process further using proc getdata filter.

For additional processing capabilities see also proc getdata filter.


Example

A Gallery example where this is used is stock where the available data is in reverse chronological order. Since lineplot must work from left to right, proc processdata is used to reverse the record order.


Variables that are set by processdata

NRECORDS = Number of rows in the data result.

NFIELDS = Number of fields per row in the data result.

TOTALS = If totals, percents, or accumulation are being done, this variable will be set to hold the field total(s). If more than one field is being operated on, this will be a comma-delimited list of totals; individual totals may be accessed in your script using something like the following, which would access the first total in the list: #set T = $nmember(1,@TOTALS)

BREAKFIELD1 .. n = Current contents of break fields with action: breaks

action: stats sets a number of other variables, described in the stats section below.


The action attribute

The action attribute controls the type of operation that will be done. Often the fields attribute is used to indicate which field(s) are involved.

action: accumulate     Rewrite field as a cumulative series (accumulation). The field(s) to operate on must be given in the fields attribute. For example, the data set on the left would be transformed to the one on the right (fields: 2):
	A21 3			A21 3
	A22 5		-->	A22 8 
	A23 2			A23 10
	A24 1			A24 11
action: breaks     Perform break processing (the act of passing through an ordered data set and taking some action when a key field or fields change). Break processing is significantly more efficient than scanning a data set multiple times with a select statement. The data set must be sorted such that key fields are grouped. The key field(s) must be specified using the fields attribute. The BREAKFIELD1 .. n variable(s) will be set to the current contents of the break field(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0). For more discussion of break processing see below.

action: breakreset     Reset the "current row" to the beginning of the data set, for the occasional time when more than one pass through a data set will be done using action: breaks . This is automatically done if a new data set is read by proc getdata.

action: count     Collapse data by counting the number of instances of a key field. This action can also summate some other field based on a key field. Input data must be sorted (or at least grouped) on the key field. Resulting data set will always have two fields. One or two fields must be specified using the fields attribute. If one field is specified, the result fields will be 1) key field, 2) count. For example (fields: 1):
	062698	 		062698 2
	062698 		  -->	062898 1
	062898 			070198 3
	070198 			070498 1
	070198	 
	070198
	070498 
If two fields are specified, the result fields will be 1) key field, 2) sum of the numeric contents found in the second specified field. For example (fields: 1 2):
	062698 4		062698 10
	062698 6	  -->	062898 3
	062898 3		070198 9
	070198 2		070498 2
	070198 4
	070198 3
	070498 2
See also action: segment and action: segmentb described below, which are similar. A gallery example that uses action: count is hitcount


action: join     Perform a relational join operation. (2.30+) All records come from the current data set; left side records are selected using the leftselect attribute and right side records are selected using rightselect. The records are joined on one or more fields specified in the fields attribute. The data must be ordered on these fields (integer fields should be ordered numerically; alphanumeric fields should be ordered alphabetically). One (and only one) left side record and one (and only one) right side record must exist in order to be joined (but see leftjoin and rightjoin below). For example, suppose we want a scatterplot where the X component and Y component are represented in different sets of records within the data set:
	001 X 4.3		001 X 4.3 001 Y 5.2
	001 Y 5.2	-->	002 X 3.2 002 Y 2.9
	002 X 3.2
	002 Y 2.9
In order to do a scatterplot of X vs. Y we need to do a join so that all data for case 001 are on the same row, and all data for case 002 are on the same row. We can do this using:
	#proc processdata
	showresults: yes
	action: join
	fields: 1
	leftselect: @@2 = X
	rightselect: @@2 = Y
Also available are action: leftjoin and action: rightjoin. With a left join, if a left side record exists with no right side companion, the right side is filled in with a missing data code (= by default; can be specified in missingdatacode attribute). A right join is the same thing but in reverse.


action: numberrows     Prepend a row number field to each row. For example:
	A AA AAA 0		1 A AA AAA 0
	B BB BBB 0     --->  	2 B BB BBB 0
	C CC CCC 1		3 C CC CCC 1
	D DD DDD 1		4 D DD DDD 1
	E EE EEE 0		5 E EE EEE 0
	F FF FFF 1		6 F FF FFF 1


action: percent     Rewrite one or more fields as percentages of its field (column) total. The field(s) to operate on must be given in the fields attribute. For example (fields: 1):
	8		40
	4	   -->	20
	3		15
	5		25


action: reverse     The last record becomes the first one; the record order is reversed. For example (fields: 2):
	AXB 34		DIF 14
	BYA 22	   -->	CES 52
	CES 52		BYA 22
	DIF 14		AXB 34 


action: rotate     First row becomes 1st field, 2nd row becomes 2nd field, and so on. This may be useful in that most of the plotting procs work from data fields, but sometimes data is given (or is more intuitive) in rows. For example:
	A 2 4 6 8 10	-->	A B
	B 3 6 9 12 15		2 3
				4 6
				6 9
				8 12
				10 15


action: segment     Similar to action: count, except that instead of a count result, a range is produced, useful for plotting bar segments, etc. Resulting data set will always have three fields: 1) key field, 2) start position, 3) end position.

If one field is specified, the start and end position will be record numbers of the first and last records in the range (where first record is 1). For example (fields: 1):

	062698			062698 1 3
	062698          --->	070198 4 6
	062698			070498 7 7
	070198
	070198
	070198
	070498
If two fields are specified, the start and end position will be the contents of the second specified field, for the first and last records in the range. For example (fields: 1 2):
	062698 A		062698 A C
	062698 B         --->	070198 D F
	062698 C		070498 G G
	070198 D
	070198 E
	070198 F
	070498 G

action: segmentb     Same as action: segment, except that groups will butt up against one another. For example (fields: 1 2):

	062698 A		062698 A D
	062698 B         --->	070198 D G
	062698 C		070498 G G
	070198 D
	070198 E
	070198 F
	070498 G


action: stats     Find the mean, SD, min, max, etc. for data field(s), and put mean, SD, min, max (etc) into ploticus variables. No new data set is created. Versions 2.30+. The data field(s) to operate on should specified in the fields attribute. Sets the following ploticus variables:
MEAN = mean of all numeric values found
SD = standard deviation
N = number of numeric values found
MIN = the lowest numeric value found
MIN_ID = contents of identifier/abscissa field in data row where minima was found
MAX = the highest numeric value found
MAX_ID = contents of identifier/abscissa field in data row where maxima was found
NMISSING = number of non-numeric observations
TOTAL = sum of all numeric values in the field

(The following is a change in functionality starting in 2.31:) This action can operate on one data field or several. The data field(s) to operate on should specified in the fields attribute. If more than one data field is specified, the data values from all the fields will be examined as a group and the result will still be a single mean, SD, etc. tagfield may be specified (see below) to indicate an identifier or abscissa field which will be used to identify min and max cases (MIN_ID and MAX_ID) and may be useful in annotating the min and/or max with a later invocation of #proc annotate.


action: total     Compute field total(s) only and place total(s) into the variable TOTALS (see above). No new data set is created. Field(s) to be totalled are specified in the fields attribute.
The decimal format of the total(s) is controlled by the resultformat attribute. If total(s) are to be written in presentable notation (a spacer for thousands, etc.) the resultformat attribute may be preceded by a n, e.g. n%7.0f.


Other attributes

fields     dfield list

The field(s) to be operated on. Required for any action that involves data fields.
Example: fields: 2 5 6 7

keepfields     dfield list

If specified, only the listed fields in the original data set will be kept. The others will be rejected. action may be anything.
Example: keepfields: 4 5 6

rejectfields     dfield list

If specified, the listed fields in the original data set will not be kept. The others will be. action may be anything.
Example: rejectfields: 1 2 3 4

tagfield     dfield

Used with action: stats to specify an identifier or abscissa data field, which will be used to identify min and max cases.

fieldnames     namelist

If specified, result data fields will be assigned the given names. These names can later be used in any plotting proc to identify data fields. namelist is a space- or comma- delimited list of names. Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.
Note that if field names are specified in proc getdata and then proc processdata is used to alter the order of fields or delete fields, then this fieldnames attribute must be used in order to redefine the field names properly.
Example: fieldnames: date group n

resultformat     printf-spec

Controls the decimal format of rewritten percents, accumulations, totals. Default is %g.
Example: resultformat: %f
Example for action: totals (see above): resultformat: n%g

keepall     yes | no

If yes, original fields are preserved when doing accumulate, percent, or total. Thus the result will have more fields than the original data set. Default is no.

select     selection expression

Used when action is select in order to specify the selection condition. Beginning in version 2.20 select can be used along with any action.
Example: select: @@4 = A

stack     yes | no

The default is yes which causes the result data set to be "stacked", and the original data set is preserved in memory. If no, the result data set replaces the original data set in memory (this might be desired for larger data sets).

showresults     yes | no

If yes the data are shown after processing, as a diagnostic aid.

outfile     filename

If specified, results are written to filename, in tab-delimited format, for processing by other programs, or to read back in and process further using proc getdata filter. filename can be a pathname or stdout or stderr. Use of this attribute implies stack: no

leftselect     select expression
rightselect     select expression

Used with action: join (and leftjoin, rightjoin). Specifies selection conditions for left side of join and right side of join.

missingdatacode     string

Used with action: leftjoin (and rightjoin) to specify a missing data code to use for filling in missing fields.


More on break processing

action: breaks performs break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and doing something special when a change is encountered in key field(s). For example, if we were processing a list of charges ordered by paying budget number, we could use a break processing strategy to pause and generate a statement for one budget number, when we reached the point in the data set where the budget numbers changed. Then we would continue on. action: breaks allows a similar thing to be done with plotting. Break processing is significantly more efficient than scanning the entire data set multiple times with a select statement, especially with larger data sets.

The data set must be sorted such that key fields are grouped. The key field(s) must be specified in the fields attribute. proc processdata is generally called within a #loop. When proc processdata finishes, the current data set will be the block of data from the previous break to the current break. proc usedata must be used at the bottom of the loop, to set the current data set back to the larger data set that we are passing thru. Subsequent invocations of proc processdata then continue from the most recent break location.

Your script can access the current contents of the break field(s) via the BREAKFIELD1 .. n variable(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0). The following is an example:

 	#loop
 	   #proc processdata
 	     action: breaks
 	     fields: 1 2 3
 	   #proc endproc
 
 	   #if @NRECORDS = 0
 	     #break
 	   #endif
 
 	   #proc page
 	   title: Account @BREAKFIELD1
 
 	   #proc bars
 	   ...
 
          #proc usedata
            original: yes
 
 	 #endloop

Limits: up to 5 break fields may be used. Comparisons for equality are limited to the first 50 characters.


data display engine  
Copyright Steve Grubb


Ploticus is hosted at http://ploticus.sourceforge.net
SourceForge Logo


Markup created by unroff 1.0,    June 02, 2006.