So you get it [i.e., information on the nature of work carried out by other state auditors] from New Zealand, you get it from Australia, Minnesota. So there is a Minnesota auditor who did some good work on looking at performance measures. So there is no generally accepted standard out there, but there is a lot of commonality in the criteria that auditors are working with and you can get that information. So that gives us a basis. (Office auditor, January 1998)
This quote indicates how good practice standards are constructed. A state auditor carries out an initial engagement to evaluate whether performance measures disclosed by the auditee are in accordance with some socially constructed standards, however fragile the construction might be. The auditor then reports the standards that she/he used and the results of her/his empirical investigation. A collective knowledge validation process subsequently takes place, in which auditors and interested others in other jurisdictions examine whether these standards “make sense” to them. If so, these actors may decide to use the same standards, thereby reinforcing them and contributing to their spread. Insertion in a network of documents, practices and beliefs results in the standards becoming progressively perceived as universal, rational, fundamental, and eventually even “natural” dimensions of performance (Porter, 1995). Once the global network is sufficiently developed, the validated standards provide frames of meanings for auditors and may be used to represent the profession to the outside world (Power, 2003). It is through this collective process of knowledge production and validation that auditors become increasingly convinced, and are perceived increasingly by others, to speak authoritatively about government performance measurement.
The Office auditors also became involved in knowledge-producing institutions (Latour, 1999), established to sustain the production of facts and maintain their durability. The Office and several other Canadian state audit offices provide financial support to the CCAF, and are involved in the Canadian Council of Legislative Auditors, which organizes conferences on performance measurement. These conferences allow auditors to share information on the standards used by their peers, and assess the extent to which they are used across jurisdictions.
In sum, the Office becomes one element in the network of auditors and other actors who collectively are involved in developing and validating knowledge on the measurement of government performance. This network in effect operates international “laboratories”, providing inscriptions of government performance and audit practices. Similar to the dispersed laboratories of high energy physicists studied by Knorr Cetina (1999), state auditors network and develop their knowledge and expertise through e-mail exchanges, downloading, and then referring to, reports from each other’s jurisdictions, and occasionally meeting at conferences. However, like the microbiologists and cathedral builders but in contrast to the physicists studied by Knorr Cetina (1999), audit experiments are a matter of trial and error, of sharing what worked (or not) elsewhere, and are not guided by an explicit theoretical framework. It is through this large-scale laboratory of dispersed experiments that auditors (and other interested others) developed standards of performance measurement and gradually became self-confident that they are qualified to intervene in the performance measurement domain. The Office was thus equipped with a relatively solid, networked, background to which it could attach its claims of expertise. Dissenters to the Office’s claims would not only have to dispute the Office’s views but also what numerous other actors through experiments around the world had thought and asserted about performance measurement.
In contrast to the increasingly stable and self-confident network that supported the claims of the Office’s expertise in performance measurement, other occupations (notably program evaluators) had significant difficulties in trying to anchor their claims to expertise to a laboratory-sustained body of practices and inscriptions. Because it is mainly carried out within government settings, evaluation lacks the legitimacy of being based on business practice. It also tends to be more time-consuming and costly than auditing given the basic methodological differences between the two disciplines:
Evaluators do a lot of things. They might send out [written] interviews […], a satisfaction survey to see how satisfied are you in the program, how would you improve it, a mixture of closed ended and open-ended questions. Now, do auditors do that? They don’t really interview people. To me what auditors deal with is they deal with much more hard data. What auditors look at they really want to look at what are the inputs to the program, what are the outputs, those kinds of things. That is what they are looking at. They want the hard data. […] They don’t really go out and interview the public, “Are you happy with this performance measure?” They won’t do anything like that. [While] in evaluation there is a combination of both the qualitative and the quantitative aspects to it. […] So there is that qualitative component and that is very important because you can’t get certain things just straight with quantitative data. […] I don’t know how an auditor would do an audit of something that has very little quantitative data. (Evaluator #1, March 2003)
Whereas evaluators consider that performance can only be grasped through a detailed examination of program specifics and by taking into account qualitative dimensions of performance, auditors focus on the measurement of quantifiable inputs and outputs, reflecting a background in financial accounting (see alsoPower, 1997a).18 The Office, for example, was sceptical of non-quantitative measures and customer satisfaction surveys.
Program evaluators in Alberta experienced turbulence following government’s adoption of a performance measurement agenda (Bradley, 2001). A significant number of the people who were involved at the beginning of the 1990s in program evaluation had to reorient their career into business planning and performance evaluation or leave public service. One evaluator described the evaluation community as navigating “blindly”, especially since evaluators could not rely on a large-scale network of resources like that sustained by accountants, dedicated, among other things, to the construction of expert knowledge in performance measurement.19 Also, the reports of program evaluators were then not issued publicly, which made it more difficult for evaluators to be informed of experiments carried out by their peers. In contrast, the effectiveness of state auditors’ laboratories is crucially connected with auditors’ ability to publish and widely disseminate the successful accomplishment of their performance measurement and value-for-money experiments, both federally (where the Federal Auditor General has a tradition of publishing extensive value-for-money reports) and in each of the Canadian Provinces. Politicians and media widely use their reports, and in so doing reinforce the claims of auditors to expertise in these fields.
Also, auditors’ laboratories are ultimately aimed at the development of quasi-universal measures and standards of performance, which, like generally accepted accounting principles (GAAP), are presumed to facilitate users’ comparison of performance across different settings. Auditors’ claims to expertise were therefore more likely to appeal to Alberta politicians and public servants, especially given the institutional emphasis on cost containment and short-term efficiency. Although the evaluators generally viewed performance measures as “very superficial” and “very shallow”, we suggest that de-contextualized measures like those advocated by auditors possess a key rhetorical strength, generalizability (Latour, 1999). Evaluators appealed to restricted fields of production and specific expertise, in contrast to the auditors’ appeal to fields of wide-scale production and generalized expertise (Oakes et al., 1998). As a result, it is not surprising to find that program evaluation remains a marginal practice in the Alberta government – although there are a few “strong pockets of support” around it (Bradley, 2001).20