Evaluating the evaluation - EY, APVMA and the move to Armidale part one

Note to readers. This is the first of two posts examining the proposed shift of the Australian Pesticides and Veterinary Medicine Authority from Canberra to Armidale. I had intended to do a single post, but length dictated two. 

This first post reviews the Ernst and Young cost benefit analysis on the proposed transfer. The second will compare the dynamic elements in the transfer as they relate to the ACT and the broader Capital Territory and Northern NSW.    

The decision by the Commonwealth Government to shift the Australian Pesticides and Veterinary Medicine Authority (APVMA) from Canberra to Armidale has attracted widespread criticism. You will find examples here, here, here, here. Reflecting ownership concentration in the Australian media, the same stories are repeated multiple times across mast heads.

Central to the criticism is a cost benefit and risk analysis of the potential relocation of APVMA carried out by Ernst and Young (EY). Much of the commentary has taken headline numbers from the EY report. There has not been an evaluation of the report itself, nor of what that report might tell us about broader issues beyond this particular move. These posts attempts to fill that gap.

Headline Points

Given the stated regional development policy objective behind the move, three questions needed to be addressed:

• What are the costs and risks associated with the move?
• What are the gains from a regional development perspective?
• Do those gains outweigh the identified costs and risks?

The EY analysis concentrates on the first question.

The cost benefit analysis focuses on the financial costs of the move and is especially influenced by property and staff redundancy and replacement costs. It specifically excludes a range of benefits, including secondary benefits linked to the regional development objective. Depending on the assumption used, the analysis suggests a NPV cost over a twenty year period in the range of a bit over $9 million to $23.19 million.

 The risk analysis does highlight transition risks and options that may lead to higher costs. However, the assumptions used to generate the headline numbers are such that the numbers involved have little meaning.

The analysis of comparative economic impacts on the ACT and Armidale says little more than the move will have negligible economic impact on the ACT, substantial economic impact on Armidale. It does not properly capture the economic dynamics involved, nor the flow-on benefits beyond the narrow boundaries of the original Armidale.Dumaresq LGA  

Background

Established in 1993, APVMA is an Australian Government statutory authority created to centralise the registration of all agricultural and veterinary chemical products entering the Australian marketplace. Currently located in the Canberra suburb of Symonston,  APVMA regulates these products up to and including the point of retail sale.

The National Party has long had a policy of relocating agencies from capital cities to regional areas as a way of encouraging decentralisation. On 15 May 2015, the Minister for Agriculture and Water Resources (Barnaby Joyce) reaffirmed the Government’s commitment to increasing the regional presence of three Rural Research and Development Corporations (RDCs) and the Australian Pesticides and Veterinary Medicines Authority (APVMA) by establishing offices and/or relocating core operations from Canberra to regional Australia.

On 10 February 2016, the Minister announced that three RDCs were to establish regional offices outside Canberra (with the Rural Industries RDC relocating core operations to Wagga Wagga, the Fisheries RDC to establish a regional office in Adelaide and the Grains RDC to establish several regional offices outside Canberra) and that the proposed relocation of the APVMA to Armidale would go through the process of an independent cost benefit risk analysis.

On 25 November 2016 Barnaby Joyce, now Leader of the National Party and Deputy Prime Minister, announced  (link above) that APVMA would move to Armidale  The day before, Mathias Cormann as Minister for Finance issued an administrative order: This stated:

3 Authority
This instrument is made under subsection 22(1) of the Public Governance, Performance and Accountability Act 2013.
4 Location of corporate Commonwealth entities
(1) It is a policy of the Australian Government that a corporate Commonwealth entity with agricultural policy or regulatory responsibilities is to be located:
(a) in a regional community; and
(b) within 10 kilometres by road of the main campus of a regional university that is recognised for research and teaching in the field of agricultural science.
(2) In this section:
regional community means a community that is not within 150 kilometres by road of Canberra or the capital city of a State.
5 Application
The policy of the Australian Government mentioned in section 4 is applied to the Australian Pesticides and Veterinary Medicines Authority.

In announcing the move, Minister Joyce also released the EY cost benefit and risk analysis on the Armidale move.

Given the stated regional development policy objective behind the move, three questions need to be addressed:

• What are the costs and risks associated with the move?
• What are the gains from a regional development perspective?
• Do those gains outweigh the identified costs and risks?

Overview of the EY Evaluation

EY considered two options:

• Status quo. APVMA as is 
• APVMA moves to Armidale.

In the final decision stages before the 10 February  announcement, Towoomba appears to have been included in the mix, but was then dropped out. The wording of the administrative order would preclude Toowoomba since it is only 125k west of Brisbane. However, from the EY report, questions about Toowoomba do appear to have been included in staff surveys. For reasons I will outline later, the methodology applied by EY would have given broadly similar results for a move to any regional centre including Toowoomba.

In considering the two options, EY:

1. Assessed the economic costs and benefits of relocating APVMA to Armidale
2. Examined the key risks associated with relocation
3. Analysed the economic impacts on Canberra and Armidale.

 EY estimated an NPV (net present value) cost of the move to Armidale of $23.19 million over twenty years driven largely by the cost of a new building plus redundancy associated costs.  These were concentrated in the first five years and only partially offset by subsequent property savings. The discount rate used was 7%.

EY's risk assessment focused on the impact of the loss of technical assessment staff (regulatory scientists) on APVMA processes. It concluded that the costs a poorly executed move could be far higher than the economic costs identified via the cost benefit analysis. For example, the estimated cost of a one year delay in approvals for new agricultural products could be between $64 million and $193 million in reduced crop production..

On the relative economic impacts of the move on Canberra and Armidale, EY concluded that the move would cost the ACT economy around $108.88 million per annum gross equivalent to an economic hit of 0.2%, while Armidale would benefit by $77.54 million after the construction phase for a gain of 3.7%. The difference between the gross numbers represents leakages because of Armidale' smaller economy; more money would be spent outside Armidale.

I have given you the bald numbers since these form the basis of much of the discussion, especially in Canberra. I will now look at the numbers in detail. As we shall see, things are not always what they seem.

 Staff Attitudes and Costs

I am treating this item first because it is central to both the cost and risk analysis.

The percentage of staff who said they might relocate to Armidale was 15.2%. The cost side of the EY report therefore assumes that 84.8% of total staff will opt for redundancy with consequent redundancy, recruitment and training costs. The projected redundancy level also feeds into the risk analysis.

The reasons staff gave for unwillingness to relocate are instructive for everybody interested in effective decentralisation. Ranked by size, the reasons were:

• My partner will/may have difficulty in finding equivalent work - 123 staff members. 
• Limited opportunities for future employment progression   - 113 staff
• I have strong ties to the Canberra region - 108 staff 
• I have extended family responsibilities or receive assistance from family friends in this area - 84 staff
• I don't want to move children/dependents out of current or intended school - 72 staff
• I have concerns about the availability of suitably priced real estate (rent or buy) - 65 staff
• I have concerns about living in a regional area -  60 staff
• The proposed region does not support my cultural/community requirements - 39 staff
• I rely on specialist medical or other support services not available in the proposed region - 34 staff
• My family requires a special needs school/programs - 17 staff
• I know nothing about Armidale - 17 staff    

The importance of the partner issue is quite clear. In this context, 73% of partners were employed full time, a further 7% part time. If you add to this the numbers connected with disrupting schooling or other family or community relationships, you get a feel for just how hard it is to move existing activities.

I will return to this point in discussion. For the moment, I note that the sensitivity analysis conducted by EY indicated that a 20% increase in staff retention from the 15.2% base would reduce the NPV cost of the transfer by around two million dollars.

Cost Benefit Analysis

EY ruled out the following matters put forward by Armidale stakeholders from consideration :

• Co-location with UNE:  Given the nature of APVMA work, there were no identifiable benefits from co-location with UNE
• Enhanced proximity to end users: Given the nature of APVMA work, there were no identifiable benefits from proximity with end users. In any case, APVMA had access to the rural sector in Southern NSW
• NBN leverage: There were no or minimal leverage benefits from Armidale's NBN connection. To the degree that there were, Canberra would have the NBN by the time the move took place
• UNE offering courses in regulatory science: There were no obvious benefits here and in any case the costs of introducing such courses would need to be offset against the benefits. 

EY also noted the claims made about particular types of economic benefits claimed for Armidale, simply pointing to the later comparative analysis that concluded that Canberra lost more than Armidale gained. For reasons outlined later, this is a misuse of data.

Finally, EY explicitly stated in its assumptions that where a policy's primary objective is regional development, secondary benefits are listed only and not included in the net benefit calculations.

The practical effect of these restrictions limited the cost benefit analysis to the NPV of the relative financial costs associated with the transfer, including the estimated costs to industry, as compared to the status quo.

Within those costs, property costs including the time remaining on the existing lease on the Symonston facility dominated. This comes about because the cost benefit analysis is dominated by the front-end costs of a new building in Armidale. If an existing building is rented or a new one constructed and leased, the NPV cost falls from $23.19 million to $11.54 million on the EY assumptions.

Staff related costs are the next major element. On the basis of 84.8% redundancies, EY estimated the following costs:

• redundancy costs $1.83 million
• recruitment costs $2.62 million
• training costs $569,600
• additional oversight costs during the transition $200,00  

Because these various costs are concentrated in the first five and especially the first two years, they have considerable impact in NPV terms.  The effect of an increase in staff retention rates in reducing NPV costs has already been noted.

Other identified costs include:

• increased travel for APVMA staff $112,800 per annum subsequently indexed at the rate of inflation
• cost of secure internet connection $80,000 per annum subsequently indexed
• moving costs $516,000
• additional travel costs to industry starting at a base cost of just over $163,000 per annum subsequently indexed. 

In calculating costs to industry, EY excluded any additional time or accommodation costs that might be involved. To that degree, the estimates understate costs. In addition, in a comment stream on a post on my my personal blog, Monday Forum - George Brandis, drop bears, yowies and other Australian fauna, 2 tanners suggested that the move would reduce APVMA's advisory effectiveness. I am not convinced, although 2 tanners has worked in this space.

Finally, and as noted by EY in its risk analysis, action required to maintain effectiveness and to mitigate risks may lead to some increase in costs as well as in the timing of variations and costs. This is discussed further below under risk

Risk Assessment

In carrying out its risk analysis, EY classified risks into four groups:

• Risk 1. The APVMA is unable to effectively relocate or recruit and replace key APVMA executive, management and technical assessment staff within the first two years Likelihood high, impact high
• Risk 2. During transition and in the short term, the APVMA is unable to sustain its rate of effort for registration of new agricultural and veterinary chemical products. Likelihood possible, impact high
• Risk 3. The APVMA is unable to maintain and grow its capability in the medium term. Likelihood possible, impact high
• Risk 4. The APVMA has reduced access to stakeholders. Likelihood possible, impact low. 

EY then attempted to calculate potential costs, concluding that a one year lag in approval of new chemical products would result in a reduction of from $64 million to $193 million in agricultural crop value per annum. The assumptions here can be best described as heroic. As I understand them, I haven't tried to replicate the underlying spreadsheets, they appear to include:

1. Average national sales of new chemical products based on APVMA data are $59.938 million in year 1 rising to $213.043 million in year 2
2. Many new chemical products substitute for existing products. EY assumes that 5% (option 1) or 15% (option 2) represent new uses. The value of sales not substituting for existing uses is calculated by multiplying the numbers given in 1 for average national sales of new products by 5% and 15% respectively 
3. By expressing the amount calculated under 2 as a percentage of total product sales, EY generates a percentage representing the contribution of new products used in new ways to total product sales. 
4. Drawing on Deloitte Access Economics work, EY suggests that 68% of the total value of  crop production can be attributed to crop protection products. If you take the value of crop production and multiply it by .68, you get the estimated chemical contribution to crop value.
5.  EY then generates a figure for the contribution of new chemicals to new uses and expanded crop production by taking the percentage generated under 3 and applying it to the estimated value of the chemical contribution to crop production calculated at 6.  
6. EY then calculates the value of lost crop production as a consequence of deferral of regulatory approvals by applying lags to the numbers generated at 5.       

I imagine that your eyes have glazed. However, you can see the range of assumptions required to generate the numbers that grabbed the headlines. I also have a problem, perhaps I'm just tired, with EY's treatment of lags. In calculating the potential risk loss for a one year regulatory lag, they add together the estimated production totals for the first two years to calculate the loss. I can't see the rationale for that approach.

In the end, the most that can be said for the EY numbers is that the modelling provides an indicative base setting out some issues.

EY then goes on to consider various options for risk amelioration. In this context, the APVMA has released its initial transition plan setting out its own response to service maintenance while complying with the Government's policy wishes. This says in part.

2.2 Virtual network of regulatory scientists 

Suitably qualified regulatory scientists are in short supply and it takes time to train them to be fully functional. Continued access to these regulatory specialists is one of the highest risks associated with the relocation. The priority, therefore, is to implement a new way of working through a ‘virtual science network’ to maintain access to the skills needed. With such a network, regulatory scientists would work remotely from anywhere in Australia, but still be connected to the Armidale office through a new digital strategy. This means the APVMA will not be constrained by availability of scientists in Armidale.

2.3 Regulatory hub in Armidale 

The APVMA handles over 5000 applications every year. The office in Armidale will embody the flexible teambased approach taken by the APVMA to handle so many applications with around 200 staff. There will be a call centre and case management hub to service clients; collaborative space to enable people to come together to work on applications – even if people are working remotely; and technology enabled facilities with virtual meeting rooms. This infrastructure will also make it easier to work with our regulatory counterparts in other countries. The facility will be modern and provide a quality employee experience – acting as an inducement for staff to move to Armidale. The CEO, executive, key management roles and staff from case management, call centre, application administration, corporate, legal, licensing and compliance will be based in Armidale, along with those regulatory scientists wishing to be located in Armidale. While many regulatory scientists will be operating remotely, the scientific leadership will be based in Armidale. Over time, the APVMA will build networks with universities to bring through the next wave of regulatory specialists through the system. The initial target will be 100 people operating out of Armidale, building up to 150 over time. 

2.4 Client service in a digital world 

The APVMA will enhance its online client experience by making it easier for applicants to submit data underpinning an application. This will be done by implementing internationally agreed standards for electronic submission of data (where possible), expanding the functionality of the client portal to enable full tracking of applications and improving on-line communication and correspondence with the APVMA. The APVMA will enhance its capability to conduct web conferencing with clients, regardless of where they are in the world. The APVMA also recognises that some clients still want to discuss their applications face to face – to save client travel expenses, the APVMA will trial ‘surgeries’ in Sydney and Melbourne.

On the surface, this seems a sensible approach. The move to Armidale is, I suspect,  accelerating changes in service delivery that were already under way. At this point I have not attempted to estimate the effect of the APVMA approach on the analysis beyond noting that the employment shifts will be smaller and somewhat longer term than the numbers used in the modelling, the risk profile reduced.  

Meantime, the University of New England has now introduced a suite of options in regulatory science from Graduate Certificate to PhD. While this possibility was dismissed by EY as irrelevant to the cost benefit analysis, it does affect the risk profile associated with the move.

Economic Impacts

The last part of the EY report compares the economic impacts on the ACT and the former Armidale Dumaresq LGA.  To the degree that the EY approach addresses follow on benefits, they are meant to be dealt with in this section.

In assessing relative economic impacts, EY uses the REMPLAN input-output model. As noted earlier, the analysis concluded that while the economic impact on Armidale would be relatively far greater than the loss in economic activity to the ACT, the total gain to Armidale would be less than the total loss experienced by the ACT.

The outcome here was pre-determined determined by the structure of the model. Because the ACT (population 357, 278 in 2011) is far larger than the previous Armidale Dumaresq LGA (population 24,105), more spend leaks from Armidale than the ACT. The analysis says little more than the move will have negligible economic impact on the ACT, substantial economic impact on Armidale. It does not properly capture the economic dynamics involved, nor the flow-on benefits beyond the narrow boundaries of the Armidale.Dumaresq LGA. A dollar not spent in those boundaries is not a lost dollar.

Conclusion   

Given the stated regional development policy objective behind the move, I suggested that three questions needed to be addressed:

• What are the costs and risks associated with the move? 
• What are the gains from a regional development perspective? 
• Do those gains outweigh the identified costs and risks?

The EY analysis concentrates on the first question.

The cost benefit analysis focuses on the financial costs of the move and is especially influenced by property and staff redundancy and replacement costs. It specifically excludes a range of benefits including secondary benefits linked to the regional development objective. Depending on the assumption used, the analysis suggests a NPV cost over a twenty year period in the range of a bit over $9 million to $23.19 million.

 The risk analysis does highlight transition risks and options that may lead to higher costs. However, the assumptions used to generate the headline numbers are such that the numbers involved have little meaning.

The analysis of comparative economic impacts on the ACT and Armidale says little more than the move will have negligible economic impact on the ACT, substantial economic impact on Armidale. It does not properly capture the economic dynamics involved, nor the flow-on benefits beyond the narrow boundaries of the original Armidale.Dumaresq LGA  

In my nextpost  I will look at the dynamic aspects comparing the ACT and broader Capital Territory region and Northern NSW to try to tease out some of the broader questions involved from a regional development perspective.