Peterborough SIB

Rikers Island social impact bond (SIB) – Success or failure?

Emma Tomkinson Rikers Island, Social Impact Bonds , , , , ,

There’s been a lot of discussion over tRikershe past few weeks as to whether Rikers Island was a success or failure and what that means for the SIB ‘market’. You can read the Huffington Post learning and analyses from investors and the Urban Institute as to the benefits and challenges of this SIB. But I think the success and failure discussion fails to recognise the differences in objectives and approaches between SIBs. So I’d like to elaborate on one of these differences, and that’s the attitude towards continuous adaptation of the service delivery model. Some SIBs are established to test whether a well-defined program will work with a particular population. Some SIBs are established to develop a service delivery model – to meet the needs of a particular population as they are discovered.

1.     Testing an evidence-based service-delivery model

This is where a service delivery model is rigorously tested to establish whether it delivers outcomes to this particular population under these particular conditions, funded in this particular way. These models are often referred to as ‘evidence-based programs’ that have been rigorously evaluated. The US is further ahead than other countries in the evaluation of social programs, so while these ‘proven’ programs are still in the minority, there are more of them in the US than elsewhere. These SIBs are part of a movement to support and scale programs that have proven effective. They are also part of a drive to more rigorously evaluate social programs, which has resulted in some evaluators attempting to keep all variables constant throughout service delivery.

An evidence-based service delivery model might:

  • be used to test whether a service delivery model that worked with one population will work with another;
  • be implemented faithfully and adhered to;
  • change very little over time, in fact effort may be made to keep all variables constant e.g. prescribing the service delivery model in the contract;
  • have a measurement focus that answers the question ‘was this service model effective with this population’?

“SIBs are a tool to scale proven social interventions. SIBs could fill a critical void: other than market-based approaches, a structured and replicable model for scaling proven solutions has not existed previously. SIBs can give structure to the critical handoff between philanthropy (the risk capital of social innovation) and government (the scale-up capital of social innovation) to bring evidence-based interventions to more people.” (McKinsey (2012) From potential to action: Bringing social impact bonds to the US, p.7).

2.    Developing a service delivery model

This is where you do whatever it takes to deliver outcomes, so that the service is constantly evolving. It may include an evidence-based prescriptive service model or a combination of several well evidenced components, but is expected to be continuously tested and adjusted. It may be coupled with a flexible budget (e.g. Peterborough and Essex) to pay for variations and additions services that were not initially foreseen. This approach is more prevalent in the UK.

A continuously adjusted service delivery model might:

  • be used to deliver services to populations that have previously not received services, to see whether outcomes could be improved;
  • involve every element of service delivery being continuously analysed and refined in order to achieve better outcomes;
  • continuously evolve – the program keeps adapting to need as needs are uncovered;
  • have a measurement focus that answers the question ‘were outcomes changed for this population’?

Andrew Levitt of Bridges Ventures, the biggest investor in SIBs in the UK, “There is no such thing as a proven intervention. Every intervention can be better and can fail if it’s not implemented properly –it’s so harmful to start with the assumption that it can’t get better.” (Tomkinson (2015) Delivering the Promise of Social Outcomes: The Role of the Performance Analyst p.18)

Different horses for different courses

Rikers New York City

The New York City SIB was designed to test whether the Adolescent Behavioral Learning Experience (ABLE) program would reduce the reoffending of the young offenders exiting Rikers Island. Fidelity to the designated service delivery model was prioritised, in order to obtain robust evidence of whether this particular program was effective. WYNC News reported that “Susan Gottesfeld of the Osborne Association, the group that worked with the teens, said teens needed more services – like mental health care, drug treatment and housing assistance – once they left the jail and were living back in their neighbourhoods.”

In a July 28 New York Times article by Eduardo Porter, Elizabeth Gaynes, Chief Executive of the Osborne Association is quoted as saying “All they were testing is whether M.R.T. by itself would make a difference, not whether something you could do in a jail would make a difference,” Ms. Gaynes said. “Even if we could have raised money to do other stuff, we were not allowed to because we were testing M.R.T. alone.”

This is in stark contrast with the approach taken in the Peterborough SIB. Their performance management approach was a continuous process of identifying these additional needs and procuring services to meet them. The Peterborough SIB involved many adjustments to its service over the course of delivery. For example, mental health support was added, providers changed, a decision was made to meet all prisoners at the gate… as needs were identified, the model was adjusted to respond. (For more detail, see Learning as They Go p.22, Nicholls, A., and Tomkinson, E. (2013). Case Study: The Peterborough Pilot Social Impact Bond. Oxford: Saïd Business School, University of Oxford.)

Neither approach is necessarily right or wrong, but we should avoid painting one SIB a success or failure according to the objectives and approach of another. What I’d like to see is a question for each SIB: ‘What is it you’re trying to learn/test?’ It won’t be the same for every SIB, but making this clear from the start allows for analysis at the end that reflects that learning and moves us forward. As each program finishes, let’s not waste time on ‘Success or failure?’, let’s get stuck into: ‘So what? Now what?’

Huge thanks to Alisa Helbitz and Steve Goldberg for their brilliant and constructive feedback on this blog.

Fewer criminals or less crime? Frequency v binary measures in criminal justice

Emma Tomkinson Criminal Justice, Social Impact Bonds, Social Impact Measurement, Social Innovation , , , , , , , , ,

The June 2013 interim results released by the Ministry of Justice gave us a chance to examine the relationship between the number of criminals and the number of crimes they commit. The number of criminals is referred to as a binary measure, since offenders can be in only one of two categories: those who reoffend and those who don’t. The number of crimes is referred to as a frequency measure, as it focuses on how many crimes a reoffender commits.

The payments for the Peterborough SIB are based on the frequency measure. Please note that the interim results are not calculated in precisely the same way as the payments for the SIB will be made. [update: the results from the first cohort of the Peterborough SIB were released in August 2014 showing a reduction in offending of 8.4% compared to the matched national comparison group.]

In the period the Peterborough SIB delivered services to the first cohort (9 Sept 2010-1July 2012), the proportion of crimes committed over the six months following each prisoner’s release reduced by 6.9% and the proportion of criminals by 5.8%. In the same period, there was a national increase in continuing criminals of 5.4%, but an even larger increase of 14.5% in the number of crimes they commit. The current burning issue is not that there are more reoffenders, it is that those who reoffend are reoffending more frequently.

Criminals or crime 1Criminals (binary measure) in this instance are defined as the “Proportion of offenders who commit one or more proven reoffences”. A proven reoffence means “proven by conviction at court or a caution either in those 12 months or in a further 6 months”, rather than simply being arrested or charged.

Crime (frequency measure) in this instance is defined as “Any re-conviction event (sentencing occasion) relating to offences committed in the 12 months following release from prison, and resulting in conviction at court either in those 12 months or in a further 6 months (Note: excludes cautions).”

The two measures are related – you would generally expect more criminals to commit more crimes. But the way reoffending results are measured creates incentives for service providers. If our purpose is to reduce crime and really help those who impose the greatest costs on our society and justice system, we would choose a frequency measure of the number of crimes. If our purpose is to help those who might commit one or two more crimes to abstain from committing any at all, then we would choose a binary measure.Criminals or crime 2Source of data: NSW Bureau of Crime Statistics and Research

The effect of the binary measure in practice: Doncaster Prison

A Payment by Results (PbR) pilot was launched in October 2011 at Doncaster Prison to test the impact of a PbR model on reducing reconvictions. The pilot is being delivered by Serco and Catch22 (‘the Alliance’). The impact of the pilot is being assessed using a binary outcome measure, which is the proportion of prison leavers who are convicted of one or more offences in the 12 months following their release. The Alliance chose to withdraw community support for offenders who are reconvicted within the 12 month period post-release as they feel that this does not represent the best use of their resources. Some delivery staff reported frustration that support is withdrawn, undermining the interventions previously undertaken. (Ministry of Justice, Process Evaluation of the HMP Doncaster Payment by Results Pilot: Phase 2 findings.)

I have heard politicians and policy makers argue that the public are more interested in reducing or ‘fixing’ criminals than helping them offend less, and thus the success of our programmes needs to be based on a binary measure. I don’t think it’s that hard to make a case for reducing crime. People can relate to a reduction in aggravated burglaries. Let’s get intentional with the measures we use.

What do the Peterborough SIB interim results tell us?

Emma Tomkinson Evidence-based policy, Peterborough SIB, Social Impact Bonds, Social Impact Measurement , ,

Update: actual results are now out.

First cohort results from the Peterborough SIB were released August 7 2014. The Social Finance UK press release on the results has lots of great information, with a quote below.

“Results for the first group (cohort) of 1000 prisoners on the Peterborough Social Bond (SIB) were announced today, demonstrating an 8.4% reduction in reconviction events relative to the comparable national baseline. The project is on course to receive outcome payments in 2016. Based on the trend in performance demonstrated in the first cohort, investors can look forward to a positive return, including the return of capital, on the funds they have invested.”

Outdated information below:

Peterborough Interim ResultsOn the 13th of June, the Ministry of Justice released interim results from the Peterborough Pilot SIB. The results were seen as very encouraging, although Social Finance stressed that the results “do not measure  reoffending behaviour over as long a period as the Social Impact Bond will be judged and are not compiled on precisely the same basis as will be used by the Independent Assessor during the course of 2014 to determine whether a payment is due.”

What the results do tell us

The results tell us that the reoffending events have improved in the Peterborough cohort as the national average has worsened. We can be fairly confident that the reduction in reoffending is due to the Peterborough SIB, or more specifically the One Service. This in itself is quite an achievement for Government policy.

These results are also an excellent demonstration of the need to have a contemporary comparison, in this case the Police National Computer control group, rather than a historical baseline. If the historical re-conviction rates at Peterborough had been used as the only comparison, it would appear that a 6% reduction had been produced. Using the national comparison group shows that the programme also counters an increasing trend in re-convictions to produce a relative reduction of 23%. The inability of historical baselines to represent the fluctuating environment affecting reoffending is further illustrated when we look at results leading up to the 2010 launch of the One Service. The graph below shows that reoffending by both Peterborough inmates and prisoners across the nation was increasing until SIB services began in 2010, but then reversed in comparison to the national rates.

Peterborough Interim Results 2

What the results don’t tell us

The results do not tease out for us which aspects of the One Service might be more or less responsible for success. So, the results do not tell us if the reduction in reoffending is due to the:

  • use of volunteer mentors
  • voluntary, rather than mandatory, participation
  • long-term nature of the SIB funding
  • flexibility of funding
  • ability to innovate programme delivery to optimise outcomes
  • focus on a single outcome
  • extraordinary skills of the people involved in managing and delivering services
  • continuous evaluation and improvement of the One Service
  • discipline of reporting to external investors
  • alignment of financial and social returns.

The April 2014 evaluation commissioned by the Ministry of Justice from Rand Europe sheds some light on the perceived benefits of the SIB model, including the way the flexibility of funding allows for the service to improve in response to performance management data.

Update April 2014

See 24 April results and press release from the Ministry of Justice stating “Before the pilot, for every 100 prisoners released from Peterborough there were 159 reconviction events annually. Under the scheme this figure has fallen to 141 — a fall of 11 per cent. Nationally that figure has risen by 10 per cent over the same period.”

Toby Eccles blog analyses how well the Peterborough SIB achieved its objectives to:

  • Enable innovation
  • Enable flexibility and focus on outcomes
  • Bring rigour to prevention
  • Better alignment
  • Investment in social change.

And a good analysis of the Peterborough journey and what was learnt is given by the Social Spider here. I have a slightly broader view of SIBs in the context of policy reform, but I like the discussion.

References

Ministry of Justice, Statistical Notice: Interim re-conviction figures for the Peterborough and Doncaster Payment by Results pilots. 12 June 2013.

Social Finance, Interim Re-Conviction Figures for Peterborough Social Impact Bond Pilot, press release 13 June 2013.

Ministry of Justice, Mentoring Scheme Reduces Reoffending, press release 13 June 2013.

Vibeka Mair, Peterborough social impact bond has slashed reoffending rates says MoJ, Civil Society Finance, 13 June 2013.

Alan Travis, Pilot schemes to cut reoffending show mixed results, The Guardian, 13 June 2013.

BBC, Prison payment-by-results schemes see reoffending cut, 13 June 2013.

Nicholls, A. & Tomkinson,E. Case Study – The Peterborough Pilot Social Impact Bond – Oct 2013, Oct 2013.

Ministry of Justice (UK) innovation pilot invites the market to design new financing mechanisms

Emma Tomkinson Social Impact Bonds, Social Innovation, Social Investment & Finance , , , , ,

This pilot was put on hold (indefinitely) by the new Justice Secretary as he introduced his widespread probation payment-by-results program

In August 2011, Minister for Prisons and Probations, Crispin Blunt MP introduced the innovation pilot programme with

It has, for too long in my view, been held that Whitehall knows best.  I can only imagine how dispiriting it must be for well-intentioned organisations to try to influence the government and come up against what must sometimes seem like an impenetrable barrier…  The government has listened and is not shying away from creating a market for public service delivery that is open to this innovation. 

The briefing presentation that went with this includes:

  • a request for proposals that are of sufficient size for payment and evaluation, focus on outcome measures, create incentives to intervene with the entire cohort and use measurement as a trigger for payments
  • an intention for at least one pilot to result
  • an allocation of £20m for success payments
  • outlines of Peterborough SIB and Doncaster Prison payment by results contracts

Following this, in December 2011, the Ministry of Justice (UK) released an open tender asking organisations to propose both innovative funding and service delivery models to tackle reoffending called Payment by results – innovation pilots. These pilots are on top of six other payments-by-results pilots they are running.

What’s interesting about this?

  1. This is the first tender I’ve seen where a government is open to market suggestions for financing mechanism to reward outcomes. We might see new payment by results mechanisms.
  2. Prior to this, the only government organisation to run a competitive process for a Social Impact Bond had been the NSW Government with their Social Benefit Bonds Trial. This tender goes beyond that and is open to any payment by results financing model.
  3. Innovation = new = increased perceived risk. The Ministry of Justice has a decade-long history of tying payment rewards and penalties to their prison management contracts and was the first government organisation to implement a Social Impact Bond. It also broke ground with its Doncaster Prison payment-by-results contract, where 10% of contract payments are tied to a single reoffending indicator. It will be interesting to see how the risk of new financial mechanisms may be reduced, for example by cornerstone payments or existing programs with strong evidence behind them. Or the risk of new programs may be offset by a more established financial mechanism.
  4. Justice is leading the field in payment by results initiatives around the globe. Employment services contracts used to be the most innovative. Which service area will be next to emerge and how far can payment by results be feasibly extended?

The official tender information on Payment by results – Innovation pilots was released in December 2011. Prequalification questionnaires were submitted in January 2012 and organisations then engaged in developing proposals.

Both lots contain the text:

We seek to develop the market for payment by results through innovative forms of finance and strong involvement from voluntary and social enterprise organisations, including smaller organisations and; as part of the payment by results programme of pilots learn what works and develop a broader evidence base to support design of future policy.
Contract duration is dependent on proposal details. We expect pilots to run for no less than 2 years and no more than 6 years. Up to 20 000 000 GBP has been identified for rewarding successful outcomes of innovation pilots. This funding for outcome payments is for the whole innovation pilots programme. The potential for a small amount of start up funding or working capital is also available. MoJ will consider providing working capital, the precise amount to be negotiated as part of the contracting process.

Lot 1 is looking for programs for short term prisoners that are an

innovative approach to tackling re-offending of offenders who have served a sentence imposed by a Court and have been released from sentences of less than 12 months.

And Lot 2 asks proponents to define their cohort for an

innovative approach to tackling re-offending of offenders; we will accept proposals that focus on offender cohort and any size cohort.

The tender will be awarded to

the most economically advantageous tender in terms of the criteria stated below:

1. Operational service delivery. Weighting 20

2. Commercial financial. Weighting 203. Partnership & stakeholders. Weighting 15

4. Commercial legal. Weighting 10

5. Supply chain management. Weighting 10

6. Implementation. Weighting 107. Human resources & organisational change. Weighting 5

8. ICT & information security. Weighting 5

9. Continuous improvement. Weighting 5