What if government doesn’t refer enough people into your social impact bond (SIB)?

One of the problems faced when developing a SIB is how people end up as part of the intervention cohort. If the process involves government or some other body referring participants to the service delivery organisation, how does this service delivery organisation manage the risk that not enough people will be referred, or that the wrong kind of people (i.e. those with little potential for change) will be referred?

Referral mechanisms that are used can be split into the following three categories:

  1. Eligible: eligibility criteria are defined and everyone who meets it is considered part of the SIB
  2. Self-selected: either the delivery organisation chooses participants or people choose to join
  3. Referred: government refers individuals into the program

All of these mechanisms involve some eligibility criteria that participants must meet to be included.

Examples of each mechanism are listed below.

Eligibility – everybody eligible is considered part of the intervention cohort and measured

Peterborough: All male prisoners exiting HMP Peterborough after serving less than 12 months sentence.

Other examples: New York City, New York State

One of the key benefits to government of this approach, is that the responsibility and incentive for convincing people to participate in the program lies with the service provider. It is also more suitable for a rigorous measurement approach where the intervention cohort is compared to another cohort, as the potential for bias in the selection process is reduced.

Self-selection – either the service delivery organisation chooses people to be part of their program or participants choose to join, usually with some eligibility criteria that must be met

DWP Innovation Fund: Service delivery organisations were asked to propose which young people they would work with and how they would attract them. “Your proposal must clearly demonstrate how you are identifying and working with the most disadvantaged and socially excluded young people, the vast majority of whom would otherwise not achieve educational and employment outcomes” (DWP, Round Two Specifications). Some programs asked schools or other organisations to refer students to them.

One of the key benefits of this approach is that the service delivery organisation is in control of how many participants are included in the program, so if they need more people they can do something about it themselves. They can also make sure that the participants have the type of needs that their program was designed for.

Referred – Government refers people it consider suitable for the program, usually using some eligibility or referral criteria

Australia The Benevolent Society: Referrals are made by the Department of Family and Community Services according to the processes and criteria set out in an operations manual.

Other examples: Essex County Council, Australia Newpin

This referral system is sometimes preferred by government, as it gives them control over who participates in the program, but it means that providers are exposed to the risk that not enough people are referred.

But how do we make sure enough participants are referred to a SIB?

Let’s look at two SIBs where this issue has been responded to.

Educating/marketing to government referrers: Essex County Council

The Essex County Council SIB relied on referrals from the council, however they weren’t getting enough of the right type. In order to fix this, the delivery organisation went in to educate and encourage council staff about the program and who they should refer to it.

“We discovered that Essex weren’t referring enough children who would most benefit from the intervention, due to a combination of things, including competing priorities of senior staff and referral staff not knowing the program existed, or who and how to refer. Solving this problem would not usually fall in the remit of a service provider, but our performance managers went into the council to do a marketing push and went right up to a senior level to change the way they were referring.  The board is also considering whether to add an additional ‘marketing’ function to the service, to ensure that the barriers to referral are continually being addressed proactively” (Andrew Levitt, of Bridges Ventures, deliveringthepromise.org).

Incentives for government to refer (or penalties if they don’t): The Benevolent Society

The Australian Social Benefit Bond delivered by the Benevolent Society has a second part to its payment formula that corrects for a lack of referrals. The first part is related to the improvement that the program makes against its three outcome metrics: Out-of-home care entries; Safety and Risk Assessments; Helpline Reports. The second part is where the improvement is adjusted according to whether adequate referrals have been made and whether the children in the program can be compared against children not in the program. It’s called the ‘Performance percentage’.

While payments will not be made until 2018, we can use the 2014 preliminary results to understand how the metric works.

IMPROVEMENT PERCENTAGE
Measure Improvement Weighting
Out-of-home care entries 17% 66%
Safety and Risk Assessments (70%) 17%
Helpline Reports (3%) 17%
Improvement Percentage (1%)

In 2014, the program made a minus one percent improvement against its outcomes.

The second part of the payment metric combines the improvement percentage with two metrics that address the risk related to (1) the measurement and (2) the referral methods.

  1. The counterfactual for this program is a propensity-score-matched control group, so to manage the risk of children not being able to be matched, a fixed improvement of 15% is assigned to each unmatched child.
  2. The SIB has set a number of children the Department of Family and Children’s Services guarantees to refer. For each child under the guaranteed number, an improvement of 40% is assigned. You can see below that in the first year of the program, referrals from government were 21% below where they should be, which combined with the score for unmatched children, lifted the performance percentage from -1% to 8%. This is a huge incentive for the Department to make sure they are meeting their referral guarantee.
PERFORMANCE PERCENTAGE
Measure Performance Weighting
Improvement Percentage (1%) 77%
Unmatched Children Percentage 15% (fixed) 2%
Guaranteed Referrals Shortfall Percentage 40% (fixed) 21%
Performance Percentage 8%

So the 2014 results carried a huge penalty for government falling short in referrals.

Interestingly enough, the 2015 results show the Guaranteed Referrals still falling short by 13%. This could be due to over-calculation of eligible families during the contract development, or it could indicate a continuing lack of referrals by government staff to the program. It will be interesting to watch how this mechanism works as we head towards payment in 2018.

Changing a Social Impact Bond (SIB) metric mid-flight

Social impact bonds are new, so involve a lot of learning on the job. This learning is less that one SIB ‘works’ and another SIB does not, but more about the iterative adjustments that allow for more effective services, more flexible procurement processes, more alignment of incentives in contracts. One aspect of SIBs that is new for many jurisdictions is long-term contracts. Long-term contracts have many benefits to those delivering and receiving services, but in order to respond to information that comes to light over this time, they must also allow for adjustment and termination.

As its second year draws to a close, investors in Australia’s first social impact bond, the Newpin Social Benefit Bond, have been asked to approve an amendment to its payment metrics, so that they more faithfully reflect success for the children and families it serves.

The reason for this is that the metric rewards investors when children in foster care are restored to their families by the court (“restorations”). When developing the metrics, the breakdown of restorations where children would return to foster care (“reversals”) was discussed, but failed to be written into the contracts. The intention of the metrics is to reward social outcomes with financial return, which means that while restorations should result in payments, reversals should not.

The Newpin Social Benefit Bond has two different payment metrics. One that determines payment from government to the delivery charity, UnitingCare, and a different one metric to determine payment from UnitingCare to Investors.

Newpin SBB payment metrics

The problem with each metric and the amendments proposed to rectify these problems are shown below. For original contracts and a more detailed summary of metrics see the bottom of this page.

What is the problem?

  • Metric 1 NSW Government => UnitingCare: a drafting error was made that did not deduct reversals in the current measurement period, only previous measurement periods. So the government does not make payments for reversals that occurred in the previous financial years, but does make payments for reversals in the most recent financial year.
  • Metric 2 UnitingCare => investors: UnitingCare makes payments based on the cumulative rate of restoration of children to their families. This includes restorations that have been reversed and thus does not faithfully reflect success for families.

In the second year of the SIB, several reversals occurred. So all parties wanted to ensure that these were not paid for in the same way as successful restorations. Government and UnitingCare agreed to amend their metric (Metric 1). An amendment to the investor metric (Metric 2) was also sought to limit payments for unsuccessful restorations. In order to change the metrics, all 60-odd investors had to agree to it. They were also given the option of selling their investment. Using the results to the end of May 2015, the amended metric would return 7.5% interest to investors for the year, while the original contract would return 13.5% interest for the year. If investors did not agree to the amendment UnitingCare would have to pay the inflated interest and continue the service with an imbalance between payments from government and the interest paid to investors. This may have led to UnitingCare exercising their termination rights at the end of year 3. A 10% cap on reversals was proposed in order to limit the ongoing risk to investors and increase the likelihood that they would agree to the amendment.

The proposed amendment

  • Metric 1 NSW Government => UnitingCare: change to not paying for reversals that occur within 12 months of a restoration.
  • Metric 2 UnitingCare => investors: change to not paying for reversals that occur within 12 months of a restoration, but only up to a cap of 10% of restorations. If more than 10% of restorations are reversed, then reversals over this cap are treated as successful restorations for the purpose of calculating the interest paid by UnitingCare to investors.

The amendment still leaves us with an imperfect metric.

  1. If the proportion of reversals is above 10%, UnitingCare will make interest payments to investors (based on the cumulative restoration rate) which will include restorations that have been reversed, even though this does not reflect success for the families involved.
  2. If the proportion of reversals is above 10%, UnitingCare will make a success payment to investors that includes reversals over the cap, but for these reversals, UnitingCare will not receive government outcome payments.
  3. There were no reversals in the first year of the SBB and 28 children were restored from the mothers’ centres. In the second year to May 2015 (not a full year), 20 children were restored from the mothers’ centres, and 7 of these restorations were reversed, some being reversals of restorations from the previous year. The reversals at the mothers’ centres represent 15% of cumulative restorations. Therefore reversals for the second year are above 10% of restorations. If the amendment is agreed and applied retrospectively to year two, UnitingCare will pay investors for restorations that were not maintained, and for which they themselves receive no payments.

What can we learn?

There are several key lessons I draw from this experience. Note that every other stakeholder may have a completely different list!

  1. It’s important to be able to learn as you go and respond to new information, allowing for amendments, dispute and termination on fair terms.
  2. Having different metrics determining payments to the delivery agency and payments from the delivery agency means that there is some misalignment of incentives.
  3. The Newpin SBB has a mix of ‘impact-first’ and ‘finance-first’ investors. The 10% cap was a way of striking a balance between them. While the fiduciary duties of those investing through structures such as self-managed super funds and Private Ancillary Funds do not conflict with them making social/impact investments, some perceived agreeing to a lower rate of return as conflicting with their fiduciary duties as trustees.
  4. When contracting for outcomes, enormous attention has to be paid to thinking through all potential scenarios, however unlikely, to ensure the intended social outcomes are reflected in the legal terms.
  5. It is very difficult to reflect the journey of someone through social service systems with a binary measure. The definitions and metrics deem the program as either successful or unsuccessful for children and their families, with no ability to accommodate degrees of success or episodes of care over time.

Update on results to July 2015 (2.25 years of service delivery and second payment to investors)

The amendment was passed by all investors. Without the amendment, the Restoration Rate would have been calculated at 68% and the Interest Rate at 15.08%. With the amendment, the Restoration Rate was calculated at 62% and the Interest Rate paid to investors was 8.92%. If there had been no cap, and all reversals were considered unsuccessful outcomes, the Restoration Rate would have been calculated at 58% and the Interest Rate would have been 5.6%. So the investors did agree to forgo much of the interest that was due to them under the original agreement, but gained over 3% more than if all reversals were treated as unsuccessful outcomes. The difference in the investor interest was paid by the charity UnitingCare. The amount they were paid by NSW Government paid was not affected by this.

Newpin SBB metric - year 2

References

Metrics summary

  • Metric 1 NSW Government => UnitingCare: for Cohort 1 Outcome Payment = (the total number of restorations for all Mothers’ Centres and Fathers’ Centres – the counterfactual restorations) x the amount in payments look-up table. The counterfactual restorations are set at 25% for the first three years and then by a live control group. There are also payments that do not depend on outcomes and outcome payments for other cohorts.
  • Metric 2 UnitingCare => investors: Interest Rate = 3% + [0.9 x number of restorations for all Mothers’ Centres/(number of referrals to Mothers’ Centres– 55%)] subject to:
    • if the Restoration Rate is below 55%, the Interest Rate is nil; except
    • a minimum of 5% is applied over the first three years; and
    • a maximum of 15%.

*Note that investor payments relate only to Mothers’ Centres as they were considered lower risk at the time the metric was developed. The discussion above focuses on Mothers’ Centres only.

Disclaimer: Emma is a retail investor in the Newpin Social Benefit Bond. She bought her parcel from a wholesale investor when the restrictions around types of investors expired. She firmly believes that Newpin does wonderful and important work with families.  

Developing a counterfactual for a social impact bond (SIB)

The following was taken from a presentation by Sally Cowling, Director of Research, Innovation and Advocacy for UnitingCare Children, Young People and Families. The presentation was to the Social Impact Measurement Network of Australia (SIMNA) New South Wales chapter on March 11 2015. Sally was discussing the measurement aspects of the Newpin Social Benefit Bond, which is referred to as a social impact bond in this article for an international audience.

The social impact bond (called Social Benefit Bond in New South WaleSally Cowlings) was something very new for us. The Newpin (New Parent and Infant Network) program had been running for a decade supported by our own investment funding, and our staff were deeply committed to it. When our late CEO, Jane Woodruff, appointed me to our SIB team she said my role was to ’make sure this fancy financial thing doesn’t bugger Newpin up’.

One of the important steps in developing a social impact bond is to develop a counterfactual. This estimates what would have happened to the families and children involved in Newpin without the program, the ‘business as usual’ scenario. This was the hardest part of the SIB. The Newpin program works with families to become strong enough for their children to be restored to them from care. But the administrative data didn’t enable us to compare groups of potential Newpin families based on risk profiles to determine a probability of restoration to their families for children in care. We needed to do this to estimate the difference the program could make for families, and to assess the extent to which Newpin would realise government savings.

Experimenting with randomised control trials

NSW Family and Community Services (FACS) were keen to randomly allocate families to Newpin as an efficient means to compare family restoration and preservation outcomes for those who were in our program and those who weren’t. A randomised control trial is generally considered the ‘gold standard’ in the measurement of effect, so that’s where we started.

Child's drawing of a happy kidOne of my key lessons from my Newpin practice colleagues was the importance of their relationships and conversations with government child protection (FACS) staff when determining which families were ready for Newpin and had a genuine probability (much lower than 100%) of restoration. When random allocations were first flagged I thought ‘this will bugger stuff up’.

To the credit of FACS they were willing to run an experiment involving local Newpin Coordinators and their colleagues in child protection services. We created some basic Newpin eligibility criteria and FACS ran a list from their administrative data and randomly selected 40 families (all of whom were de-identified) for both sets of practitioners to consider. A key part of the experiment was for the FACS officer with access to the richer data in case files to add notes. Through these notes and conversations it was quickly clear that a lot of mothers and fathers on the list weren’t ready for Newpin because:

  • One was living in south America
  • A couple had moved interstate
  • One was in prison
  • One had subsequent children who had been placed into care
  • One was co-resident with a violent and abusive partner – a circumstance that needed to be addressed before they could commence Newpin

From memory, somewhere between 15 and 20 percent of our automated would-be-referrals would have been a good fit for the program. It was enlightening to be one of the non-practitioners in the room listening to specialists exchange informed, thoughtful views about who Newpin could have a serious chance at working for. This experiment was a ‘light bulb moment’ for all of us. For both the government and our SIB teams, randomisation was off the table. Not only was the data not fit for that purpose, we all recognised the importance of maintaining professional relationships.

In hindsight, I think the ‘experiment’ was also important to building the trust of our Newpin staff in our negotiating team. They saw an economist and accountant listening to their views and engaging in a process of testing. They saw that we weren’t prepared to trade off the fidelity and integrity of the NewpiChild's drawing of happinessn program to ‘get’ a SIB and that we were thinking ethically through all aspects of the program. We were a team and all members knew where they did and didn’t have expertise.

Ultimately Newpin is about relationships. Not just the relationships between our staff and the families they work with, but the relationship between our staff and government child protection workers.

But we still had the ‘counterfactual problem’! The joint development phase of the SIB – in which we had access to unpublished and de-identified government data under strict confidentiality provisions – convinced me that we didn’t have the administrative data needed to come up with what I had come to call the ‘frigging counterfactual’ (in my head the adjective was a bit sharper!). FACS suggested I come up with a way to ‘solve’ the problem and they would do their best to get me good proxy data. As the deadline was closing in, I remember a teary, pathetic midnight moment willing that US-style admin data had found a home in Australia.

Using historical data from case files

Eventually you have to stop moping and start working. I decided to go back to the last three years of case files for the Newpin program. Foster care research is clear that the best predictor of whether a child in the care system would be restored to their family was duration in care. We profiled all the children we had worked with, their duration in care prior to entry to Newpin and intervention length. FACS provided restoration and reversal rates in a matrix structure and matching allowed us to estimate that if we worked with the same group of families (that is, the same duration of care profiles) under the SIB that we had in the previous 3 years, then the counterfactual (the percentage of children who would be restored without a Newpin intervention) would be 25%.

As we negotiated the Newpin Social Benefit Bond contract with the NSW Government we did need to acknowledge that a SIB issue had never been put to the Australian investment market and we needed to provide some protection for investors. We negotiated a fixed counterfactual of 25% for the first three years of the SIB. That means that the Newpin social impact bond is valued and paid on the restoration rate we can achieve over 25%. Thus far, our guesses have been remarkably accurate. To the government’s immense credit, they are building a live control group that will act as the counterfactual after the first three years. This is very resource intensive but the government was determined to make the pilot process as robust as possible

In terms of practice culture, I can’t emphasise enough the importance of thinking ethically. We had to keep asking ourselves, ‘Does this financial structure create perverse incentives for our practice?’ The matched control group and tightly defined eligibility criteria remove incentives for ‘cherry picking’ (choosing easier cases). The restoration decisions that are central to the effectiveness of the program are made independently by the NSW Children’s Court and we need to be confident that children can remain safely at home. If a restoration breaks down within 12 months our performance payment for that result is returned to the government. For all of us involved in the Newpin Social Benefit Bond project behaving thoughtfully, ethically and protecting the integrity of the Newpin program has been our raison d’etre. That under the bond, the program is achieving better results for a much higher risk of group of families and spawning practice innovation is a source of joy which is true to our social justice ethos.

Breakdown of social impact bond investors in Australia

There are two social impact bonds in Australia that have raised investment and are delivering services. They have been called “Social Benefit Bonds (SBBs)” by the New South Wales Government, the state government that initiated them. They are the Newpin SBB and the Benevolent Society SBB and both of them work to strengthen families to ultimately reduce the need for children to be placed in care outside their homes. Investment for both deals was open to wholesale investors upon application. One parcel of investment in the Newpin SBB was transferred to a retail investor in August 2014, so is not shown on the chart. Also interesting to note is that the Benevolent Society, the charity delivering services for one SBB, has invested in the Newpin SBB, as well as in their own. A breakdown of investor types for each SBB is below.

Newpin investor breakdown(Source: Presentation by Ian Learmonth, 2013, p.14)

Bensoc investor breakdown(Source: KPMG Evaluation of the Joint Development Phase of the NSW Social Benefit Bonds Trial, 2014, p.30)

A second social impact bond for NSW Australia

Benevolent Society Social Benefit Bond(Source: The Benevolent Society. “First Charity” refers to the Benevolent Society being the first charity established in Australia.)

On June 14 2013 it was announced that the terms of a second social benefit bond have been agreed in New South Wales, Australia. It will provide services to strengthen 400 families and reduce the need for out-of-home care over five years, beginning October 2013. Press releases were issued by the charity service provider, The Benevolent Society and one of their partners in the SIB, Westpac Institutional Bank. The other partner was a second bank, the Commonwealth Bank of Australia and the involvement in this deal of two of the four large Australian banks has captured media attention. The banks were involved in the construction of the bond as pro bono advisors and are also investors. The Benevolent Society is also investing in the Social Benefit Bond, a move promote confidence in their ability to deliver outcomes. Other investors include institutional investors NRMA Motoring & Services and Australian Ethical Investments.

Investors will provide working capital for the services up-front and the NSW Government will cover all repayments once outcomes have been produced and measured. The social benefit bond also involves financial services firm Perpetual as trustee, a move expected to give investors further confidence in the deal. The role of the intermediary as played by Social Finance in Peterborough and MDRC in New York is not played by any organisation in this arrangement in a comparable way. 

Benevolent Society SBB tiers

The offering closed on Friday October 4th having raised $10m. 40 investors make up the low-risk tier and 17 investors make up the high-risk tier. In contrast to expectations that social impact bond risk needs to be reduced to attract investors, the high-risk tier proved more attractive and sold out much faster.

Once again, we see the NSW Government offering a guarantee, which means that the Government will pay even if the service does not produce the desired outcomes for recipients. In the previous NSW social benefit bond this guarantee was for between 50% and 75% of investor capital. This time it’s 100% of investor capital guaranteed, but only for the low-risk tier of investments, worth $7.5m or 75% of total investment. This may be an investment by the Government in establishing a market and track record for social impact bonds, rather than a model we can expect to see replicated in future years. The $7.5m of the bond with protected capital will also accrue variable returns up to 10%, dependent on outcomes. An additional $2.5m high-risk tier of investment will involve capital at risk, but will offer outcome-dependent returns up to 30%.

Media Coverage

Date Author Publisher Title
14/06/2013 Benevolent Society First charity issues Social Benefit Bond with Westpac and CBA
14/06/2013 Bela Moore Super Review Westpac Institutional and Commonwealth team up to launch social benefit bond
14/06/2013 James Fernyhough Financial Standard Big banks move into SBB space
14/06/2013 Australian Financial Review NSW strikes deal for $10m social bond
14/06/2013 Westpac First of its kind Social Benefit Bond supports efforts to keep families together – paves the way for socially responsible investors
15/06/2013 Clancy Yeates The Age / Brisbane Times Lenders back bond to keep children out of foster care
18/06/2013 Pro Bono Charity Issues Bank-Backed Social Benefit Bond
19/06/2013 Rick Morton The Australian Banks pitch in for $10m bond
4/10/2013 Rachel Alembakis The Sustainability Report Westpac, CBA raise $10 million for Benevolent Society SBB
5/10/2013 Sally Rose The Australian Financial Review NRMA and Australian Ethical buy Benevolent Society bond

This is the second of three social benefit bonds that are the product of a request for proposal issued by the NSW Government in September 2011 and awarded March 2012. The first was the Newpin social benefit bond announced in March and the final bond to reduce adult reoffending is still under development by the Government and Mission Australia, a charity service provider.

(Updated 8 October 2013.)

NSW Newpin social benefit bond – returns to investors

Image

(Social Ventures Australia: Newpin Social Benefit Bond)

The NSW social benefit bond (social impact bond) for UnitingCare Burnside’s Newpin programme is attracting interest from a range of investors, including NGS Super – the first time we’ve seen a pension/superannuation fund sign up to a social impact bond. The Newpin programme is for families with children aged 0-5 and results are measured on the proportion of children in out-of-home care (which includes foster care, institutional care and placement with extended family) that are returned to their families by the courts. This is called the restoration rate.

Eureka Report’s A high-yield bond with social benefits recently revealed the social and financial returns over the seven year social benefit bond as follows:

Restoration rate (r) Return to investor (IRR)
≥ 70% 15%
65% ≤ r < 70% 12%
60% ≤ r < 65% 7.5%
55% ≤ r < 60% 3%
<55%
•minimum 5% yield over first three years
•no minimum yield after three years
•75% of capital returned if bond redeemed at four years
•50% capital returned if redeemed after four years

The Newpin restoration rates were 74.5% last year. Their approach to this social benefit bond, to measure their results for a similar cohort for the year prior to the bond, gives social investors information with which to judge the social and financial risk of the investors. Providing this information attracts investors beyond the die-hard philanthropists who have backed the programme from the start.

Funds were raised from 59 investors by Social Ventures Australia (SVA) with a minimum investment of AU$50,000. The SIB was oversubscribed. Below is a breakdown of investment by investor categories as presented by Ian Learmonth, Executive Director, Social Ventures Australia at the 2013 Social Finance Forum in Sydney/Australia.

NewPin SIB investors