What do we know about the Utah SIB results (without a counterfactual)?

November 9, 2015November 9, 2015 Emma Tomkinson Social Impact Bonds, Utah SIB SIB, SIBs, social impact bond, Utah

The Utah SIB recently paid a return for Goldman Sachs, and press releases from both Goldman Sachs and United Way of Salt Lake deemed the program a success. But this was met with some criticism, most notably by the New York Times in Nathaniel Popper’s article Success Metrics Questioned in School Program Funded by Goldman. Now I would argue that success for each stakeholder is achieving whatever they wanted to achieve. So as far as I’m concerned, claiming success simply means that things happened as you wanted. But we might also assume that a government’s objectives are what it’s prepared to pay for via the SIB payment metric.

So how does the payment metric for the Utah SIB work?

For the first year results, Goldman Sachs was paid 95% of the savings to the state. Savings to the state are calculated from the number of children identified as ‘likely to use special education in grade school’^[i] (110 in year 1) minus the number of children who used special education (1 in kindergarten) multiplied by the cost of a special education add-on for one year ($2607).

Is that a success?

Well, the program is doing very well at delivering on its payment metric. Of the 110 children identified as likely to use special education, only one of them is using special education in kindergarten. If this is the definition of success, then the program is definitely a success!

(United Way (2015) SIB fact sheet)

So what’s the problem?

Many people who aren’t involved in the SIB would define success a little differently to the payment metric. They would define the success of the program by the reduction in how many children would require special education support. What we don’t know is how many of the 110 children would have needed special education without the program. I teach my probability classes that ‘likely’ means 50%-80%. But the payment metric seems to assume that 100% of the children would have needed special education without the program, according to the savings-to-government calculation. In order to know how much the program improved things for the children involved, we need a comparison group or ‘counterfactual’, an estimate of how many of the children would have ended up using special education. A counterfactual means you can claim you caused the results, the absence of a counterfactual means you can only say you contributed to them.

What’s a counterfactual?

A counterfactual or comparison group can be constructed in several ways. “A good comparison group is as similar as possible to the group of service users who are receiving an intervention, thus allowing you to be confident that the difference in outcomes between the groups is only caused by the intervention.”^[ii] Some of the more commonly used counterfactuals in SIBs are shown below.

If you would like to know more, I recommend this Guide to Using Comparison Group Approaches from NPC and Clinks in the UK. And for guidance on randomised control trials in public policy setting you can’t go past the UK Cabinet Office’s Test, Learn, Adapt.

The Utah SIB involved no comparison group – certainly the payment metric didn’t.

So without a counterfactual, what can we say about this SIB?

“Of the 110 four-year-olds had been previously identified as likely to use special education in grade school…only one went on to use special education services in kindergarten.”^[iii]
“These results triggered the first payment to investors for any SIB in the United States.”^[iv]
“As a result of entering kindergarten better prepared, fewer children are expected to use special education and remedial services in kindergarten through 12th grade, which results in cost savings for school districts, the state of Utah and other government entities.”^[v] [note this says ‘fewer children are expected to use’, not ‘fewer children use’]
“109 of 110 At-Risk Utah Students Avoid Special Education Services Following High-quality Preschool”^[vi] [this would be untrue if the word ‘following’ was changed to ‘due to’ or ‘because of’]
“Utah’s [curriculum and testing] methodology was vetted both locally and nationally by early education and special education experts and leaders”^[vii]
“They lacked certain basic data on what would have been expected to have happened to the students without the Goldman-funded preschool”^[viii]
“My kids have really grown. I don’t think [my kids] would be where they are if it wasn’t for the preschool. That basic step is what prepares you to succeed in school, and later, in life.”^[ix]

What can’t we say?

“School districts and government entities saved $281,550 in a single year, based on a state resource special education add-on of $2,607 per child.”^[x][we have no idea what they would have spent on this group otherwise]
“High-quality preschool changes the odds”^[xi][we simply don’t know what the odds would have been without the preschool program, so we can’t say that they’ve changed]
“Fewer children used special education services and remedial services by attending the SIB-financed Preschool Program, saving money for school districts and government entities”^[xii]

What other SIBs don’t have a counterfactual?

UK: ten DWP Innovation Fund programs (seven of which were SIBs) [the Impetus-PEF ThinkForward SIB press release shows similar difficulty to the Utah SIB in understanding the difference made to young people. While 90% of young people engaged in further education, employment or training seems a wonderful result, there is no estimate of what might have happened otherwise.]
UK: seven Fair Chance Fund SIBs
UK: four Youth Engagement Fund SIBs
UK: Manchester Children in Care
UK: It’s All About Me – Adoption SIB
Canada: Saskatchewan single mothers’ SIB
Australia: Newpin SIB (for the first three years while a control group is established)

Note that most government spending on social services is not compared to a counterfactual. Some people argue that the perceived requirement for a SIB counterfactual creates an unnecessary additional barrier to SIB development, but others argue that it’s the best thing about SIBs – for the first time we are having mainstream discussions about standards of measurement and evidence in social services.

If you know of any government-funded social programs other than SIBs that do have a counterfactual, please post a link to them in the comment box below.

Why doesn’t every SIB have a counterfactual?

In order to estimate the effect of an intervention with any confidence, you need a large sample size. This is called ‘statistical power’ – I’ve tried to explain it in SIB Knowledge Box: Statistical Power. If a program is working intensively with just a few people, as is the case in Saskatchewan (22 children in SIB), then a reliable comparison to a counterfactual is not possible.
It is more work to set up a counterfactual – a similar comparison group must be established and this can take varying degrees of effort. It also takes skill that is in short supply. Biostatisticians are one of the the best resources for this kind of work. Most government stats units do not have experience in this kind of work.
Without a counterfactual, results can be counted as they are achieved, rather than waiting for a statistical comparison for the group, so investors can get paid earlier and more frequently and managers can ‘track’ performance.

As always, if there’s anything in this article that needs correcting or information that should be included, please either comment below or use the contact page to send me an email.

[i] United Way (2015) SIB fact sheet

[ii] NPC & Clinks (2014) Using comparison group approaches to understand impact

[iii] Edmondson, Crim, & Grossman (2015) Pay-For-Success is Working in Utah, Stanford Social Innovation Review

[iv] Edmondson, Crim, & Grossman (2015) Pay-For-Success is Working in Utah, Stanford Social Innovation Review

[v] United Way of Salt Lake 2015, Social Impact Bond for Early Childhood Education Shows Success

[vi] United Way of Salt Lake 2015, Social Impact Bond for Early Childhood Education Shows Success

[vii] Bill Crim, 2015, When Solid Data Leads to Action – Kids’ Lives Improve

[viii] Nathaniel Popper, 2015, Success Metrics Questioned in School Program Funded by Goldman

[ix] United Way (2015) SIB fact sheet

[x] Edmondson, Crim, & Grossman (2015) Pay-For-Success is Working in Utah, Stanford Social Innovation Review

[xi] United Way (2015) SIB fact sheet

[xii] United Way (2015) SIB fact sheet

Delivering the Promise of Social Outcomes: The Role of the Performance Analyst

February 10, 2015February 10, 2015 Emma Tomkinson Social Impact Bonds, Social Impact Measurement, Social Innovation collective impact, Delivering the Promise, Impetus-PEF, Laboratorio de Investimento Social, performance analyst, performance analysts, performance management, SIB, SIBs, social finance, social impact bond, social impact bonds, Social Investment Lab, social outcomes, Think Impact

I’ve wanted to write about performance management systems for a long time. I knew there were people drawing insights from data to improve social programs and I wanted to know more about them. I wanted to know all about their work and highlight the importance of these quiet, back-office champions. But they weren’t easy to find, or find time with.

I worked at Social Finance in London for three months in late 2013, a fair chunk of that time spent skulking around behind Dan Miodovnik’s desk. I’d peer over his shoulders at his computer screen as he worked flat out, trying to catch a glimpse of these magic performance management ‘systems’ he’d developed. At the end of my time at Social Finance, I understood how valuable the performance management role was to their social impact bonds (SIBs), but I still had no idea of what it actually entailed.

Antonio Miguel, The Social Investment Lab

Then early 2014 Antonio Miguel and I took a 2-hour bullet train ride through Japan while on a SIB speaking tour. On this train journey I asked Antonio to open his computer and show me the performance management systems he’d worked on with Social Finance. Two hours later, I understood the essential components of a performance management system, but I didn’t fully grasp the detail of how these components worked together.

So I proposed to Dan that we join Antonio on the beaches of Cascais in Portugal in August 2014. My cunning research plan was to catch them at their most relaxed and pick their brains over beach time and beers. Around this time I saw a blog written by Jenny North, from Impetus-PEF that mentioned performance management. A call with her confirmed that they were as enthused about performance management as I was. So I drafted a clean, six-step ‘how to’ guide for constructing a performance management system. I hoped that a quick edit from Dan and Antonio, a couple of quotes and I’d be done.

Interviewing Dan and Antonio blew me away. Only when I heard them talk freely about their work did I realise the magic wasn’t in their computer systems, it was in their attitudes. It was their attitude to forming relationships with everyone who needed to use their data. It was their attitude to their role – as the couriers, rather than the policemen, of data.

They told me that there were plenty of ‘how to’ guides for setting up systems like theirs, but that the difficult thing was getting people to read and implement them.

Isaac Castillo, DC Promise Neighbourhood Initiative

They suggested I throw out my draft and interview more people. People who were delivering services and their investors. I didn’t just need to understand the system itself, I needed to understand what it meant for the people who delivered and funded services. I gathered many of these people at San Francisco’s Social Capital Markets (SOCAP) conference and several more from recommendations. One of these recommendations was Isaac Castillo, who works with the DC Promise Neighbourhood Initiative’s collective impact project. He is now managing not only his team of performance analysts, but the service delivery team too. It’s revolutionary, but it makes complete sense.

Interviewing these people has been a most humbling experience. It has revealed to me the extent of their dedication, innovation and intelligence. It has also revealed to me how little I knew, and in turn, how little we, as a sector, know about these people and their work. I am honoured to share their stories with you – please read them at deliveringthepromise.org.

This research is published by The Social Investment Lab (Portugal), Impetus-PEF (UK) and Think Impact (Australia).

Malaysian Innovation: Building a Social Impact Bond (SIB) Pipeline

February 3, 2015February 3, 2015 Emma Tomkinson Social Impact Bonds Agensi Inovasi Malaysia, impact investing, Islamic finance, SIB, SIBs, social finance, social impact, social impact bond, social impact bonds, social innovation

Agensi Inovasi Malaysia, part of the Malaysian Government, has embarked on a journey towards Social Impact Bonds that reflects the Malaysian social and policy context. There are three innovative features of their program, ‘Social Service Delivery’, worth highlighting:

Explaining SIBs as a public-private partnership for social good
Creating a market of new interventions to contract via a SIB
Exploring Islamic finance as a source of SIB funding

Let’s explore each of these innovations in turn.

Explaining SIBs as a public-private partnership for social good

Social Impact Bonds were first implemented by an organisation called Social Finance in the UK in 2010. The idea has since generated interest all over the world. The concept can be overwhelming for stakeholders, who seek to understand how far away this model might be from their current reality. In Malaysia, Social Impact Bonds have been framed as the logical next step after the recent introduction of other long-term partnerships and privately financed initiatives (PFIs) towards new infrastructure such as buildings and roads. The 2010 New Economic Model for Malaysia from the National Economic Advisory Council called for ‘academia, business, the civil service, and civil society’ to ‘work together in partnership for the greater good of the nation as a whole’ (Part 1, p. 68). Social Impact Bonds are one vehicle by which these recommendations will be delivered. They are an arrangement where a non-government organisation delivers an intervention that is first financed by private investors who stand to be repaid with interest from government funding if a social outcome is achieved. There are incentives for each stakeholder to be involved (see the Agensi Inovasi Malaysia diagram below).

(Agensi Inovasi Malaysia)

Creating a market of new interventions to contract via a SIB

Most jurisdictions that have developed a SIB have first scanned their market for investors, intermediaries and proven or promising social delivery organisations. And then they’ve thought about how to run a procurement process that brings the best of these players together, along with an intervention to achieve a priority outcome for Government. Although procurement approaches have varied, all have rested on the ability of the market to delivery suitable interventions that can be managed by organisations with sufficient capabilities to produce the desired social outcomes. Agensi Inovasi Malaysia has enhanced their opportunity to engage with capable service providers by holding a competition for new ideas in priority areas, and then incubating and collecting evidence on these new initiatives, with the end goal of a Social Service Delivery contract. This is not only a way to provide services that are suitable for the first Social Impact Bonds in Malaysia, but creates a pipeline of evidenced programs for the future.

Social impact bonds emerged in the UK in 2010, with 23 currently in operation. Development plateaued, however, during 2013 and 2014 (see chart below).

In the latter half of 2013, attention turned to the development of a pipeline of SIBs to bring to market. Big Lottery Fund and the UK Cabinet Office are working together on “a joint mission to support the development of more SIBs” through their social outcomes funds totalling £60 million. Social Finance, in partnership with the Local Government Association, has been commissioned to support applicants to their funds and there is also a program of grants for organisations requiring specialist technical support to apply (Big Lottery Fund).

Agensi Inovasi Malaysia will potentially avoid the problems of the UK, by seeding and supporting a pipeline of interventions up front. This pipeline has been created through the ‘Berbudi Berganda: Social Impact Innovation Challenge’ which called for social organisations to submit their ideas for interventions to tackle the priority issues of:

youth unemployment
homelessness
elderly care.

The top 12 organisations won funds and support to implement their ideas, the impact of which will be the subject of action research over their first four months. This research will form the basis of a framework and delivery model addressing the priority issues. The pilot program timeline is below.

Apr 2014	Feasibility study
Sep 2014	Focus group discussion
Oct – Nov 2014	Social Innovation Challenge
Jan – Apr 2015	Incubation
Jan – Apr 2015	Intervention
Jan – Apr 2015	Action research and impact study
2015	Social Finance Policy Framework
2015-16	Model for ‘Social Service Delivery’

The benefits of the competition and incubation approach include:

focusing NGO innovation in government priority issue areas
government being able to work with NGOs over a longer period of time, thus gaining a better understanding of the ability of the organisation to deliver effective programs and outcomes
creating an evidence base that will inform the design of ‘Social Service Delivery’
supporting organisations to build and test interventions suitable for a Social Impact Bond.

The Agensi Inovasi Malaysia approach might require more up-front government funding than other jurisdictions have been or will be able to provide. But for a government that has limited experience outsourcing social services, it is a collaborative and supportive way to create a market of interventions that might otherwise not exist.

Exploring Islamic Finance as a source of SIB funding

The potential for Islamic finance to become a source of funding for Social Impact Bonds is significant and has not yet been explored. The Islamic religion obliges its followers to give the zakat, a portion of their wealth to ease inequality and suffering. The total given each year is estimated at 15 times that of global humanitarian aid contributions, and in Malaysia the zakat collected by Government is over US $400 million (Irin News).

Islamic finance includes Musharakah (Joint Venture Partnership), Waqaf (charitable donations), Debt Structure, and Sukuk (Islamic Bonds). A Musharakah could be used as the structure that holds the contracts with other parties. Sukuk could be used for investment, although their flexibility in terms of repayments that are dependent on outcomes will need to be determined. Waqaf could be used to fund a specific fixed cost such as legal fees, extra staff for development of a SIB, software, premises, audit, insurance, performance management or evaluation. The way this could fit into a Social Impact Bond structure is shown below.

Conclusion

Agensi Inovasi Malaysia has created a unique pathway towards Social Impact Bonds. Their approach mitigates the risks of implementing the model in a country without a history of outsourcing social services. They have framed this new contracting model in the broader policy context of public-private partnerships, which aids wider understanding of both the model and the objectives of government. By seeding and supporting new programs that address priority issues, the Government will be able to understand and evidence the impact of these new programs, before contracting them for ‘Social Service Delivery’. Finally, the exploration of the role Islamic finance can play in a Social Impact Bond has the potential to be applied in other jurisdictions and extends the ability of Islamic finance to achieve social outcomes.

This blog was written as a result of a project Emma is working on with Agensi Inovasi Malaysia. It describes aspects of their programs that she found interesting and relevant. These are Emma’s personal views and should not be taken as representative of Agensi Inovasi Malaysia or any other organisation.

Procurement precedents for social impact bonds (SIBs)

January 14, 2015January 14, 2015 Emma Tomkinson Procurement SIB, SIBs, social finance, social impact bond, social impact bonds, social investment

There are many ways to procure for a SIB. The following examples of procurement processes have been chosen to demonstrate variation. The advantages and disadvantages of each are context specific – if you are developing a new procurement process you might want to think about whether each variation promotes or hinders your objectives.

I the word ‘procurement’ to refer to any of the means by which governments might ask external organisations to deliver a service under contract.

Please refer to the source information if you are producing further publications – I have tried to faithfully summarise each procurement process, but my interpretations have not been checked with the parties involved. Happy to accept corrections or suggestions.

Ontario, Canada

Deloitte won the initial RFP is currently in the final stages of that contract, with an Ontario Government decision expected in the next few months. An interesting feature of this process is the parallel ‘internal’ and ‘external’ streams, where public servants are proposing their outcome ideas at the same time as people in the market are also proposing. External ‘registrations’ of interest were called for in the following priority areas:

Housing – Improving access to affordable, suitable and adequate housing for individuals and families in need.
Youth at Risk – Supporting children and youth with one or more of the following: overcoming mental health challenges, escaping poverty, avoiding conflict with the law, youth leaving care, Aboriginal, racialized youth, and other specific challenges facing children and youth at risk, for example employment.
Employment – Improving opportunities for persons facing barriers to employment, including persons with disabilities.

New South Wales, Australia

In New South Wales we suffered from locking ourselves out of developing the idea with organisations over the 6 months it took to run RFP and negotiate the contracts for the next stage. We did not agree a maximum budget or referral mechanism until the joint development stage – we asked for organisations to come up with these as well as a full economic and financial model in their RFP. None of us who were involved in designing the procurement process feel we got it quite right, yet given the opportunity, we would all redesign it in different ways! (See NSW Treasury page on ‘Social Benefit Bonds’)

Procurement timeline:

November 2010	NSW Government commissions a feasibility study from Centre for Social Impact
February 2011	SIB Feasibility Study report submitted and published
March 2011	State government elections and change of government (left to right)
September 2011 (due Nov)	SBB Trial Request for Proposal released
March 2012	3 consortia announced joint development phase begins
March 2013	Newpin Social Benefit Bond contracts agreed
June 2013	Benevolent Society + 2 banks Social Benefit Bond contracts agreed

New York City, USA

An interesting feature of the New York City SIB development process was that service delivery partners were procured for first, and started delivering services while being involved in developing a SIB for future financing of the service.

New Zealand

The New Zealand process appears to be the only one where the government procured for the intermediaries and service providers separately. It is not yet clear what the benefits of this might have been or how they will be matched up.

Massachusetts

Several US states have followed a similar procurement process to Massachusetts, which first involved a Request for Information from organisations external to government. This approach allows the market to shape government thinking and recognises that there may be social issues and intervention types that government hasn’t previously considered. Some jurisdictions have accomplished this with less formal consultations e.g. Queensland Government’s cross-sector payment-by-outcomes design forum and Nova Scotia Government’s cross-sector SIB Working Group.

Massachusetts Selection Criteria:

Government leadership to address and spearhead a public/private innovation.
Social needs that are unmet, high-priority and large-scale.
Target populations that are well-defined and can be measured with scientific rigor.
Proven outcomes from administrative data that is credible and readily available in a cost effective means.
Interventions that are highly likely to achieve targeted impact goals.
Proven service providers that are prepared to scale with quality.
Safeguards to protect the well-being of populations served.
Cost effective programs that can demonstrate fiscal savings for Government.

Department of WOrk and Pensions UK

The Department of Work and Pensions developed a ‘rate card’ for payment per individual outcome for their procurement. They asked organisations to choose a subset of outcomes to deliver, nominate a price per outcome and the intervention that would achieve them. A social impact bond structure was not mandated – seven out of the ten chosen programs involved external investors. The following process occurred twice in 2012:

DWP Rate Card: DWP pays for one or more outcomes per participant which can be linked to improved employability. A definitive list of outcomes and maximum prices DWP was willing to pay for Round 2 is:

Saskatchewan, Canada

This process may be followed if an unsolicited proposal is received. An interesting feature of the Saskatchewan SIB is that the investor has also signed the contract with government.

Essex, UK

The process of developing the Essex social impact bond is described in Social Finance’s Technical Guide to Developing Social Impact Bonds. Social Finance worked closely with Essex County Council to research and develop a SIB, with the final step being procuring for a service provider.

Conclusion

Governments need to think about which information need to be included in a procurement document. For example, if it is desired that organisations external to government come up with completely new service areas, then a procurement process that does not state the social issue to be addressed or contracting department might be suitable. But information and constraints that are known should be included in a tender document. It’s simply irresponsible to have a criminal justice organisation spend time working on a response offering intensive services for 30 female offenders if there was never any possibility the SIB was going to be in justice, or with female offenders, or with a small group of people.

Key questions when procuring for a social impact bond (SIB)

January 5, 2015January 5, 2015 Emma Tomkinson Procurement SIB, SIBs, social finance

While SIBs can be brought to government through unsolicited proposals (in jurisdictions that allow this), most governments will procure for a SIB by going to market and asking for organisations to respond. (This approach to market is usually called a tender or Expression of Interest or Request for Proposals or something else with a different name but similar meaning.)

The question is, what to procure for?

an outcome?
an intermediary?
a service provider?
ideas for procurement?
ideas for SIBs that could be constructed in the future, should procurement proceed?
an organisation to tell government what to procure for?

The first step for government is clarifying its objectives in pursuing a SIB. SIBs are a catalyst for change, but different governments use them to change different things. (See analysis of stated objectives for the first few SIBs.) The objectives of government will make a big difference to how procurement is approached. For example, if a key objective is rigorous measurement of attribution, then a large population and randomised control trial might be required. If a key objective is to adequately fund small providers in remote locations to work together to help their communities, then ‘roadshows’ for awareness, accessible information and additional resources for capacity building might be part of the procurement process. If long-term funding for outsourced social programs is a big change for government, then it may require legislative change or significant work with internal government stakeholders to understand and manage perceived risks.

The main decisions government will need to make in deciding a SIB are:

Social issue area / Contracting department
Payment metric e.g. number of reconviction events or number of days happily employed
Maximum budget OR value per payment metric
Period of time over which budget will be deployed
Cohort / Referral mechanism

These decisions can be made before an approach to market, or after responses have been received. Some jurisdictions have asked their markets to suggest responses to almost all the above decisions, and some have made almost all of them internally before approaching the market.

The final decision is which organisation to contract with, which is usually (but not always e.g. Peterborough, Saskatoon) the result of a procurement process.

I haven’t spoken to anyone so far who thinks their procurement process was perfect, so let me know if you find one where all stakeholders agree it’s good!

Governments going to market with as much of the draft contract as possible may help speed and clarify the process. A lot of the contract will be relevant to most situations, so provision of basic clauses provides clarity and allows negotiation time to be focused on the issues that are new.

The following questions might be useful for governments to ask themselves when approaching procurement:

What are we trying to achieve or test?
Does it matter how much investors stand to return?
Does it matter how many organisations are involved in delivering the SIB and what their relationship is?
Does it matter how money flows?
What really matters to us?
- service improvements in difficult areas?
- innovation in services for populations of service failure?
- a service model that can be used in other locations or service areas?
- savings?
- a shift of focus into preventative services?

Using SROI for a Social Impact Bond

May 7, 2014October 29, 2014 Emma Tomkinson Social Impact Bonds, Social Impact Measurement global value exchange, GVE, outcome measurement, outcome valuation, pay for success, SIB, SIBs, Social Benefit Bond, social impact bond, social impact bonds

Social Return on Investment (SROI) and Social Impact Bonds (SIBs) are two ideas that are increasingly mentioned in the same breath. SROI is a measurement and accounting framework and SIBs are a way to contract and finance a service. Both require three common ingredients:

the quantification of one or more social outcomes for beneficiaries,
a valuation of these outcomes, and
an estimation of the cost of delivering these outcomes.

While not a necessary ingredient, SROI can contribute to the design, operation and evaluation of SIBs.

*NB the word ‘outcome’ is used here to represent a change in someone’s life – some readers (particularly from the US) may use the word ‘impact’ to mean the same

Continue reading →

Social impact bond (SIB) research questions

April 11, 2014May 30, 2014 Emma Tomkinson Social Impact Bonds academic, evidence, legacy, research, SIBs, social impact bonds

It seems that more and more students around the world are keen to do some research on social impact bonds. Great! But we can do better than ‘Do SIBs work?’ (try defining success first…) or ‘What’s the relationship between financial return and effect size?’ We don’t have the data for questions like this yet, but there are so many other wonderful questions we can be asking. There’s also a lot of data on twitter and in the media that could be used for interesting studies of stakeholder perception and reaction.

If you are a student researching SIBs and would like to be connected with other students, please use the contact form and include your university, level of study and research topic. I will connect you via email with other students around the world. If anyone else has more questions they’d like on this list, please pop them in the comment box.

Some questions

What are the effects of publicly announcing a SIB? How does timing affect whether a SIB gets agreed and how long it takes to agree?
What are the key characteristics of SIBs that have been announced by haven’t happened? How is this different from those that have happened? What has been the result of these projects e.g. funded by other means, directly commissioned etc.
Beyond risk and return: what are investors in social impact bonds looking for and attracted to? On what factors does the success of a SIB fund-raising effort rest? E.g. SVA great brochure, Westpac worked with existing customers, Social Finance builds closer relationships and confidence with investors.
Procurement – how do SIBs challenge the assumptions, processes of procurement? Comparison of jurisdictions. Implications for changes in law.
What do you procure for i.e. organisation, idea, full blown proposal or service – advantages and disadvantages from cases around the world.
Making responses public – e.g. Illinois for their request for information – does this impede or encourage innovation?
Appropriation risk (US only?) – how is this quantified, perceived, legislated against and what are the effects of that on a SIB? Conversely, how might the increased profile of a SIB affect appropriation risk?
Unstable governments – what are the implications for contractual partnerships in those countries? What’s the financial cost too?
Relationship between payment metric, measurement confidence and timing of payments – perhaps already in existing literature on performance-based contracting.
The role of guarantees – how does it change perception of what you’re doing by different stakeholders?
Who initiates and drives a SIB – how does this shape perceptions of risks and benefits by different stakeholders?

Legacy

Alex Nicholls and I published a set of questions in our Case Study: The Peterborough Pilot Social Impact Bond (2013, published by Saïd Business School, University of Oxford) around the often overlooked issue of legacy: what happens when the SIB is over. Not all of these are research questions, but studies that go some way to addressing these broader questions might be useful. The following is quoted directly from the paper.

One key question of a SIB is when and why should it end? Other key questions to consider are set out below from the perspective of each stakeholder typically involved in a SIB:

Government

How to institutionalise innovation into future welfare programmes and in the wider social services market?
If early prevention is successful, how to maintain and fund preventative services after SIB ends? Do SIBs need to ‘rollover’ to produce sustainable change?
How can SIB outcomes data (likelihood, effect size, cost of delivery, value or savings to tax payer, related externalities/proxy outcomes) drive better commissioning across government?
How to achieve key outcomes post SIB?
How to continue to grow the social finance market to fund welfare services?
How to report on and share SIB learning and data more widely?
How to calculate savings from SIB interventions?

Investors

How to develop a secondary market exit?
How to develop a follow-on SIB investment?
How to adjust risk and return dynamically to the availability of new information from SIBs in the market?
How to tranche investments in a single SIB according to different risk and return profiles and different personal costs of capital?

Service Providers

How to ensure continuity of funding of increased capacity?
How to institutionalize SIB performance data?
How to build capacity to engage in future SIBs?
How to manage on-going collaborative relationships?
How to disseminate learning?
How to leave a community stronger when a service ends?

Intermediaries

How to build a pipeline of SIB deals?
How to build capacity in providers so that they are stronger for having worked on a SIB?
How to continuously innovate?
Where to apply SIBs and develop other models that build upon SIB learning?
When are SIBs no longer necessary, if ever?
How to build a business model, given high transaction costs?
How better to segment the investor market to the real, rather than perceived, risk and return opportunities of SIBs?
How to manage the involvement of commercial, rather than purely social, investors in terms of expectations of high returns and the potential for risk dumping?

Service Users

How to ensure that a service gap does not arise for current participants and relevant future populations/cohorts?
How to avoid worse outcomes in the long term?
Will improved outcomes be sustained for those who participated in a SIB?

Who do you need in government to get a SIB off the ground?

December 9, 2013December 9, 2013 Emma Tomkinson Social Impact Bonds DWP Innovation Fund, government, Harvard SIB Lab, resourcing, SIBs, social impact bonds

The social impact bonds (SIBs) so far agreed have displayed striking similarities in how they have been resourced. They have, at a minimum, featured two key roles: (1) the worker and (2) the senior champion. After speaking to most of the people responsible for SIBs agreed across the globe, it appears to me that these two roles are essential to driving SIB development all the way to signing a contract. These two roles are by no means sufficient to achieve a SIB contract, but they are necessary.

The worker

Somebody needs to take responsibility for doing the bulk of the work within the commissioning body. This involves briefing and negotiating with internal stakeholders, acting as secretariat for decision-making committees, collecting and analysing data analysis and liaising with external stakeholders. Imagine this person researching, writing and then physically chasing up decision documents and moving them from one desk to the next. They need the flexibility to work in this way, requiring different processes and a broader scope than those in established roles, and they need to be enough of a risk-taker to keep fighting the barriers to change. They need to enjoy the exploration of something unknown and be naturally inclined to include both internal and external stakeholders on the journey. They cannot do this work on top of their normal day job. People who have performed this role say:

“I almost created my own job”

“I was never part of the wider set-up – I was never a civil servant within the hierarchy.”

“I’m a bit of a square peg in a round hole”

The focus of developing a SIB is to get everyone to agree. Working towards an agreement necessarily involves a range of stakeholders as the project progresses in an iterative manner. Sometimes external service delivery providers, data custodians, other government agencies and investors have been involved in discussions before a government decision has been sought to work on development. Sometimes the government decision is made first and then work with stakeholders sets the direction of the development process. A SIB won’t happen because one person worked very hard, alone at their desk for a long time, they need to be saying, “I’m a networker – I connect with people and get things done”.

All successful SIBs have been driven by people who are passionate about the project, who won’t give up on finding solutions to new problems on a daily basis “it was about keeping the energy going – you can’t recreate that in someone that is not interested”. Several SIB development projects have stalled within public agencies due to the nominated SIB producer rationally declaring this mountain too high to climb.

The senior champion

Doing the legwork on a project that breaks new ground every day is too hard a task if the ‘worker’ also has to justify why they are creating work for everyone by pushing through change. A senior champion is crucial to validate the project and its resourcing, as well as compelling the necessary people at all levels across government to participate in discussions. Interviewees said things like, “My boss allowing me to have time to work on it was absolutely key” or “Having them state this was their pet project opened doors.” The senior champion is also responsible for political negotiation and positioning.

Informal connections may lead to successful formal ones

Vital SIB partnerships have been started in unlikely places. These include parties, running groups and corridors. This reflects the cross-sectoral nature of SIBs and that at the moment, they are built by alliances between first-movers, rather than through established communication channels.

Why are SIBs so difficult for government agencies to develop?

SIBs are difficult because they involve change. Public service agencies aren’t set up to change often – they are structured to maintain huge, vital service systems and every public servant is a dedicated cog in this machinery. SIBs ask for a whole lot of changes at once. The first is the measurement of outcomes and the valuation of those outcomes, such that payment may be made on the basis of measurement. That’s pretty new for most public agencies delivering services. SIBs also involve procurement practices that don’t specify ‘how’ services should be delivered, which can be contrary to traditional risk and quality management practices. Social impact bonds require new ways of thinking and decisions from public service units responsible for legal, finance, procurement, service design and data. And then all the external stakeholders are required to change the way they’ve been thinking and operating too!

The Harvard SIB Lab

The Harvard SIB Lab is a clever model because it recognises the difficulty public agencies have with resourcing SIB development. The Lab rewards successful applicants with someone dedicated to working on the project. This person may or may not take on the role of owner of the work, depending on the person and situation.

Department of Work and Pensions Innovation Fund

This suite of programs, which includes several SIBs, was developed differently to other SIBs so far. An allocation of £30m was made for an innovation fund to explore early intervention programs with young people. The next step was to think about how to distribute this funding and a decision made to tender for ten programs financed by social investment, some of which are SIBs. Public servants involved in this project seem to have been able to do so within their existing roles.

Did we already have social impact bonds?

November 12, 2013November 22, 2013 Emma Tomkinson Social Impact Bonds definition, not-for-profits, outcomes-based contracting, performance-based contracting, SIBs, social impact bonds, UK Cabinet Office, US

(Social Finance, Towards a New Social Economy, 2010)

During a recent lecture on social impact bonds for the Saïd Business School’s Executive Education Impact Investing course, one of the students asked me, “Am I already part of a social impact bond?”

Social impact bonds (SIBs) are a new construct, with the term coined in 2008 and the concept piloted by Social Finance in Peterborough in 2010. But arrangements that fit our current understanding of SIBs already existed and continue to operate. For organisations involved in these arrangements, the benefits and challenges of identifying with the SIB phenomenon will determine whether or not they associate themselves with the term.

The lady who asked the question was running a non-profit organisation that had an outcomes-based contract with the Louisiana State Government to provide a service. A loan from a local finance institution helped to cover their working capital.

In order to examine whether she was indeed part of a SIB, we turned to the Cabinet Office definition.

In this arrangement, the first two conditions were certainly met.

To determine whether the third was, we examined whether her creditor could be considered an investor. Her organisation existed for the purpose of providing the service and the contract accounted for the entirety of their service provision. The loan was made on the basis of this single contract with government. The lender was a legally separate entity from both the non-profit and the government. So it looked like her creditor could be considered an investor.

The fourth condition relates to the financial risk of the investor. Was the finance institution truly taking on any financial risk? Indeed, there were two distinct types of financial risk taken on in this situation. Firstly, there was the risk of non-performance; that the non-profit would fail to deliver the results that would trigger payment from government. Secondly, there was ‘appropriations risk’, the risk that government doesn’t approve the payment of a previously agreed contract [1]. If the non-profit, for either of these reasons, did not receive sufficient funds from the government to pay the loan, they would become insolvent and the finance institution would not have the loan repaid. So the fourth condition was met.

This non-profit was certainly part of a social impact bond as defined by the UK Cabinet Office, although none of the parties involved were using the term.

Other examples that may meet the definition of a SIB but not identify with the term include charter schools, which are independently run with three year contracts based on outcomes. If they establish a bridge loan fund to cover the cash flow gap, they may meet the four necessary criteria. These bridge loans are often used for capital infrastructure spend and the financial risk for the lender is sometimes rated, so certainly exists.

Implications

When we assess SIBs and compile data or ‘track record’, it may be worth looking at these existing arrangements that fit the SIB definition.
Organisations that are part of arrangements that fit the SIB definition might consider the benefits and challenges associated with the SIB label (e.g. will appropriation and other payment risks be reduced if the contract is ‘announced’ as a SIB).
The Cabinet Office definition came after several SIBs had already been developed and was not written by the Social Finance visionaries responsible for articulating and implementing the original concept. Does it misrepresent the concept? If so, do we need to change the definition?

[1] Appropriations risk is a feature of the US governmental systems and is described by Steve Goldberg in his fourth SIB Trib: “Even if a state signs a valid contract calling for the payment of money, it doesn’t (indeed, can’t) pay what the contract says it owes unless and until the state adopts an appropriation law specifically and affirmatively authorizing the expenditure. What’s more, legislative authority to appropriate funds only lasts as long as the state’s constitution allows budgets to remain in force. In most states that’s one year, in some, two. So not only does a state have to enact budget legislation and then pass separate appropriations bills to legally spend any money, they can only do the latter within the one- to two-year time horizon of the former” (Goldberg, 2013).

The Justice Data Lab – an overview

July 16, 2013April 3, 2015 Emma Tomkinson Criminal Justice, Evidence-based policy, Open project/source/data/government, Social Impact Bonds, Social Impact Measurement Minstry of Justice, NPC, reoffending, SIBs, social impact, social impact bonds, Transforming Rehabilitation

What is the Justice Data Lab?

The Justice Data Lab allows non-government organisations to compare the reoffending of the participants in their programmes with the reoffending of other similar ex-offenders. It “will allow them to understand their specific impact in reducing re-offending… providing easy access to high-quality re-offending information” (Ministry of Justice, Justice Data Lab User Journey p.10). There is no charge to organisations that use the Justice Data Lab.

The Justice Data Lab is a pilot run by the Ministry of Justice. The pilot began in April 2013. Each month, summaries of results and data are published, including Forest plots of all results so far.

Who might use it?

The Justice Data Lab can be used by “organisations that genuinely work with offenders” (Justice Data Lab User Journey p.11). One request will provide evidence of a programme’s effect on its service users’ reoffending. Several requests could compare services within an organisation or over time to answer more sophisticated questions about what is more effective.

This information could be used by non-government organisation for internal programme improvements, to report impact to stakeholders or to bid for contracts. It was set up at the time the Ministry of Justice’s Transforming Rehabilitation Programme was encouraging bids from voluntary and community sector organisations to deliver services to reduce reoffending.

What are the inputs?

Input data are required to identify the service users from a specific program and match them with a comparison group. Information on at least 60 service users is required and the organisation must have worked with the offender between 2002 and 2010.

Essential:

Surname
Forename
Date of Birth
Gender

At least one of the following:

Index Date
Conviction Date
Intervention Start Date [note: feedback from applicants is that this is required]
Intervention End Date [note:feedback from applicants is that this is required]

Highly Desirable: PNC ID and/or Prison Number

Optional: User Reference Fields

What are the outputs?

The one year proven re‐offending rate – defined as the proportion of offenders in a cohort who commit an offence in a one year follow‐up period which received a court conviction, caution, reprimand or warning during the one year follow‐up or in a further six month waiting period. The one year follow‐up period begins when offenders leave custody or start their probation sentence. A fictional example of the output provided by the Ministry of Justice is quoted below:

The analysis assessed the impact of the Granville Literacy Project (GLP) on re‐ offending. The one year proven re‐offending rate for 72 offenders on the GLP was 35%, compared with 41% for a matched control group of similar offenders. The best estimate for the reduction in re‐offending is 6 percentage points, and we can be confident that the reduction in re‐offending is between 2 and 10 percentage points.
What you can say: The evidence indicates that the GLP reduced re‐offending by between 2 and 10 percentage points.

Publication
Applicants should note the following requirement: “an organisation requesting data through the Justice Data Lab must publish the final report, in full, on the organisation’s website within four months of receiving the final report.”

I’d be very interested in the opinions of applicants on this requirement. Is it an issue? Does it create perverse incentives?

What are the implications?

The implications are huge. Prior to the Justice Data Lab it was very difficult for non-government organisations to establish a comparison group against which to measure their effect. Evaluations of effect are expensive and thus prohibitive, particularly for smaller organisations. In addition, the differences in their methods and definitions meant that evidence was more difficult to interpret and compare.

This is exactly the type of evidence that developers of social impact bonds find so difficult to establish and will be essential to constructing social impact bonds to deliver Transforming Rehabilitation services. It is a measure of outcome, which is desirable, but often more difficult to quantify than input (e.g. how much money went into the programme), activity (e.g. what services were delivered) or output (e.g. how many people completed the programme).

New Philanthropy Capital (NPC) were involved in designing the Justice Data Lab and their Data for Impact Manager, Tracey Gyateng, is specifically thinking about applications to other policy areas.

How is it going?

See my November 2014 post on information coming out of the Justice Data Lab.

Also note the announcement of an Employment Data Lab by NPC and the Department of Work and Pensions.

More information

Information on the Justice Data Lab home page includes links to a series of useful documents:

User journey document – information on what the justice data lab is, and how to use its services.
Data upload template – use this template to supply data to the justice data lab. Further descriptions of the variables requested are given, and there are key areas which must be filled in on the specific activities of the organisation in relation to offenders.
Methodology paper – this document gives details of the specific methodology used by justice data lab to generate the analysis
Privacy impact assessment – this is a detailed analysis of how an organisations’ data will be protected at all stages of a request to the justice data lab
Example report template – two examples of a standard report, completed for two fictional organisations showing what will be provided.

Criminal justice service providers might also benefit from getting involved in the Improving Your Evidence project, a partnership between Clinks, NPC and Project Oracle. The project will produce resources and support, so follow the link and let them know what would be of most use. The page also links to an introduction to the Justice Data Lab – a useful explanation of the service.

The bulk of this post has been copied directly from the Ministry of Justice documents listed above. It is intended to act as a summary of these documents for quick digestion by potential users of the Justice Data Lab. The author is not affiliated with the Ministry of Justice and does not claim to represent them.