The EVIDENCE and IMPACT Cluster: The Global Alliance

In case helpful here are links to some of the evidence and impact data identified through The Communication Initiative process as potentially including compelling data:

Impact Examples: Girls' and Women's Empowerment

Impact Examples: Child Health Communication Programming

Impact Examples: Natural Resource Management (NRM) Communication Programming

Impact Examples: Media Development Programming

Impact Examples: Polio Communication Programming

Impact Examples: ICT4D Programming

Impact Examples: Early Child Development Communication Programming

Impact Examples: Democracy and Governance Communication Programming

Impact Examples: HIV/AIDS Communication Programming

AGENDA and BACKGROUND NOTES
For Global Alliance EVIDENCE and IMPACT CLUSTER Conference call
Wednesday May 16th at 10am EST
CALL IN NUMBERS follow at bottom

To: Particpants in the EVIDENCE and IMPACT Cluster in The Global Alliance - Johns Hopkins University Center for Communication Programs; BBC Media Action; CORE; IAMCR; ISMA; Open Society Foundations; Soul City; UNICEF; USAID

cc. Others in The Global Alliance - if you are not yet part of a cluster and wish to particpate you are most welcome.  

Chair: Susan Krenn (Johns Hopkins University Center for Communication Programs)

AGENDA

1. Review of overall brief for the EVIDENCE CLUSTER - See Appendix: 12 month plan - EVIDENCE AND IMPACT below extracted from the Agreement to Cooperate document as developed by the interim Alliance group.

2. Consideration and decisions related to the short-term options for moving forward - see Evidence and Impact - Short-term option proposal below.

3. Consideration of the long-term options for moving forward - see Evidence and Impact - initial long-term ideas and options below.  

*********

RELEVANT PAPERS and NOTES

*********

A. Appendix: 12-month plan – EVIDENCE and IMPACT

This is the text included in the Agreement to Cooperate. It outlines the focus for the Evidence and Impact Cluster.
 

Goal:

To identify and agree upon 20 compelling (results, methodology, and journal published) research results that draw a direct connection between a communication, media, social and/or behaviour change strategy, programme, or social trend, and positive movement related to one or more SDGs in local, national, or global contexts.

Products:

1.    A 2-page distillation of the results with the most compelling data.
2.    A 4-page outline of the strategic implications from that data.

Supporting this field of work:

a.    Organise extensive communication of the 2-page distillation and 4-page strategic outline through all major networks relevant to this field of work.

Responsibility:

A small action group from within the Alliance to pursue this goal and develop these products with updates on progress and issues reported to meetings of the Alliance.

The group would select their own Chair. Interim Secretariat support would come from The Communication Initiative.

Commitment: Staff time as available from the Alliance participants involved.

******

B. Evidence and Impact - Short-term option proposal

The proposal below was first considered by the interim group. The initial responses from within the interim group inform the longer-term ideas outlined in the next section below. 

Short-term: In the consultations that took place related to the development of what is now the Global Alliance, there was overwhelming agreement that we should try to move quickly on the key items including evidence and impact. Hence why there is the outline of a short-term measure related to evidence and impact.

Overall approach: There is very considerable evidence and impact knowledge within the Alliance participating organizations - either from research they have undertaken or from reviews of the literature related to their own priorities in order to inform and improve their programmes and highlight the impact of their work.   

Proposal is to tap into the evidence and impact knowledge of the participants in the Alliance in order to develop an initial list of 20 to 30 strong data points that indicate the demonstrated added-value of our common field of work. This would be a preliminary output - not the definitive list. It would provide both a valuable product for initial communication (including for the Policy Engagement cluster) and also allow us to assess the best way forward for the longer-term evidence and impact work of the Alliance.

Steps:

1. That we agree a set of criteria for acceptable, compelling evidence and impact data, for example: (a) published in a peer review journal, book or other published literature; (b) results draw a direct connection between a social change, behaviour change, communication for development or media development strategy, action or trend and a positive change in relation to one or more development priorities; and (c) a solid research methodology, with a bias towards methodologies that have comparative data at the centre of that methodology.

2. That we develop a simple survey monkey for Alliance focal point people that asks them to identify the 3 to 5 pieces of evidence and impact that they have found to be the most compelling and credible for them in their social change, behaviour change, communication for development and media development work. Some key questions for that survey, (with an example answer from the Sesame Street meat-analysis as an example to illustrate) could include:

Title of paper, chapter or book: (example) Effects of Sesame Street: A meta-analysis of children's learning in 15 countries

Name of author(s) : (example) Marie-Louise Mares and Zhongdang Pan

Journal, Book or Paper published: (example) Journal of Applied Developmental Psychology
Volume 34, Issue 3, May–June 2013, Pages 140-151

URL (if available); https://www.sciencedirect.com/science/article/pii/S0193397313000026

Development Issue(s): Education

Strategic Approach: Entertainment-Education

Compelling data: “Researchers found an overall effect size of 0.29. This translates into an 11.6 percentile gain (in terms of education). That is, an average child who does not watch Sesame Street is at the 50th percentile, whereas a child who watches is at the 62nd percentile.”

Research Methodology: A meta-analysis of 24 studies of the effects of Sesame Street internationally.

3. That when the results are in we assess and build a list of 20 to 30 pieces of evidence and impact data balanced across development issues if possible as an initial output from the Alliance. It would be branded as initial with a clear indication that this is the first step in a longer process.

4. This would be communicated through the networks in this field of work (a) for their use if of value and (b) for review and comment - perhaps with the same form as above for them to complete.  

NOTES

a. The initial outline of a different version of this approach attracted the following comments and critique from the interim group:

“It strikes me that 3 - 5 papers from each participating organization -- with a possible high-end total of almost 90 papers -- might be overwhelming even before we begin.”

“Are we highlighting certain technical themes- around SDGs or some other criteria? Will there be sub-groups of certain themes that are relevant to move development and humanitarian evidence SBCC forward? I think if we structure this a bit more, and have thematic areas that we can identify key evidence/examples from those groups, then it may be easier to prioritize what evidence we select among the multiple great examples.”

 
b. By way of background The Communication Initiative has tried to comply lists of compelling evidence and impact data culled and derived from the evidence and research  submitted by The CI network. In case of interest you can review from at this link.

 
********

C. Evidence and Impact - initial long-term ideas and options

The outline above is an intermediate measure for a solid and quick product. But there is agreement that we would need to go further over the longer-term.

As a way to start the dialogue here are some ideas gathered from the interim group for the Alliance:

“Perhaps we should first draw on efforts that have already been done by the professional societies to pull together a compendium of data that demonstrates the efficacy of a social and behavior change approach to addressing social challenges. In my mind, the work of this group should be focused on sorting through what has already been done in this area rather than trying to find, review, and select new papers or articles on this topic. And, of course, because of this, i think those of us who spend more time in the literature than i do, i.e., Radhika and Jeff French, are well positioned to make if not short, than lighter work of this Make sense and, if so, what is the next step?”

“I agree, let's build upon ongoing efforts. HC3 worked in SBC evidence collection for various health areas and so did others. Maybe a review could look at the evidence gaps. Meanwhile, the Breakthrough Action project is also undertaking a similar activity. Could we map out ongoing efforts first and see how we can leverage them best?”

“Echoing (the above) points on this. In addition to the HC3 evidence reviews for SBC in HIV/AIDS, malaria, and FP/RH, and the more recent Breakthrough-RESEARCH health sector-wide research landscaping, I'd recommend we revisit the various outputs of the 2013 Evidence Summit on Population-Level Behavior Change for RMNCH, and the ongoing WHO evidence review for SBCE in RMNCH.

I know the Breakthrough-RESEARCH and WHO folks are just beginning to work to align their respective activities - do we want to reach out to them now, or try to carve out some time for a focused face-to-face conversation at the Summit?”
 

*******

CALL-IN DETAILS

Join by Skype - https://meet.unicef.org/cbondanza/TG1TLM2P

Join by phone:
Conference ID: 161689917
 
United States : +16467571480 (Global)                         
Switzerland : +41225083200 (Global)                            
United Kingdom : +443300102423 (Global)                    
Panama : +5073017399 (Global)                    
South Africa : +27879403508 (Global)                            
France : +33176542732 (Global)                       
Netherlands : +31705680050 (Global)                           
Brazil : +556136860910 (Global)                      
Belgium : +3228080292 (Global)                       
UNICEF VOIP: 4103010000 (Global)                                  
 
Conference ID: 161689917

Notes from meeting
Evidence and Impact Cluster meeting
Wednesday May 16th, 2018
Chair: Susan Krenn - JHU CCP

Participants

Patrick Cook - ISMA
Kama Garrison - USAID
Urvashi Ghandhi - Breakthrough
Brett Davidson - Open Society Foundations
James Deane - BBC Media Action
Antje Becker-Benton - Save the Children USA
Deborah Freitas- CORE
Doug Evans - ISMA
Vincent Petit - UNICEF
Rafael Obregon - UNICEF
(Apologies if I missed anyone - please advise)

Warren Feek - The Communication Initiative

The main themes of the meeting were outlined in the agenda and background notes.

The major points from the meeting include:

General points to commence the discussion

1. Different ideas about what constitutes evidence and data
2. As we develop the process we will need to be clear about who the evidence is for
3. Evidence is not enough - will need to figure out how we communicate that evidence in a  way that has impact
4. A number of systematic reviews - need to tap into those opportunities long-term
5. Will be important to frame what we do for people who are “outside our tent”.
6. Will we just be restricting what counts as valid data to “peer reviewed journal articles” or can we expand to so-called grey literature.
7. It is not just collecting the data - the distillation and analysis processes and outcomes are really important.  

Short-term action

1. Agreed to the strategy of asking Alliance participating organizations to submit the 3 to 5 compelling “pieces of evidence and data that they find (from their perspectives) to be most compelling.
2. In doing this need to be clear with everyone about what judgements we are making about this evidence and data
3. Established a small working group to develop the survey monkey for Alliance participants to complete - Sonia Whitehead (BBC Media Action), Kana Garrison (USAID) and Antje Becker-Benton (Save the Children) who will also involve her colleague Paula Valentine (Save the Children - UK) and Warren
4. Also established a working group to organise and analyse the data after it comes in - Doug (ISMA), someone from JHU CCP (Susan to identify), Urvashi (Breakthrough), Deborah (CORE) and Warren
5. Stressed that though these relate to this short-term task/outcome there will be important learnings for the longer term work that takes place.
6. Concerning the time frame it was agreed that it would be ideal if the results were available for the POLICY ENGAGEMENT work by the Alliance at the HLPF in midiJuly. The draft schedule agreed was - launch survey by 3rd week of May; deadline for responses is 3rd week of June; compilation and analysis by end of 1st week in July.

Longer-term action

People referred back to their opening comments - see above. We will return to this in future meetings.

Next meeting

Rafael and Warren are working out an overall meeting pattern across all of the Alliance and will present some proposals soon

A sub-group of the Evidence and Impact Cluster has considered the draft survey designed to elicit from Alliance participant orgs evidence and impact that they consider compelling. This post draws together that dialogue.

FROM WARREN 

Kama, Antje and Sonia (James volunteered you Sonia!)
 
cc  James, Susan, Rafael, Chris, Charlotte
 
 
Hi and thanks for volunteering this morning to review the survey to Global Alliance participating orgs as the very initial step 1 in the process of distilling compelling evidence and impact data around which we can confidently gather to both advocate the importance of our work and inform strategic development.
 
Attached is the first iteration, for your review and comment, of the survey for Alliance participating organisations only.  A few comments: 

1. The intro is long as (a) wanted to ensure there was clear background and orientation in the same place as the survey itself and (b) there are some important foundations for this initiative.

2. You may/will need to complete the person and org Q at the bottom of page 1 in order to move to page 2. We will delete that data before launch of course.

3. Pages, 2, 3, 4 ,5 and 6 are all the same – the up to 5 responses requested, except that for pages 5 and 6 the answers are not compulsory of course as only 3 responses required minimum.

4. So to review the core questions you will only need to access page 2 – the rest are the same. Again any data you insert can be deleted before launch.

5. I could not see how to remove the image from pages 2 to 6 but keep it on page 1 (home) - aaaaarrrrggghhhhhh – will keep looking

6. I tried to keep this as basic as possible.

7. There are always loads of little things with these surveys so all comments – big and little really appreciated. (Have cc’ed Chris also as he is much better than me on this.)

When we have this the way we want it we will get some full test responses done to test the collection and analysis process. 
 
OK the survey is at this link -  https://www.surveymonkey.com/r/GQSFHYS – please do not share with anyone else at present.

RESPONSE FROM SONIA WHITEHEAD (BBC MEDIA ACTION)

Hi Warren,
 
I think it may be good to broaden the options out – so that the evidence could include grey literature (reports) rather than just published journal articles and books.  Also people may cite internal documents, and this may be useful to know too.
 
I agree with Kama that I think it would be good to add questions such as:

 
1. How have you used this evidence in your work?

2. Why do you think you have found this evidence useful?

 
May be also good to understand what it was about the evidence that they liked e.g. whether it was the language used, the fact that it linked to a particular piece of work.
 
I think you could drop the what methodologies they used (as can check from report/paper) and also it may not be from SBCC specifically.  I just did a trial round the office and a few things that people used most weren’t communication related, were around frameworks and models – e.g. https://pdfs.semanticscholar.org/679d/eb3ea4aa502aa3980e5ef5e5a94725d2e…
 
Could also ask how did you come by this evidence?  Be good to see if its through web, word of mouth, hearing about it in conferences etc.  
 
Thanks,
 
Sonia

RESPONSE FROM KAMA GARRISON (USAID)

 Hi Warren,

To start the conversation here is my initial feedback. I may have a few other comments/suggestions once I chew on it a bit:

 
1. In the intro rather than saying we are developing an initial set of data (which we are but the word data has so many meanings to people) we are really just gathering a list of references to start with...so an initial list of reference

2. I'd add something such as " this effort is not to replace other ongoing formalized evidence reviews" or something along those lines. That document that was posted on CI after the summit about a summit research agenda confused many so lets  try to be clear that this isn't an evidence review or at least a systematic evidence review nor is it exhaustive. It is an effort to get a sense of what evidence the alliance members find meaningful for their work.

3. The policy effort will reflect this information plus, I hope, other systematic evidence reviews that are ongoing. I don't think it would be in our best interest to have the HLPF only reflect an informal effort we made to go out to people and ask what evidence they find meaningful/useful. So the sentences  in the intro about the policy group using this information  needs to say something about the policy effort and HLPF will reflect this information plus the more formal systematic reviews.

 
4. potential additions: Why is this evidence useful to you? How do you use this evidence in your work?

 
5. Last question/suggestion: do we want this format so open or should we try to use drop down menu choices with an "other" option for additional information? With Survey Monkey you can then summarize in tables quickly.  So, for example, in the methodology(ies) question we'd have a drop down of different methods (they could select more than one) and an other category and a don't know category. We can do that for each question as appropriate.
 
Thanks for getting us started. Sorry if this is a bit sloppy and informal...moving quickly.
Kama

RESPONSE FROM ANTJE BECKER-BENTON (SAVE THE CHILDREN)

Hi Warren,

 
Sorry for the delay, it was an extended holiday weekend for Paula as well. And, there is barely anything to add to the thoughtful comments of Kama and Sonia. Here a few practical things

1. The task of the exercise needs to stand out more in bold

2. Agree with Kama to explain that people can revert to existing Meta Analyses if that is what they have been working with and referring to

3. Do we need to discourage this as an exercise where everyone is just showing off their own latest publications, or do we not?

4. We need a “back” button from the beginning and a way to save inputs

5. Define which browser works best, I got stuck on explore.

6. In order to pull relevant evidence, I would skip the question on quotes and do that in a second round, you may end up with nothing because of it.

7. Also, the questions in the introduction on the type of publication are much shorter and better than the extended versions in the actual survey. I LOVE Kama’s idea of a drop down menu!

8. I would ask for evidence and outcomes in the title, I find impact confusing (Health Impact?)

 
 
That’s all for now and kudos for the great tempo in developing this.

Antje
 
 
  

Evidence and Impact Survey

Hi and best wishes. Following work by the sub-group of the Evidence and Impact Cluster, at this link please find the Evidence and Impact Survey. Can your organisation please complete and submit by Wednesday, June 20th, 2018 (in 2 weeks).

A few quick notes:

1. This point has not been raised by anyone but just in case it is of silent concern, there is no intention to make the selections of each of the individual organizations publicly available and identified as their selections. There will be a compiled list that does not identify who submitted which of the pieces of research in that compilation.

2. Though this is stressed in the introduction to the Evidence and Impact Survey to repeat that we are looking for “compelling” not “most compelling”. If we had asked for “most compelling” that requires all sorts of criteria being developed and applied in order to justify the “most” tag. At present all that is being asked for is 3 to 5 “pieces of evidence/impact evidene and data" that your organisation, from your perspective and experience, regard as “compelling” in demonstrating evidence of the impact of this field of work on priority development issues.

3. For that reason we are comfortable with the June 20th, 2018 deadline date. And this will very significantly help with planning the policy engagement efforts around the High Level Political Forum in July 2018. 

Thanks - very much appreciated - Warren (and on behalf of Rafael)

PS - A final reminder re the “Global Alliance: Meetings time preferences and Agreement to Co-operate signatory” survey. If you have not yet completed please do so immediately. We will make proposals Thursday June 7th close of business. Thanks - W

To: Global Alliance - EVIDENCE and IMPACT Cluster
 

Sonia Whitehead (BBC Media Action); Urvashi Gandhi (Breakthrough); Leena Sushant (Breakthrough); Marian Casey-Maslen (CDAC); Anne Marie Hammer (GFMD); Radhika Gajjala (ICA); Bruce Girard (IAMCR); Tom Jacobson (IAMCR); Patrick Cook (ISMA); Doug Evans (ISMA); Susan Krenn (JHU CCP); Doug Storey (JHU CCP); Brett Davidson (Open Society Foundations); Antje Becker-Benton (Save the Children); Paula Valentine (Save the Children); Sue Goldstein and Sue (Soul City); Rafael Obregon (UNICEF); Ketan Chitnis (UNICEF); Hope Hempstone (USAID); Kama Garrison (USAID); Warren Feek - Secretariat (The Communication Initiative); Charita Bondanza - Adminstration (UNICEF).

cc - Other Alliance contacts for information and coordination purposes 

Hi - hope that all are well. Below please find the AGENDA and CALL-IN numbers for the EVIDENCE and IMPACT CLUSTER meeting on Wednesday, June 20th at 10am New York time. You will have received the online notification from Charita at UNICEF. Please do respond and confirm (if you have not already done so).

The meeting will be chaired by Susan Krenn. 

AGENDA - EVIDENCE and IMPACT CLUSTER - Wednesday, June 20th at 10am New York time (Call in numbers are repeated below.)

1. Review notes from the previous meeting on May 16th, 2018 - please review at this link 

2. UPDATE: Alliance participant evidence and impact survey. 

This was agreed by the sub-group working on this task and sent to all Alliance participant contact people with a deadline of June 20th, 2018. The survey is at this link. Please do complete if you have not yet so done! 

3. UPDATE: Longer term planning to identify the most compelling evidence and impact data. 

There are 3 different parts to this: 

a. Parameters for identifying most compelling evidence and impact data: Warren was asked to put together a set of options for defining and agreeing the boundaries we will put in place to ensure that the most compelling evidence and impact data is focused. 

This has been dome in the form of a survey - seemed the best way to identify and assess those boundary lines and bulls eyes! That ’survey” can be reviewed (and completed if you wish) at this link. 

An opportunity for comment and input. 

b. Process to identify most compelling evidence and impact data: Susan and Doug at JHU CCP volunteered and were agreed to work with a sub-group of this cluster on the process for identifying the most compelling evidence (within the boundaries of [a] above). You can review at this link Doug’s recent note to the sub-group (which I have uploaded to the thread for sharing purposes) with the attachments that provide a draft of the initial thoughts. 

An opportunity for any thoughts from participants as this process moves forward.

c. Evidence and Impact from the SBCC Summit - interest was expressed in the evidence and impact data presented at the SBCC Summit. I looked at all of the presentations from sessions coded on the Summit web site as “Evidence and Results”. Please see attached. It is a working document. The commentary is mine and has already been challenged in the Summit Secretariat. This is shared for your review and as an overall insight, through one lens,into what came out of the Summit related Evidence and Impact. 

For review and comment 

4. Any other business. 
   

CALL-IN NUMBERS AND LINKS

To Join by Skype
https://meet.unicef.org/cbondanza/9HVTJCCF

To Join by phone

Conference ID: 61531133
 
United States: +16467571480                          
Switzerland: +41225083200                             
Hungary: +3617909400 (Global)                        
United Kingdom: +443300102423                    
Jordan: +96265509679                     
Panama: +5073017399                    
Cambodia: +85523260206                              
South Africa: +27879403508                             
Bulgaria: +35924928220                   
Denmark: +4578793993                    
France: +33176542732                      
Netherlands: +31705680050                            
Australia: +61730628687                    
Brazil: +556136860910                      
Belgium: +3228080292                      
Greece: +302112340307                    
Italy: +390697632494                        
Italy: +390552033399                        
Albania: +35544548424                      
Moldova:  +37322893037                  
Georgia: +995322422802                   
Ukraine: +380443922163                   
Austria: +43720775772                       
Turkey: +902127055614                    
Thailand: +6623049399                     
Serbia: +381114410154                     

Please let me know if you require other numbers 
 
 
Conference ID: 61531133

Participants: Susan Krenn - Chair (JHU CCP); Doug Storey (JHU CCP); Doug Evans (ISMA); Rafael Obregon (UNICEF); Debora Freitas (CORE Group); Urvashi Ghandi (Breakthrough); Sonia Whitehead (BBC Media Action).

Key items from this meeting include:

1. The deadline for the Alliance participants compelling data survey - was extended to June 30th, 2018. Warren was asked to let everyone know.

2. On behalf of CORE Group Debora asked if that survey could be shared with their members so that the could build their responses based on the answers from members. This was agreed for CORE Group and any other membership based organizations in the Alliance.

3. Warren apologized that he had jumped the gun on the work being done by the small group charged with developing the longer term plan for identifying the “most compelling evidence”. The first draft of the paper “Global Alliance Evidence SBCC Review process” was meant for that small group only. However discussion did ensue as it had been shared.

a. Urvashi wanted to check on duplication in the criteria between the 2 documents. Susan explained that there was some overlap but they were designed to be mutually supportive. The draft survey seeks to provide the overall paramaters whilst the detailed player from Doug and Susan seeks to provide a detailed set of criteria to ally.

b. Rafael asked that for both processes we had some clear statements about what constitutes evidence and impact; that we develop a shared understanding of those concepts relative to our field of work.

c. Doug summarized two main questions that we would need to review (i) evidence of what? and (ii) what do we wish to do with this evidence - how do we wish to use it?

d. Warren provided an example of a difference with the paper - in his view “reach” is not part of impact. 

e. Antje wanted to make sure that we were covering  the full range of SBCC approaches and methodologies.

ACTIONS

The following action was agreed:

A. All participants in the EVIDENCE and IMPACT Cluster to review the paper by Doug and Susan - see attachment “Global Alliance Evidence Review Guidelines Draft” at bottom of this post - please open, use track changes and reply to this email with the  attachment.  

B. Ensure that we all complete the compelling evidence and impact data survey at this link 

C. On June 30, 2018 Warren to share all of the results from that survey (providing we have all responded and submitted our own survey answers!)

D. Warren promised to share the review of the "Evidence and Results" coded presentations from the Nusa Dua Summit. These are attached (in the attachments at bottom of this post when opened online - see Click here ...). They are very rough and all comments are most welcome.

Greetings - below is the suggested agenda for the EVIDENCE and IMPACT Cluster meeting on Tuesday August 21st at 7am New York time. UNICEF will send the call in numbers. The Chair is Susan Krenn.

The Evidence and Impact Cluster thread is at this link:

From my records the participants in this Cluster are: 

Sonia Whitehead (BBC Media Action); Urvashi Gandhi (Breakthrough); Leena Sushant (Breakthrough); Marian Casey-Maslen (CDAC); Debora Freitas (The CORE Group); Anne Marie Hammer (GFMD); Radhika Gajjala (ICA); Bruce Girard (IAMCR); Tom Jacobson (IAMCR); Patrick Cook (ISMA); Doug Evans (ISMA); Susan Krenn (JHU CCP); Doug Storey (JHU CCP); Brett Davidson (Open Society Foundations); Antje Becker-Benton (Save the Children); Paula Valentine (Save the Children); Sue Goldstein and Sue (Soul City); Rafael Obregon (UNICEF); Ketan Chitnis (UNICEF); Hope Hempstone (USAID); Kama Garrison (USAID); Warren Feek - Secretariat (The Communication Initiative); Charita Bondanza - Adminstration (UNICEF). But we are sending this to all as everyone is most welcome!

AGENDA - Evidence and Impact Cluster - Global Alliance - Tuesday, August 21st, 2018 - 10 am New York time

1. Introductions - there may be people joining for the first time.

2. Evidence and Impact developments within your organisations - a round table provding people the opportunity to update others on any research, evaluation, evidence and impact developments within your organisations. 

2. Update and discussion: Alliance participating organizations survey - Global Alliance: Compelling Evidence and Impact Data - What do you use or quote? https://www.surveymonkey.com/r/GQSFHYS  (Warren)

3. Update and discussion: The long-term compelling impact data strategy and progress (Susan Krenn and Doug Storey)

4. Next steps - what is the plan of action including tasks and milestones moving forward on the Evidence and Cluster priorities?

As a starter for this discussion please see attached the Evidence and Impact Cluster section of the draft plan of action of the Alliance. This outlines some possible key actions for each quarter from Oct 1 2018 to Sept 30 2019. 

4. Any other items that people wish to raise.

Background documents:

Notes - Evidence and Impact Cluster meeting - June 20, 2018

Notes - Global Alliance Meeting - July 18, 2018

NOTES - Evidence and Impact Cluster - Global Alliance - Tuesday, August 21st, 2018 - 10 am New York time

Participating: Susan Krenn (JHU CCP - Chair), Sonia Whitehead (BBC Media Action), Rafael Obregon (UNICEF), Antje Becker-Benton (Save the Children), Paula Valentine (Save the Children), Warren Feek (Interim Secretariat - The CI)

1. Introductions - there may be people joining for the first time.

 
There were no new people present

2. Evidence and Impact developments within your organisations - a round table provding people the opportunity to update others on any research, evaluation, evidence and impact developments within your organisations.

Each organisation did a quick update on some of their research and evaluation work that could be relevant for the work of the Global Alliance. Unfortunately I did not take good notes on this. Can I suggest that you reply to this note and share in written form. Thanks and sorry! 

3. Update and discussion: Alliance participating organizations survey - Global Alliance: Compelling Evidence and Impact Data - What do you use or quote? https://www.surveymonkey.com/r/GQSFHYS  (Warren)

Warren provided a review of where we are at with this exercise: A number of organizations have responded; UNICEF was the most recent over the past weekend; Johns Hopkins had sent 15 by email - they are now paring down to the top 5 in their view and will submit soon; and the compilation process has commenced with the following features to support the analysis phase (a) full details of all submissions anonymized (sic) as promised so that who submitted what can not be identified (b) a list of titles of the research submitted with links to each of the individual items and (c) extracting and compiling the answers to Q6 “Please include the best extracts from the relevant paper, article, book (chapter) or other publication that highlight the evidence for the impact of a communication for development, social change, behaviour change, public engagement, informed citizen strategy on a development issue and priority” as this should provide the most compelling evidence so would be good to see in compiled form.

a. It was noted that this task has taken a lot longer than expected just to get the Alliance participating organisation’s responses ... and
b. ... that if organizations had more than 5 (UNICEF and JHU are examples) they should submit their top 5 and send the others to Warren by email as a ‘reserve” list.  
 

4. Update and discussion: The long-term compelling impact data strategy and progress (Susan Krenn and Doug Storey)

Doug was on vacation and Susan just back from vacation so Warren and Susan shared an update. This process is moving quickly.

a. The major emphasis has been on the key words/phrases for the search and filter process.

b. The SDGs have been reviewed and key words and phrases identified. This was a large task led by Urvashi and Warren, with Doug doing a final review and edit.

c. That process has now moved to a consideration of the key words and phrases that cover all aspects of our common field of work, making sure that we cover everything across the full spectrum of strategic activities that seek behaviour change, social change, informed societies, and engaged societies.

d. Decisions have been made concerning the research and evaluation scope - it will cover peer reviewed journals and grey literature.

e. The time frame has been decided - research and evaluation from 2005 and after (2005 being the Rome Summit)

f. In relation to geography there will be no restrictions.

g. Efforts will be made to include French and Spanish

h. When the keywords and phrases process is complete and agreed Johns Hopkins have very kindly made available a person from their staff with high skill in running these kinds of search processes. All expressed thanks.

i. UNICEF will also make someone available through their internship arrangement with New York University.

Overall there was support and congrats from the Cluster for the work being undertaken.

One issue raised was how to connect with the WHO Maternal and Neonatal Child Health evidence gap mapping initiative. Rafael reported that he was just back from a meeting of that group; Doug Storey was also present and chaired parts of the meeting. It was agreed that we should explore linking with them.

Task: Rafael was asked to set up a conference call with the relevant WHO people.    
   

5. Next steps - what is the plan of action including tasks and milestones moving forward on the Evidence and Cluster priorities?

As a starter for this discussion please see attached the Evidence and Impact Cluster section of the draft plan of action of the Alliance. This outlines some possible key actions for each quarter from Oct 1 2018 to Sept 30 2019.

For time reasons this was delayed. Everyone was asked to comment online.

Task: Warren to initiate the online review process of the plan of action

6. Any other items that people wish to raise.

Related to the planning issue above Susan raised the question of a possible close(r) relationship between the Evidence and Policy clusters. The basis outlined was that the evidence on which the Cluster is working would be the key material for the policy engagement processes by that Cluster. On this there was agreement.

Task: Rafael and Warren were asked to explore the best mechanism.

By email or online please provide any corrections or clarifications

To: Evidence and Impact Cluster - Global Alliance

FYI and also for comment - Other participants in the Alliance.

Best wishes. We now have sufficient responses to the survey Global Alliance: Compelling Evidence and Impact Data - What do you use or quote? to share the initial results.

Before that just a quick recap on the purpose of this work. We are seeking to establish a body of compelling research evidence and impact data that is agreed by the Alliance derived from each Alliance participating organization submitting the 5 most compelling pieces of research in their assessment. As a secondary element the research and evaluation submitted will help to check the search process for the longer term compelling evidence and impact work being undertaken by the Evidence and Impact Cluster. 

1. None of this work and these pages have been published. We are providing you with access to the unpublished pages. Obviously you will need to log in to see them. if this proves difficult please let me know. We may need to change your permissions.

2. Please do not worry at this time about the wrappping text on pages. This is very draft.  As these pages are unpublished it can be draft text. We will clean a lot of this up soon. 

3. Thirty-nine papers were submitted in total - They can be accessed at this link.

4. Interestingly there is only one duplication (see 14 and 15 in the list). So with that one exception everyone submitted different papers and studies. This may in itself be an interesting and instructive story about our field of work.

5. As promised none of the papers are identified with the organizations or people who submitted them - they are anonymous - as we needed a network contact for all of these (compulsory in the content type requirements) we put in my name but that can easily be changed of course.

6. If you click on a title you can open up the more detailed summary which is exactly as was submitted through the survey form (typos and all!). For example Community Dialogue to Shift Social Norms and Enable Family Planning: An Evaluation of the Family Planning Results Initiative in Kenya and Impact of the Integrated Radio Communication Project in Nepal, 1994-1997  

7. The CI is of course the acting secretariat to the Alliance - not a participating organization - but we did take the liberty of submitting. Just in case you wish to take those out of the equation they are numbers 11, 26, 28, 36 and 37 

8. In order for this process to be effective we will need to reduce these studies and papers to perhaps a one pager that contains the most compelling data. Please see this link where we have extracted and compiled everyone’s answers to Q6: “Please include the best extracts from the relevant paper, article, book (chapter) or other publication that highlight the evidence for the impact of a communication for development, social change, behaviour change, public engagement, informed citizen strategy on a development issue and priority.” 

9. At that link you can click the answer to Q6 and access the full submission to which that compelling piece of data relates.

10. As you will see that page is long. We have been faithful to what was submitted. The example answer for Q6 was “Compelling data extract and quote: (example) “Researchers found an overall effect size of 0.29. This translates into an 11.6 percentile gain (in terms of education). That is, an average child who does not watch Sesame Street is at the 50th percentile, whereas a child who watches is at the 62nd percentile.”  But most of the responses submitted were much longer - in some cases multiple long paragraphs.

11. As a result I have started a page that will seek to extract the key data in much shorter form. Will try to share that on Monday or Tuesday next week.

12. A couple of additional items about the individual submission pages - eg HIV Communication Programs, Condom Use at Sexual Debut, and HIV Infections. 

13. Where we already have a summary of the full paper (about 75% of the ones submitted in the survey I think) we have included a link to the summary - eg “Editor's note: Click here for a summary of the journal issue in which this article is included.” We have recoded that The CI summary for the Global Alliance page.

14. We did the Q 6 answer - most compelling extract - in larger font just to accentuate - eg here at this link

15. You will be able to search these with all sorts of filters. Of course they do not show up in the search process at present because they are not published.

16. Finally two organizations wanted to send a reserve list - one had 15 and the other 10 papers/studies to submit. The 15 have been received and we await the 10. These will be organized in an appropriate way soon.

At this stage please just review. All comments most welcome. Most happy to facilitate a discussion. Would be very interested in your observations.

Thanks - Warren

AGENDA - EVIDENCE and IMPACT Cluster - Tuesday September 18 2018 at 10am NY

Hi to eveeryone. Below is the suggested agenda for the EVIDENCE and IMPACT Cluster meeting on Tuesday September 18th 2018 at 10 am New York time. UNICEF will send the call in numbers. The Chair is Susan Krenn.

The Evidence and Impact Cluster thread is at this link:

From my records the participants in this Cluster are:

Sonia Whitehead (BBC Media Action); Urvashi Gandhi (Breakthrough); Leena Sushant (Breakthrough); Marian Casey-Maslen (CDAC); Debora Freitas (The CORE Group); Anne Marie Hammer (GFMD); Radhika Gajjala (ICA); Bruce Girard (IAMCR); Tom Jacobson (IAMCR); Patrick Cook (ISMA); Doug Evans (ISMA); Susan Krenn (JHU CCP); Doug Storey (JHU CCP); Brett Davidson (Open Society Foundations); Antje Becker-Benton (Save the Children); Paula Valentine (Save the Children); Sue Goldstein and Sue (Soul City); Rafael Obregon (UNICEF); Ketan Chitnis (UNICEF); Hope Hempstone (USAID); Kama Garrison (USAID); Warren Feek - Secretariat (The Communication Initiative); Charita Bondanza - Adminstration (UNICEF). But we are sending this to all as everyone is most welcome!

AGENDA - Evidence and Impact Cluster - Global Alliance - Tuesday, September 21st, 2018 - 10 am New York time

1. Introductions - there may be people joining for the first time.

2. Evidence and Impact developments within your organisations - a further round table provding people the opportunity to update others on any research, evaluation, evidence and impact developments within your organisations. 

3. Update and discussion: Alliance participating organizations survey - Global Alliance: Compelling Evidence and Impact Data - What do you use or quote? https://www.surveymonkey.com/r/GQSFHYS  (Warren)

The main items with relevant links embedded on this item include (note that these are not published; we have provided you with access when logged in). Please also see relevant threads here Initial Results - Alliance Survey - Top 5 Impact Data and Sharpening the impact data from the survey

: 

a. The compiled list of research and evaluation impact papers as submitted by some Alliance orgs as their top 5 - Observations, Comments and Questions

b. The compiled answers to Q6: “Please include the best extracts from the relevant paper, article, book (chapter) or other publication that highlight the evidence for the impact of a communication for development, social change, behaviour change, public engagement, informed citizen strategy on a development issue and priority.” This is everything submitted in the survey with no edits. Observations, Comments and Questions.

c.  Warren has made a very initial attempt to take the answers to Q6 and reduce them to very short statements that are more  "digestible" for both people in this field of work and policy makers, governments and foundations, for example, looking for a one pager on the evidence for the impact of this filed of work. he stresses this is a very early attempt. There are lots of issues. It is presented for your review, comment and guidance at this link.

d. Next steps: Overall guidance and decisions on next steps for this initiative.

4. The long-term compelling impact data strategy and progress (Susan Krenn and Doug Storey)

Some of the ley areas to be considered:

a. Completion of SDG related search terms

b. Completion of search terms that comprehensively cover the various strrategic strands of this field of work.

c. Decisions concerning the scope of the initial test process - for example oen or two of the SDGs

d. Planning for the search design process - JHU engagement of an information scientist and UNCEF support through a post graduate student(s)

e. Other items

5. Plan of action - October 1 2018 to September 30, 2019

In our last meeting we deferred considerationo of this item until this meeting so that everyine had tme to consider. We have reattached to this agenda (see at bottom when you open the Please click to review, comment, etc link above and view online). Please see the Evidence and Impact section. This outlines some possible key actions for each quarter from Oct 1 2018 to Sept 30 2019. 

Some key questions:

a. In general does the content of the cells (A21 to E29) make sense?

b. What do we need to add?

c. What do we need to delete?

d. Do any of the blcoks need moving - for example from October-December quarter to January-March quarter?

e. Other comments and ideas

6. Any other items that people wish to raise.

Background documents:

Notes - Evidence and Impact Cluster meeting - August 21, 2018

Notes - Global Alliance Meeting - August 30, 2018

Background: This meeting was seriously disrupted by the technial issues Unicef experienced with Skype for Business. Many people were unable to join. The meeting started late with Doug Storey (Johns Hopkins), Urvashi Gandhi (Breakthrough), Patrick Cook (ISMA) and Charlotte Lapsansky (Unicef) those who managed to get on the call. They decided to go ahead and others trying to connect indciated through the Whats App Group that they would contribute online. As a result of the delay the agenda was truncated to item 3. Alliance Participating Organisations Top 5 survey; item 4. Long term compelling evidence process; and, a brief look at item 5 Plan of Action Oct 1 2018 to Sept 30 2019. 

Notes:

Agenda item 3. Update and discussion: Alliance participating organizations survey
- link to survey itself is here

Warren recapped the purpose of this initiative: 

a. Responds to the commonly asked question of our field - where is the impact evidence data?

b. Responds to the important requirement for evidence to inform strategies and programmes. 

b. Supports this field of work by providing a short evidence document with links to full papers, agreed by a number of organisations (those in the Alliance) so not just the views of one organization, very short for quick comprehension, etc that can be used to respond to those two imperatives.

Warren moved to outline the progress to date. He stressed that these were all still, by design, in very rough draft form - the raw material distilled to date. Key links included:

The compiled list of research and evaluation impact papers as submitted by some Alliance orgs as their top 5  

The compiled answers to Q6: “Please include the best extracts from the relevant paper, article, book (chapter) or other publication that highlight the evidence for the impact of a communication for development, social change, behaviour change, public engagement, informed citizen strategy on a development issue and priority.”

Reduction of the answer to Q6 to very short statements that are more  "digestible" for both people in this field of work and policy makers, governments and foundations. Presented for review, comment and guidance at this link.

Feedback was sought from Cluster participants on the work completed to date and next steps to take. Those responses included:

Found the data difficult to digest in this very raw form.

Wonder if it is possible to frame the data in a statement for someone who is trying to advocate for SBCC, for example, to develop a narrative around the data from that perspective.

Perhaps outline the take away from each piece of data presented - for example the demonstrated link between modern family planning and spousal communication and cars. Would need a short, sharp narrative around these elements.

Focus on the take away from the evidence data shared. This is perhaps the most important thing and will indicate which tools and strategies to use.

Seek to package this data in a way that people can insert and use to strengthen their funding proposals.  

Lead with the conclusion and thenback that up with the most important statistic

Join the dots through an overarching argument for the effectiveness of this field of work that weaves in the data shared by the Alliance organizations.

A question was asked about about qualitative data as most of what was submitted through the survey wwas quantitative in nature. This is something to explore.

Action: Warren to review above and develop a further iteration of the data taking into account the comments above. Then to submit to this Cluster for further review. 

Agenda item 4. The long-term compelling impact data strategy and progress 

Doug Storey provided an update on progress:

a. Further work has taken place to develop the search platform that will be used to distill the most compelling evidence and impact data for this field of work from a comprehensive and exhaustive literature review.

b.The work to distill the key search words and terms from the Sustainable Development Goals and targets has been completed. See file "Global SBCC Evidence Review Search Words and Phrases Sept 19" attached when you open this note online. Doug confirmed that his edits in the SDGs section of the document followed the principle of only using words and phrases that were in the relevant SDG text. 

c. Initial work has been undertaken to identify the key words and phrases that will be the search terms for identifying appropriate research from that perspective. Participants in the Cluster were asked to review Table 4 of this attachment "Global SBCC Evidence Review Search Words and Phrases Sept 19" when you open online and submitted recommended edits. This will allow us to ensure we are covering all relevant strategies and change principles for our field of work.

d. Doug reported he has a meeting this week with the Knowledge Management specialist at Johns Hopkins (apologies missed the name) who will develop the seach boolean design.When a draft deisgn is complete it will be shared with the Evidence and Impact Cluster for review and comment. These comments will be required by September 25th 2018. 

Agenda item 5.  Plan of action - October 1 2018 to September 30, 2019

There was no time to review the Evidence and Impact elements of the draft plan - see attached when you open these notes online. Cluster members were asked to provide written comments by Wednwsday September 26th, 2018.

Tasks and schedule:

A. Warren to do further iterations of the Top 5 data for review by the Cluster - October 5, 2018

B. Cluster participants to provide edits for Table 4 of the "Global SBCC Evidence Review Search Words and Phrases Sept 19" attached - September 25th 2019

C. Cluster participants to review and provide inputs on the draft plan of action for 18-19 (attached when these notes are opened online) - September 26th, 2018

Greetings Friend,

Please find attached (on the platform when you open this email online) our current list of SBCC Approaches search terms that will go into the search algorithm for the Evidence Landscaping Initiative.

 
This is the product of a series of conversations and revisions among members of the Evidence and Impact Cluster over the past few months. We have been trying to come up with the most comprehensive list of terms possible to capture all the various ways that SBCC approaches might be described in the global literature (both peer-reviewed and grey), so that we don’t inadvertently miss any social and behavior change work that will shed light on what works for what and why.

 
A couple of our information specialists here at CCP (from the Knowledge for Health project) are working on converting this list into a search algorithm with Boolean operators (e.g., “X” and/or “Y” and “Z”, etc.) that will attempt to capture the possible combinations of these terms.

 
Please don’t worry too much at this stage about how these terms will be combined. That will be the next thing to share with you. At this point, it would be helpful if you would let me know if there are any key nouns or adjectives that are missing from this list and are essential referents to ways of doing SBCC.

 
Once we are happy with the terms, we can test run the algorithm any number of times to confirm that we are searching for and getting everything we think is relevant.

 
Thanks very much. I look forward to your feedback.

 
Doug

PS - If you have difficulty opening this email and the online attachment please send a message by return email - thanks.
  

Hi to everyone. Below is the suggested agenda for the EVIDENCE and IMPACT Cluster meeting on Tuesday October 16th 2018 at 10 am New York time. UNICEF (Charita) has sent the call in numbers. The Chair is Susan Krenn.

The Evidence and Impact Cluster thread is at this link  The notes from the previous Evidence and Impact Cluster meeting are at this link. From my records the participants in this Cluster are:

Sonia Whitehead (BBC Media Action); Urvashi Gandhi (Breakthrough); Leena Sushant (Breakthrough); Marian Casey-Maslen (CDAC); Debora Freitas (The CORE Group); Anne Marie Hammer (GFMD); Radhika Gajjala (ICA); Bruce Girard (IAMCR); Tom Jacobson (IAMCR); Patrick Cook (ISMA); Doug Evans (ISMA); Susan Krenn (JHU CCP); Doug Storey (JHU CCP); Brett Davidson (Open Society Foundations); Antje Becker-Benton (Save the Children); Paula Valentine (Save the Children); Sue Goldstein and Sue (Soul City); Rafael Obregon (UNICEF); Ketan Chitnis (UNICEF); Hope Hempstone (USAID); Kama Garrison (USAID); Warren Feek - Secretariat (The Communication Initiative); Charita Bondanza - Adminstration (UNICEF). But we are sending this to all as everyone is most welcome!

AGENDA - Evidence and Impact Cluster - Global Alliance - Tuesday, October 16th, 2018 - 10 am New York time

1. Introductions - there may be people joining for the first time.

2. Evidence and Impact developments within your organisations - a further round table providing people the opportunity to update others on any research, evaluation, evidence and impact developments within your organisations. 

3. Update and discussion: Alliance participating organizations survey - Global Alliance: Compelling Evidence and Impact Data - What do you use or quote? https://www.surveymonkey.com/r/GQSFHYS  (Warren)

The main items with relevant links embedded on this item include (note that these are not published; we have provided you with access when logged in). Please also see relevant threads here Initial Results - Alliance Survey - Top 5 Impact Data and Sharpening the impact data from the survey

Issue for discussion:

At the last meeting Warren was asked to take the basic evidence and impact data submitted by organisations engaged in the Alliance and develop a narrative around that data. He would like further guidance in order to complete that task.

4. The long-term compelling impact data strategy and progress (Susan Krenn and Doug Storey)

Issue for discussion:

Perhaps the most crucial part of this process is that the seaarch terms and phrases used to identfy relevant impact data accurately and comprehensively reflect and cover the main strategic approaches in the field of work. 

Following work mainly by Doug with input from Warren please see the attachment related to the post from Doug at this link. Since then there were a couple of further ideas from Warren and a comprehensive set of suggestions from Antje.  

The Cluster is asked to review the proposed set of strategic approaches search terms and phrases (Doug initial document also  attached.)

5. Any other items that people wish to raise.

Background documents:

Notes - Evidence and Impact Cluster meeting - September 18, 2018

Notes - Global Alliance Meeting - October 4th, 2018

Participating: Susan Krenn (Chair - JHU), Urvashi Gandhi (Breakthrough), Debora Freitas (The CORE Group), Rafael Obregon (Unicef), Sonia Whitehead (BBC Media Action), Warren Feek  (interim Secretariat), Antje Becker-Benton (Save the Children). (Others may have joined - please let me know.)

1. Introductions - there may be people joining for the first time.

There was no-one new joining.

2. Evidence and Impact developments within your organisations - a further round table providing people the opportunity to update others on any research, evaluation, evidence and impact developments within your organisations.

No-one had anything new to report since the previous meeting of this Cluster. 

3. Paper based on the Top 5 evidence and impacted data survey amongst the participating organizations in the Alliance.

At the prior meeting of this Cluster Warren had presented the raw outputs from the survey - see Initial Results and the initial attempt to reduce that data to its key elements  He was asked to provide a narrative around that data in order to strengthen the narrative that makes the case for our field of work. Warren communicated that he had started this process but needed further guidance and posed 3 questions:

Q1. Who is the priority group we are seeking to engage?

Q2. What do we want them to do?

Q3. Should the paper be comprehensive and relevant to all development issues or should we do a paper that focuses on for example the main themes of HLPF 19 - “Empowering people and ensuring inclusiveness and equality”

Following discussion Warren was asked to do the following:

a. The guidance was to construct the paper, style, focus, argument, selection of data, on policy makers (government, UN, Foundations, NGO organisation management, academics) who are not part of the social change, behaviour change, informed societies and engaged societies field of work.

b. Provide a document that they can introduce into their decision-making deliberations and decisions.

c. Related to whether the paper should be generic or focused on some particular priorities such as the theme of HLPF 2019, Warren was asked to see if he could do both. There was aHe said he would try.

4. The long-term compelling impact data strategy and progress (Susan Krenn and Doug Storey)

Susan and Warren (in Doug’s absence) updated the Cluster on progress with the longer-term process to identify the most compelling evidence and impact for our common field of work. Since the last meeting the final element required for this search process has been put in place with the comprehensive identification and listing of the terms and phrases that cover all strategic approaches for social change, behaviour change, informed societies and engaged societies. Doug shared that list with the full Alliance in his note of October 9th (see attachment) Further ideas had already been submitted - for example from Warren (based on a discussion with OSF) and Antje introducing the Save the Children Fund terms and phrases.

An information specialist in JHU is now developing the boolean terms based on the geography, SDG and strategi approach terms identified. This work has commenced. The draft boolean configuration will be made available to the Cluster for review. When the boolean is completed there will be a test process related to, for example, a couple of SDGs. The plan is to have that completed by the end of 2018.     

Action: Members of the Cluster were asked to review the attachment at Doug’s post - SBCC Approach - Search terms  and provide final inputs by Monday 22nd October.  

To: The Evidence and Impact Cluster - The Global Alliance
cc: Others in the Alliance

Hi to everyone. As part of The CI’s role when it was the interim Secretariat we had promised a narrative paper that drew together the data submitted by Alliance participating organizations through the Top 5 impact data survey. I have completed a very initial draft. Please note that it is very rough. You can open under Attachments when you view this post online. But before you open; all of the caveats!

1. It is not complete - needs loads of work - but I do want to get overall feedback as to the direction being taken and the style being used.

2. Please  do not worry about grammar and edits etc - all of that can be cleaned up later.

3. The paper is not for us. It is for policy makers, government officials, senior UN people, folks in funding agencies, etc - people not in our field of work. So there is not a load of nuance, not many caveats and an absence of detail.

4. The tone is brief (I promised myself I would try to keep this under two pages and only just failed!); straight forward and factual.

5. it is designed to (a) inform about the results of our work (b) position our field as having demonstrated impact; and, (c) prompt a relationship and dialogue.

6. I have focused this around HLPF 2019. All efforts to do a generic piece did not really work. I found that I needed to have an anchor in mind. And HLPF 2019 is a natural for us. When we engage in other fora the raw material is there in the survey results to be used as appropriate.

7. I need to do a thorough review of all data submitted in the survey and make sure that we are capturing everything relevant - so the present inclusions are temporary. 

8. Finally I opted out of drafting the strategic implications section. It just seemed something we should discuss and agree.

OK - enough excuses! Attached for your review when opened online - see Attachments under this post.

Thanks

Warrem
 

Hi - unfortunately I was a little late to the Evidence and Impact call this morning so missed the round of updates. Though Susan gave me an opportunity to share I did not want to delay our focus on the substance of the call. Though The CI is not part of the Alliance, Rafael has asked me to sit in for the initial round of Cluster calls. So, quickly a couple of updates: 

1. The CI platform based on submissions from the network continues to add Evidence and Impact related shared and summarized knowledge. Overall you can search and filter at this link (3,990) with the 2018 batch at this link (69). The overall Impact data summaries are here (1,629) and those for 2018 are at this link. You can filter according to a whole range of taxonomy terms. 

2. Sonia mentioned the Soul Buddyz study as “one to watch”. In case helpful there is a summary of the information we have received on this study to date from Sue Goldstein at this link

3. I know that there is a lot of interest at present in some quarters in behavioural economics including approaches like nudge theory. A network participant sent me a link to this paper “Nudge Fudge Leaves Policy Makers in the Dark”  Three things here (a) an interesting approach - they took an OECD collection of submissions from governments as the universe to examine - 111 case studies (b) a very concise, clear summary style that could be a model for us and (c) the results are not that positive - for example “16 percent did not qualify as psychological and behavioral economic methods. 58 percent were not found to have the anticipated effect on behavior. Eighteen percent of the total were later implemented by governments” and “No details about the cost of these initiatives were given, where the interventions were positive, there is no information on the scale of the impact.” But this is of course just one paper. Perhaps there is other data? If there is and you have access would be great if you could send. 

Thanks for indulging this written update - Warren

Impact Data in Narrative form

Draft paper attached when viewed on platform

Hi folks. A very happy new year to all.

When The CI was acting as the interim secretariat for the Alliance there was an initiative asking each participating organisation to identify the top 5 pieces of impact data (could be their own or others). These were submitted from many Alliance organizations and then compiled into this list here with links.

In the responses to this list there was a request for a narrative within which this data could be presented and explained - eg the strategic implications. Implied in those requests was that there could be one overview or general narrative. But for two reasons I found this really difficult:

1. The purpose in collecting the data was not to develop a one size fits all document. It was to provide a filtered and selected list of impact data that people could use and quote when developing their own funding proposals, advocacy papers, reviewing their programmes, etc. So they could for example say something like: "It has been demonstrated that our work is effective and has impact. See for example this compelling research data reviewed and selected by the Global Alliance - etc.

2. Every context is different. It is really hard to write a narrative that will cover even a reasonable range of contexts and purposes.

So, after much toing, froing and playing I have done the following:

a. Tried to get a two-for-one for the Alliance. I understand that there will be a focus on HLF 2019. So I have tried to position the data collected relative to the theme of HLPF 2019. The draft paper attached is in the form of a position paper for submission to that forum. 

b. In this way the paper attached can be used as both an example of a potential way to use the research data and as an initial input for the planning around HLPF 2019.

c. In order to get the full approach I tried to assess what may be required for HLPF 2019 - for example a (hopefully) engaging opening that reflects our field and illuminates the issue; very specific and succinct list of impact data, placed in the context of the HLPF 2019 and using the language of that event; etc.  

d. The recommendations are proposals only - they seek to reflect the data gathered and distilled of course and where appropriate to build on the Summit Statement. It will be important of course for the Alliance to work out the "ask" related of the HLPF. 

e. I have not yet had our editors take a crack at this - so it is in its raw form. Will do that soon. It is a little longer than I wanted - just at 5 pages. But a good edit should fix that.

Attached is for your review - thanks - Warren

To - The Evidence and Impact Cluster

Next draft: Evidence impact data and HLPF advocacy and positioning paper.

Hi folks and many thanks for all of the comments on the previous draft of this document shared both in the most recent cluster meeting and from some of you by email and track changes.

I have been plugging away at this next draft since we all chatted in the Cluster meeting and completed it today after receiving some final written comments.

It was my intention in this note to go through everyone's comments and explain how I took them into account. But I hope you will excuse me if I do not do that!

Instead please do review attached when you open this note online - click on "Please click to review, comment and access any attachments" above and scroll down to Attachments below this note. 

Best wishes and thanks - Warren