Social Service Policy Evaluation Using High-Fidelity Client Emulation
•
1 like•141 views
Problem Definition • Motivation: Policy makers in the area of social service delivery do not have good tools for evaluating the effectiveness of alternative programs before they become policies. • Goal: Develop a high-fidelity homeless person emulation that can be used in a simulation environment to evaluate social service policies.
Recommended
Recommended
More Related Content
Similar to Social Service Policy Evaluation Using High-Fidelity Client Emulation
Similar to Social Service Policy Evaluation Using High-Fidelity Client Emulation (20)
Recently uploaded
Recently uploaded (20)
Social Service Policy Evaluation Using High-Fidelity Client Emulation
- 2. Outline • Problem Definition • Background • Approach • Experiments 08-28-2017 Social Service Policy Evaluation Using Client Emulation 2
- 3. Problem Definition • Motivation • Policy makers in the area of social service delivery do not have good tools for evaluating the effectiveness of alternative programs before they become policies. • Goal: • Develop a high-fidelity homeless person emulation that can be used in a simulation environment to evaluate social service policies. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 3
- 4. Problem Definition 08-28-2017 Social Service Policy Evaluation Using Client Emulation 4 Why focus on the homeless? • Canada 2016 • 35,000 homeless Canadians on any given night • (+17.5% from 2014) • 27% women, 19% youth, 24% aged 50+ • India 2011 • 1% estimated homeless in cities • Mumbai: 200,000 (including Navi Mumbai) • Delhi: 150,000 - 200,000 • Kolkata: 150,000 • Ahmedabad: 100,000 • Hyderabad: 60,000 • USA 2015 • 576,450 homeless Americans on any given night • (-2% from 2014) • Varies greatly from state to state. • Requires tailored, client-centric policy. http://endhomelessness.org http://homelesshub.ca/SOHC2016 http://hlrn.org.in/homelessness
- 5. Problem Definition: Policy Evaluation • Key Insights about the homeless population: • Often seen through the filter of social norms. • Face different limitations than the rest of the population in their society, and live by different social norms. • Life choices seem irrational, incompatible with society, and detrimental to their own wellbeing. • Traditionally: • Large scale simulations try to close the gap between program trials and implementation. • Probabilistic models are based on decisions made under past policies. • Social Science models rely on social norms and structural factors. • Limitations: • Need to know how clients will react in the future under new policies, not past policies. • Social science models are not always applicable due to different social norms. • High-fidelity agent: • Capable of emulating seemingly “irrational” behaviour. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 5
- 6. Background • What are the models of decision making? 1. Rational view according to decision theory is based on an objective understanding of choices. • Economists and AI focus on understanding the process of decision-making (Etzioni, 1988; Russell, 1997). • Any factors that impact utility maximization. 2. Behavioural view is based on a subjective understanding of choices. • Psychologists and sociologists focus on the interpretation of observed behaviour (Simon, 1967; Simon, 1996; Klymchuk, 2014; Etzioni, 1988). • Any factor that can explain the observed behaviour. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 6
- 7. Approach • Merged view of decision making [Gajderowicz, 2017a]: 1. Begin with the reasoning view. • AI planner to emulate client choices. 2. Extend reasoning view with factors that change utility ( ∆U ) of different actions. • ∆U : Basic human needs and emotional states. 3. Incorporate the behavioral view: • Calibrate the model using data about client decisions from a pilot study. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 7
- 8. Scenario: “John” • Imagine John who meets with a social worker. • John wants to exit homelessness, but has had a hard time finding a place that meet his needs: • Close to favourite soup kitchen. • In neighbourhood to existing shelter. • Close to community centre to visit friends. • Close to store to make minor purchases. • On the 2nd or 3rd floor. • Must face east. • The social worker sets out a plan for John: 1. Move into an apartment that is available. 2. Apartment is on the 2nd floor but faces north. 3. Close to his friends, but not the soup kitchen. 4. Food can be delivered in the first month. 5. After one month John can get a food stipend and go to a local grocery store. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 8
- 9. Rational Reasoning 08-28-2017 sh st sk cc st sh cc st sh sk sk sh cc st cc st st sh st sk cc . . . cc . . . . . . . . . . . . . . . . . . . . . . . . sh sh cc sh sh cc sh sh . . . sk . . . . . . . . . . . . . . . . . . . . . . . . • starting state “at shelter” and • services believed to satisfy goals: “state => goal(s)” food friends clothing + food start at shelter end at home Agent Beliefs sh => security + home sk => food cc => friends st => clothing + food Initial state ( S ): at shelter (sh) Social Service Policy Evaluation Using Client Emulation 9 Search tree for sequencing and searching states that satisfy an agent’s goals. • Each state represents use of a service: shelter (sh), soup kitchen (sk), community centre (cc), or store (st) • Blue state is part of client’s chosen path. • Green state is part of SW’s path. U 3 U 4 U 6 U 5 Goals ( G ): - food - security - friends - clothing - end at home max U P : G U 2 U 1
- 10. Bounded Rational Reasoning 08-28-2017 I bound : missing or wrong information e.g. food not available in soup kitchen, only at store and community centre. sh st sk cc st sh cc st sh sk sk sh cc st cc st st sh st sk cc . . . cc . . . . . . . . . . . . . . . . . . . . . . . . sh sh cc sh sh cc sh sh . . . sk . . . . . . . . . . . . . . . . . . . . . . . . friends + food Social Service Policy Evaluation Using Client Emulation 10 max U P : G U 2 U 4 Result: Prune branches that don’t match known information. Adjust utility of remaining paths. Exclude correct plans with soup kitchen. U 3 U 1
- 11. sh st sk cc cc sh cc sh sh . . . sk . . . . . . . . . . . . st cc st sh sh cc. . . . . . . . . Bounded Rational Reasoning 08-28-2017 sh st sk cc st sh cc st sh sk sk sh cc st sh. . . G C bound : limited cognition e.g. limit depth of search to 2 levels. Result: Prune branches to available depth. Adjust utility of remaining paths. Solve sub-problem to optimality. Social Service Policy Evaluation Using Client Emulation 11 max U P : G U 1 U 4 U 2 U 3
- 12. Bounded Rational Reasoning 08-28-2017 T bound : limited time e.g. can only check the first 18 of state Result: Prune branches beyond the time boundary. Finds only locally optimal solution. sh st sk cc st sh cc st sh sk sk sh cc st cc st sh st sk cc . . . cc . . . . . . . . . . . . . . . . . . . . . . . . sh sh cc sh cc sh sh . . . sk . . . . . . . . . . . . . . . . . . . . . . . . Social Service Policy Evaluation Using Client Emulation 12 st sh max U P : G U 1 U 4 U 2 U 3
- 13. Bounded Rational Reasoning + Human Needs 08-28-2017 ∆U : Utility changes due to Maslow’s Needs (MH) • aggregate goals to MH needs Result: Utility changes based on changing MH. st sk cc st sh cc st sh sk sk sh cc st cc st st sh st sk cc . . . cc . . . . . . . . . . . . . . . . . . . . . . . . sh sh cc sh sh cc sh sh . . . sk . . . . . . . . . . . . . . . . . . . . . . . . MH Needs + Interim Goals: 1) physiological (food) 2) safety (security) 3) social (friends) 4) esteem (friends) 5) self-act (clothing) 6) end at safety (home) phys safety end at safety phys + esteem Social Service Policy Evaluation Using Client Emulation 13 sh max U P : G Self Actualization Esteem Social Safety Physiological U 6 U 5 U 3 U 1 U 4 U 2 U 7
- 14. Approach • Emotions ??? • Appraisal theory, arousal theory, etc: need to know before hand. • Expectation of success where |G S | = total satisfied goals and |G| = all goals. • Example: expectation of success when goals do not change (left) or goals increase over time (right). • Neither captures expectation change describes in behaviour psychology literature. Expectation of Success |G S | Expectation without goal growth 08-28-2017 Social Service Policy Evaluation Using Client Emulation 14 |G|0 100% Expectation of Success |G S | Expectation with initially exponential goal growth |G|0 100% |G S | |G |
- 15. Approach 08-28-2017 Social Service Policy Evaluation Using Client Emulation 15 • Emotional Cycle of Change (ECOC) • Behavior change according to ECOC theory (Kelly, 1979). • ECOC defines how expectation of success changes when information about goals becomes available. • Below emotional thresholds, goals are removed and reordered (∆G) as expectations change (Gajderowicz, 2017b). Mood Time Emotional Cycle of Change (ECOC) 1) Uninformed Optimism 2) Informed Pessimism 3) Valley of Despair 4) Hopeful Realism 5) Informed Optimism 6) Success 100% |G S | |G | ( )P(E) = ecoc P(E)
- 16. Bounded Rational Reasoning + Emotions 08-28-2017 sh st sk cc st sh cc st sh sk sk sh cc st cc st st sh st sk cc . . . cc . . . . . . . . . . . . . . . . . . . . . . . . sh sh cc sh sh cc sh sh . . . sk . . . . . . . . . . . . . . . . . . . . . . . . Social Service Policy Evaluation Using Client Emulation 16 U 6 U 5 max U P : G ∆U + ∆G : Utility changes impact goal changes • Change expectation based on ECOC U 3 Result: Change focus between different goals (G). Remove branches that have a reduced utility after ∆U Restore branches that have increased utility after ∆U U 1 U 4 U 2 U 7
- 17. Experiments 1. Objective: Predict client outcomes in intervention program. • Hypothesis: Client MH needs and/or ECOC stages have equal or better predictive power than currently used methods the use client demographics at intake. • Methods: 1.1. Recurrent Neural Network (RNN) + MH Needs => Client outcomes 1.2. Support vector machine (SVM) + MH Needs + ECOC => Client outcomes 2. Objective: Match emulated (∆G) to real (∆G). • Hypothesis: Using agent demographics and model scores, we can create cohorts that differentiate between different groups with statistical significant. • Method: Agent-based simulation (ABS) • STRIPS Planning + BR + MH + ECOC => Emulate ∆G 08-28-2017 Social Service Policy Evaluation Using Client Emulation 17
- 18. Experiment 1.x Series Design • What client needs map to MH needs? • Partnered with Calgary Homeless Foundation (CHF) to evaluate their Housing First (HF) intervention program. • Client needs captured by CHF SPDAT forms at 3 month intervals. • Each need was mapped to a level of Maslow’s Hierarchy and client status. • Example: • Mean of total Esteem-level needs for absolutely homeless versus relatively homeless clients at 3-month intervals. • Clients who exited the program at 9 months. • Clients are either missing, are successful, or failed in the program. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 18
- 19. Experiment 1.1 (RNN) Design When can you predict client success in intervention program? • Objective: • Evaluate predictive power of changing client needs (∆G) and demographics. • Hypothesis: • Client need transitions plus demographics can be used to predictive client status at end of study. • Method: • Recurrent Neural Network (RNN) is used to predict client outcomes in an intervention program with temporal (time-series) data. • Based on total needs per MH level. • Dependent Variable: • Missing, success or failure in the program. • Independent Variables: • Client demographics: top-11 as per p–value and key demographics. • Client need trajectories (∆G) mapped to Maslow’s Hierarchy. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 19
- 20. 08-28-2017 20 RNN prediction score is accuracy (y-axis): Analysis • Increase in accuracy from 3 to 6 months. • Higher standard deviation for mean of needs at 6 months. • Individual demographics have good predictive power (>= 0.75) after 6 months. • Best demographics: o Attended Mental Facility, o Relatively homeless, o Attended Health Facility, o Employment Status, o Duration of Unemployment. Social Service Policy Evaluation Using Client Emulation Experiment 1.1 Results Prediction Score = TP + TN P + N Conclusion • Hypothesis is proven true: By considering certain demographics and changes in MH needs, it is possible to predict client outcome in the HF intervention program. • Changing MH needs (∆G) are a valid predictive measure.
- 21. Experiment 1.2 (SVM) Design Predict client success in intervention program. • Objective: • Evaluate predictive power of client needs and emotional state against existing methods. • Hypothesis: • Client needs and ECOC stages have equal or better predictive power than currently used methods that rely on client demographics at intake. • Method: • Support vector machine (SVM) is used to predict client outcomes in an intervention program. • Based on total needs per MH level and ECOC state machine. • Dependent Variable: • Success or failure in the program. • Independent Variables: • Client demographics: all; mental health, top-2 (employment and mental facility); and key-3 (age, mental health, absolutely/relatively homeless). • Client need trajectories mapped to Maslow’s Hierarchy (omitting self-actualization). • ECOC state machine and calibrated weights. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 21
- 22. ML Counts Only Best with Demographics ECOC + Mental Issues Configuration Top Demographics, MH, and ECOC Stages 08-28-2017 22 SVM prediction score is accuracy (y-axis): Analysis • Demographics (“demo”) o top-2 demographics had 0.69 accuracy • Maslow’s Hierarchy of Needs o Without demographics, MH needs had 0.67 accuracy. o Adding demographics improved the accuracy to 0.72. • Simulated ECOC stages with state machine and calibrated weights. o ECOC levels only had small improvement in accuracy. Weighted score with Mental Health had best accuracy at 0.76 Social Service Policy Evaluation Using Client Emulation Experiment 1.2 Results Prediction Score = TP + TN P + N Conclusion • Just relying on ML counts gives good predictions compared to demographics. • Simulated ECOC stages with calibrated weights give 9% improvement.
- 23. Experiment 2 (STRIPS) Design Client Emulation (Gajderowicz, 2017b; Gajderowicz 2017c) • Objective: • Evaluate how well an agent model emulates changes in homeless client needs (∆G) while participating in the Housing First program administered by CHF. • Hypothesis: • Using agent demographics and model scores, we can create cohorts that differentiate between different groups with statistical significant. • Method: Agent Based Simulation: • The agent creates a plan based on their belief of the state of the world, their goal ranking, and expectation of success, and re-planning during execution. • Calculate difference between emulated (∆G) and real (∆G) for CHF clients. • Two-way ANOVA on Taken Period and Demographic to calculate variance in model scores. • Model: M = { goals(G), searchS, ecocTh, actionTh }. • Dependent Variable: • Model’s score for emulating ∆G : • Independent Variables: • Taken Period, Demographics. 08-28-2017 Social Service Policy Evaluation Using Client Emulation 23
- 25. Experiment 2 Results 08-28-2017 Social Service Policy Evaluation Using Client Emulation 25 • Analysis: • Overall metric rating. • P (sM | F, d) and P (F | sM, d) are the best metrics for assigning models M to specific demographics. • Top rated metrics by demographic. • Based on these results, the best metric with highest mean probability is P(sM|F,d) with 0.905. • This means that there is a 90.1% probability of finding a good model for a given demographic that exits the program at month F. Demographics • Conclusion: • We reject the null hypothesis and say that Taken Period and model score have an effect on model score. • We can say that these is a relationship between Taken Period, client demographics and the emulated (∆G).