What Human Services Should Know About Self-Reported Data from Clients

Human services data management is a hard job, partly because people are hard to observe accurately. When we know we’re being measured, we change our behavior. This is known as the Hawthorne Effect.

Between 1924 and 1927, a famous series of experiments became a case study for why it’s so hard to measure natural human behavior. In Hawthorne, Illinois, plant managers decided to test whether increased lighting for their workers would increase productivity. After a period of testing, their workers reported it did! But then, after lowering the lighting to see if that made any changes, productivity again appeared to increase. Also, other minor changes tended to have an effect of productivity, such as cleaning work stations more regularly, but then again, so did returning to the previous methods. Years later, this experiment became known as the Hawthorne Effect.

The gist is this: when people are observed, they tend to behave differently than if they weren’t observed—and change more dramatically when their output is being measured.

What can human service organizations learn from this phenomenon and how it relates to client data? Why should data managers care in the first place? Well, it’s because people are hard to accurately observe, and data quality remains a constant struggle for health and human services.

The Biases of Self-Reported Data

It’s often said that the plural of anecdotes is not data. Perhaps a more nuanced take on the phrase could be: conclusions should not be drawn from anecdotes alone. Because, it turns out, anecdotes (i.e., self-reported individual data) are the primary kind of data we receive at client intake, whether for HMIS, domestic violence shelter, refugee services, etc. So there’s a sense in which it’s possible to derive useful data from multiple anecdotes.

For example, if a client during intake insists his tenant rights have been violated by a landlord (i.e., privacy, property, unfair fees), who is threatening him with eviction, and he firmly insists on this fact in multiple interactions with a service provider, what conclusions can be drawn? There might be multiple ways to interpret this “anecdata”:

1. He could be accurately describing the incidents, correctly characterizing the interactions with the landlord;
2. He could be unintentionally exaggerating, having honestly acquired a hair-trigger response due to past unfavorable interactions (an example of subjective perception bias);
3. He could be intentionally exaggerating or deceiving, with the hope of adversely affecting the landlord or receiving services (an example of social desirability bias);
4. He could be tired of answering the same questions repeatedly and instead gives an oversimplified account of the incident or aggravating factors (an example of reporting fatigue); or,
5. He could be misremembering or conflating, especially if he’s experiencing a severe mental illness or was under the influence at the time of the incidents in question (an example of recall bias).

All these above interpretations are entirely possible, but clients are also prone to self-report with bias.

We want to give clients the benefit of the doubt, but let’s say a community service network provides eviction prevention services to people at risk of homelessness. Should the above client be referred to legal services for eviction prevention immediately? If the allegations against the landlord are egregious and inaction would outweigh the consequences of misguided legal action, then yes. In addition, if multiple clients with lived experience come forward with similar stories with the same landlord, then there’s your plural of anecdotes that might warrant investigation.

However, there’s also a sense in which we have to treat some client data with mild skepticism, particularly if the client has been a “super-utilizer,” received redundant services, or seems to understand how housing prioritization works and therefore knows that answering an intake question a certain way would work in his best interests, given the scarce resources many communities experience. After all, service provided to one client also necessarily takes away resources from another potential client.

Enter what is widely called the Replication Crisis, where we’ll learn about the limits of social science data and how the services sector might learn from academia.

The Replication Crisis in Social Science

In social and psychological science, there is a well-known phenomenon happening in research, dubbed the Replication Crisis. “Crisis” might be a severe term, but here’s the gist: many widely-cited, landmark social and psychological studies are failing to replicate their results in similar studies that try to find converging evidence of the same phenomena.

Reported by Nature, social psychologist Brian Nosek and his co-authors attempted to repeat the work reported in 98 original studies, but only 39 of the 100 replication attempts were successful.

Why does this failure to replicate matter? It matters mainly because convergent validity is how many psychological/sociological discoveries are correlated. If multiple lines of evidence, or multiple studies trying to replicate similar results, point to the same conclusion—even tentatively—the hypothesis is proved out more.

Examples of Data Convergence in Social Services

There are several examples of convergent validity in social sciences, but what are examples of it in social services?

1. Client Well-Being Assessments: Self-reported well-being scores might be subjective, such as the WHO-5, but case workers should provide an independent assessment too. But how to avoid prejudicing the case worker? Make sure they are not looking at the self-reported score beforehand. This “single-blind principle” follows for the other examples.
2. Mental Health Status: Clients asked to fill out a measure like the Patient Health Questionnaire (PHQ-9) to assess their levels of depression might need to also be diagnosed by a professional counselor, who should use structured clinical interviews and DSM-5 criteria.
3. Job Readiness, Substance Abuse, Housing Stability: Surveys for these three (and other similar) issues are standardized across many states, but vulnerability indices remain controversial. For example, the VI-SPDAT is used widely but is often criticized for ignoring certain equity issues among traditionally marginalized populations. However, whatever vulnerability index you use for housing prioritization, the key is that evaluators might need to have an independent evaluation without referring to self-reported data first, so as to prevent bias.

Strategies to Mitigate Bias in Self-Report Data

Data quality will always remain an issue in human-centered data. For instance, HUD indicates in the 2024 data standards for HMIS (p. 54), “staff observations should never be used to collect information on race and ethnicity,” also including gender. But there are some general principles in mitigating potential issues with social service data.

1. Triangulation: Clinical and case workers should try to verify the claims made in self-reported data. For example, administrative data might indicate “super-utilization,” or clinical records might indicate unreported co-morbidities. To that end, ensure that all case workers are trained to collect data in the same manner, so there’s consistency. It’s entirely possible for case workers to use the same tools but still collect erroneous data if they lack proper training.
2. Standardized Tools: Make sure your assessment and vulnerability indices are as stripped of bias as possible and that they are implemented the same way across the entire service network or CoC. Rigorous data collection throughout surveys does introduce the risk of reporting fatigue for clients, which is why coordinated entry/care coordination is essential, so clients don’t have to perform multiple intakes to determine eligibility. For example, multiple communities have recently presented alongside Eccovia on how they leverage ClientTrack to coordinate care.
3. Confidentiality and Informed Consent: Assure clients/respondents that their responses are indeed private to the service providers, and do everything you can to ensure they feel comfortable and understand the questions you’re asking. For example, the City of Topeka’s own CoC collaborated with law enforcement only for a specific unit that was trained to handle behavioral health and homeless encampment crises, to help homeless individuals avoid low-level law enforcement action and get them into housing. This entailed tracking client consent across the whole process.

Use Care Coordination to Improve Data Quality

All these above-mentioned strategies are possible with custom-engineered solutions, like ClientTrack, which integrates shared data with regulation-compliant workgroups across service networks. If you or your community would like to know more, please schedule a demo or enroll in our monthly newsletter so you can stay on top of new developments in our products, learn best practices, and collaborate with other communities