Busy As a Bee

You may have noticed the recent near radio silence from Sara on our blog. No, she isn't on a secret mission and hasn't left academia to join the NSA. She, however, has been busy as a bee this past year, starting with giving birth to our daughter, being named a W.T. Grant Scholar, and engaging in important academic research.

My pride in her commitment to, excellence in and passion for issues of educational and social inequality is coupled with a recognition of her unwillingness to see academic research relegated to dusty and sometimes impenetrable academic journals. Sara has been aggressive and public with her research and committed to engaging in and communicating her work in a policy relevant manner. That fits a critical need in public policy conversations.

That's why I was quite pleased to see Sara's name mentioned among the ranks of the most prominent academics in the nation in the "EduScholar" rankings issued by the American Enterprise Institute's Rick Hess. And Sara isn't even yet a senior scholar nor is she an economist (who are overrepresented). Hess says:

The academy today does a passable job of recognizing good disciplinary scholarship but a pretty mediocre job of recognizing scholars with the full range of skills that enables them to really contribute to the policy debate. Today, there are substantial professional rewards for scholars who do hyper-sophisticated, narrowly conceived research, but little institutional recognition, acknowledgment, or support for scholars who carry their efforts into the public discourse. One result is that the public square is filled by impassioned advocates, while silence reigns among those who may be more versed on the research or more likely to recognize complexities and hard truths.

I think these kinds of metrics are relevant because I believe it's the scholars who do these kinds of things "who can cross boundaries, foster crucial collaborations, and bring research into the world of policy in smart and useful ways."

Jay Greene -- seeking to give credit to more junior scholars who have had a great impact on contemporary public policy conversations and to move beyond rankings based on a single year (2010) of performance -- perfected the Hess rubric, causing Sara's ranking to increase by about 30 points to #39.

Hotshot researchers like Roland Fryer, Jacob Vigdor, Susanna Loeb, Matthew Springer, Brian Jacob, Jonah Rockoff, and Sara Goldrick-Rab are having a large impact on current education policy discussions even though their careers have not been long enough to accumulate a longer list of books and articles. The original ranking shortchanged these scholars in measuring their current “public presence.”

I agree. As I mentioned in this recent post, advocates who too often simply echo one another's opinions are too influential in policy debates. There is an important void to be filled by the likes of academic researchers as well as classroom teachers.

Congrats, Sara! Keep up the great work!

Seeking a PostDoctoral Fellow!

Hi folks-- sorry for the long absence-- I'm hoping to hire a postdoc (or doctoral student) in the next year and wondering if any of our readers might be interested? Here are details...

Sara

----------------------------

Wisconsin Scholars Longitudinal Study

Position Announcement: Funding for Junior Researcher of Color

Graduate Project Assistantship or Postdoctoral Fellowship

Sara Goldrick-Rab, Assistant Professor of Educational Policy Studies and Sociology at UW-Madison, seeks a talented junior researcher of color to join the Wisconsin Scholars Longitudinal Study in 2011 as it prepares to enroll its second cohort of students.

The WSLS is the first-ever longitudinal randomized controlled trial of need-based financial aid. It is a mixed-methods study following two cohorts of Wisconsin Pell grant recipients through college and into the workforce. It is led by an interdisciplinary team of researchers, including co-director Douglas N. Harris, and includes collection of administrative, survey, and in-depth interview data. For more information, please see the WSLS website.

Dr. Goldrick-Rab holds a William T. Grant Scholars award for her work on the WSLS. Her project, “Rethinking College Choice in America,” applies ideas and methods from developmental psychology and behavioral economics to examine how college students' use of time, emotional experiences and amounts of sleep interact with financial aid and affect chances of earning degrees.

The Scholars award includes the opportunity to seek supplementary funding (also from the William T. Grant Foundation) to help Scholars build strong mentoring relationships with students of color. According to the Foundation, “these awards address two issues that receive insufficient attention and resources: how to be a good mentor and challenges facing people of color in research careers.” The Foundation estimates awarding three to four awards in the amount of $60,000 for mentoring doctoral students and $85,000 for mentoring postdoctoral fellows (inclusive of a maximum of 7.5 percent in indirect costs). Funding will begin on July 1, 2011, and end June 30, 2012. Mentors and mentees will come together during annual winter meetings designed to support the mentoring relationships, Scholars’ development as mentors, and Junior Researchers’ development as researchers.

If selected and funded, the Junior Researcher will work with Dr. Goldrick-Rab on her Scholars project and also on an independent inquiry related to the second cohort of students participating in the WSLS.

Eligible applicants must meet the following criteria, according to Foundation rules:
• Junior Researchers of Color may be African American, Asian or Pacific Islander American, Latino, and/or Native American.
• Junior Researchers may be full-time doctoral students or postdoctoral fellows.
• At minimum, students must be in their second year of doctoral studies at the onset of the award.
• The Junior Researcher must be housed at UW-Madison or a nearby college or university.

In addition, per the needs of the WSLS and Dr. Goldrick-Rab’s project, eligible applicants must also possess all of the following qualifications:
• A background in social science coursework, preferably in sociology or anthropology and/or training in a public policy or social work program.
• Prior experience conducting in-depth interviews .
• Statistics skills, include comfort with regression analysis and preferably familiarity with STATA.
• Ability to demonstrate attention to detail, strong writing skills, and the capacity to work independently.
In short, the highly qualified applicant will have already begun to emerge as a skilled mixed-method researcher, or researcher-in-training.

This position is currently in the planning stage. At this time, Dr. Goldrick-Rab seeks interested students who would like to collaborate on an application for either a project assistantship or a postdoctoral fellowship, to begin in July 1, 2011 and conclude June 30, 2012. The final application is due March 2, 2011 and the award decision will be made in June 2011. The applicant is required to jointly prepare the application with Dr. Goldrick-Rab (as the Foundation mandates). To the extent that the student goes beyond the application planning and participates in laying the groundwork for Cohort 2, that work may be compensated on an hourly basis (to be negotiated), regardless of whether the award is granted.

If interested, students should contact Dr. Goldrick-Rab by November 5, 2010. Please send an email that includes a cover letter explaining reasons for interest, along with a CV and contact information for at least two professors who can provide recommendations to srab@education.wisc.edu.

Thank you for your interest!

The Manifesto, Income Inequality & Credibility

On Friday, I wrote a blog item ('Misleading Manifesto') chiding a group of urban superintendents for misstating educational research in a 'manifesto' published in Sunday's Washington Post. Teacher quality *is* important -- but it does not matter MORE THAN family income and concentrated poverty.

I am convinced that too many educational reformers are happy to 'spin' the truth for rhetorical purposes. I think this is exactly what we saw in this manifesto. While this may help to simplify messaging, target solutions at a more narrowly construed problem, and focus in on what education leaders have direct control over, it carries an inherent policy danger along with it. That danger is two-fold: (1) teacher policy reforms may be set up for failure by overstating their potential impact; and (2) more comprehensive strategies desperately needed to combat rising income inequality and growing poverty in our nation may be discounted and ignored.

For me, this isn't an issue of setting low expectations for children from poverty. We must train and support our teachers to have high expectations and develop the potential in all children. But, from a policy perspective, which is the world in which I work, to not even discuss poverty and inequality -- even though the research evidence points to its preeminence -- is akin to taking it off the table as a policy priority.

Nor it is a lack of belief in the ameliorative benefits that sensible teacher reforms can have on student outcomes by expanding the recruitment pool of teacher candidates, improving initial training and on-going support of classroom teachers, improving teaching and learning conditions within schools, providing differential compensation to teachers for leadership roles, difficult assignments, shortage fields, and demonstrated effectiveness, and more....

For teacher quality specifically, as I argued in my previous post, playing fast and loose with the facts isn't necessary. There is a powerful argument to be made based on the fact that teachers are the most important school-based influence on student learning. That's exactly what my colleagues at the New Teacher Center have done. We've made careful and honest declarations about teacher quality being the most critical within-school variable, but haven't framed the issue in a way that would make us education-industry Pinocchios.

And this leads us directly to the question of credibility. While I am personally inclined to support elements of what the superintendents' manifesto calls for -- and inclined to support elements of broader education and teacher reform agendas -- I am disinclined to associate myself with a clarion call that is dishonest on its face and misserves the national need for a critical conversation and accompanying set of public policies to address issues of economic inequality. That need extends well beyond the education system and requires responses much broader than merely strengthening the teaching profession and overhauling human capital systems.

Former Labor Secretary Robert Reich recently has been banging the drums challenging policymakers -- and Democrats, in particular -- to address our nation's historic levels of income inequality and rising levels of poverty. As reported by the Washington Post's Steven Pearlstein, since 1976 "virtually all of the benefits of economic growth have gone to households that, in today's terms, earn more than $110,000 a year." Further, UNICEF reports that the United States has the highest rate of childhood poverty among 24 OECD nations -- over 20% -- and the second-worst rate (barely ahead of bottom-dwelling Great Britain) of childhood well-being in the industralized world. Further, as Walt Gardner recently noted, a September 2010 U.S. Census Bureau report showed that the percentage of Americans below the poverty line in 2009 was the highest in 15 years. And the rise was steepest for children, with one in five affected. Think this has any bearing on U.S. students' relatively poor performance on international student assessments? Uh-huh.

So, let's talk about how to strengthen teaching and its central importance to student outcomes. But let's not fence ourselves in with self-serving rhetoric. Let's be honest in our communications and expansive in our thinking about policies needed to improve the lives of American children.

It's about education -- and a whole lot more.

Misleading Manifesto

I'm sorry, but the "manifesto" published in today's Washington Post really pisses me off because it is built upon a false premise. It is authored by a number of urban school superintendents, including Chicago's Ron Huberman, New York City's Joel Klein, Washington DC's Michelle Rhee, and New Orleans' Paul Vallas. And it -- intentionally? -- misstates educational research.

"[T]he single most important factor determining whether students succeed in school is not the color of their skin or their ZIP code or even their parents' income -- it is the quality of their teacher."

No. That is patently false.

Now, listen here. I work for a teacher-focused, non-profit organization, the New Teacher Center (NTC). Wouldn't it be powerful to go out and say that teachers matter more than ANYTHING else? But they don't. In terms of school-based variables, they do. But in terms of all variables that impact students, they simply do not. No research says that. In our messaging at the NTC, we are always careful to say that teacher quality is the most important school-based variable for student achievement (examples here and here (on page 4)). That's accurate, honest and powerful in its own right.

So why not make the case for improving teaching in a honest fashion? There is an incredibly strong case to make that improving teaching quality is a critically important and policy amenable part of the solution to increasing student achievement and closing achievement gaps for disadvantaged students. But it's only part of the answer which requires solutions beyond the educational system. Let's not lose sight of that.

At the Shanker Blog, Matthew Di Carlo explored this same issue last month and took journalists to task for making similar claims. Back in July, he summarized existing teacher quality research.

September 16, 2010:

The same body of evidence that shows that teachers are the most important within-school factor influencing test score gains also demonstrates that non-school factors matter a great deal more. [emphasis added] The first wave of high-profile articles in our newly-energized education debate not only seem to be failing to provide this context, but are ignoring it completely. Deliberately or not, they are publishing incorrect information dressed up as empirical fact, spreading it throughout a mass audience new to the topic, to the detriment of us all.

Even though the 10-15 percent explained by teachers still represents a great deal of power (and is among the only factors “within the jurisdiction” of education policy), it is nevertheless important to bear in mind that poor educational outcomes are a result of a complicated web of social and economic forces. [emphasis added] People have to understand that, or they will maintain unrealistic expectations about the extent to which teacher-related policies alone can solve our problems, and how quickly they will work.

July 14, 2010:

But in the big picture, roughly 60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty). [emphasis added] Observable and unobservable schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects. The rest of the variation (about 20 percent) is unexplained (error). In other words, though precise estimates vary, the preponderance of evidence shows that achievement differences between students are overwhelmingly attributable to factors outside of schools and classrooms (see Hanushek et al. 1998; Rockoff 2003; Goldhaber et al. 1999; Rowan et al. 2002; Nye et al. 2004).

Let's take Di Carlo's and Joe Friday's advice. Just the facts, please.

Positive Effects of Comprehensive Teacher Induction

Today, Mathematica Policy Research, Inc. released the final report of its IES/U.S Department of Education-funded randomized controlled trial (RCT) of comprehensive teacher induction. It shows a statistically significant and sizeable impact on student achievement in mathematics (0.20 standard deviations) and reading (0.11 standard deviations) of third-year teachers who received two years of robust induction support. That's the equivalent of moving students from the 50th to 54th percentile in reading achievement and from the 50th to 58th percentile in math achievement.

As a basis of comparison, I note that in 2004, Mathematica conducted a RCT of Teach for America (TFA). In that study, it compared the gains in reading and math achievement made by students randomly assigned to TFA teachers or other teachers in the same school. The results showed that, on average, students with TFA teachers raised their mathematics test scores by 0.15 standard deviations (versus 0.20 standard deviations in the induction study), but found no impact on reading test scores (versus 0.11 standard deviations in the induction study).

In another recent Mathematica report (boy, these folks are busy!), the authors note that "The achievement effects of class-size reduction are often used as a benchmark for other educational interventions. After three years of treatment (grades K-2) in classes one-third smaller than typical, average student gains amounted to 0.20 standard deviations in math and 0.23 standard deviations in reading (U.S. Department of Education, 1998)." In that report -- an evaluation of the Knowledge Is Power Program (KIPP), Mathematica researchers found a very powerful impact from KIPP: "For the vast majority of KIPP schools studied, impacts on students’ state assessment scores in mathematics and reading are positive, statistically significant, and educationally substantial.... By year three, half of the KIPP schools in our sample are producing math impacts of 0.48 standard deviations or more, equivalent to the effect of moving a student from the 30th percentile to the 48th percentile on a typical test distribution..... Half of the KIPP schools in our sample show three-year reading effects of 0.28 standard deviations or more."

Is it appropriate to compare effect sizes among RCTs or, for that matter, among research in general? I am told that it is, although certainly considerations such as cost effectiveness and scalability have to enter into the conversation. Implementation issues also must be attended to. With regard to teacher induction, the issue of cost effectiveness was addressed in a 2007 cost-benefit study published in the Education Research Service's Spectrum journal and summarized in this New Teacher Center (NTC) policy brief.

Disclosure: I am employed by the NTC which participated in the induction RCT, and I helped to coordinate NTC's statement on the study.

The NTC is "encouraged" by the study. However, NTC believes that "it does not reflect the even more significant outcomes that can be achieved when districts have the time, capacity and willingness to focus on an in-depth, universal implementation of comprehensive, high-quality induction. It speaks volumes about the quality of induction and mentoring provided and the necessity of new teacher support that student achievement gains were documented despite [design and implementation] limitations to the study."

UPDATE: Read the Education Week story by Stephen Sawchuk here. And the Mathematica press release here.

Politics, As Usual

The recent decision by the American Educational Research Association (AERA) to hold a news conference condemning Arizona's new immigration law was somewhat unpredictable, and according to at least a few observers, unwise. For example, Rick Hess told the Chronicle of Higher Education it wasn't "smart politics" to "baldly politicize the role of research." The Chronicle's editors fanned the flames further by titling its article, "Education-research group puts itself on the border of advocacy."

Oh, the horror--research and advocacy meeting, having coffee, perhaps even deciding to date. The children which could result are feared by PhDs everywhere, particularly those evil twins: Compromised Objectivity and Biased Conclusions.

Of course academia trains us to think, like Hess, that research is worthy only when fully divorced from politics. Our research questions should be derived from theory, stemming only from the reading of great books and dusty journals, and never from a desire to enter policy or social debates. Puhleese. Every research question is inherently political--we conceive and ask questions the way we do because we have a desire to know something. Knowledge is socially, and therefore politically, constructed.

I'm the first to admit that AERA is a deeply flawed organization, but aren't they all (Hess's included)? I think honesty and transparency are among the best qualities, and would much rather AERA's leaders and members take visible positions on issues they care about rather than pretend not to have opinions. Research lacks an agenda only in the most naïve of imaginations. But agendas lack research all-too-frequently. If AERA begins to use its members' work to create a research-backed agenda, that can only be a good thing.

Congratulations!

Congratulations to my wife, Sara Goldrick-Rab, who was one of only four academics named yesterday as a 2010 William T. Grant Scholar!

UPDATE: University of Wisconsin-Madison press release.

The W. T. Grant Scholars Program supports promising early-career researchers from diverse disciplines, who have demonstrated success in conducting high-quality research and are seeking to further develop and broaden their expertise. Candidates are nominated by a supporting institution and must submit five-year research plans that demonstrate creativity, intellectual rigor, and a commitment to continued professional development. Every year, four to six Scholars are selected and each receives $350,000 distributed over a five-year period.

The four new William T. Grant Scholars and their research projects are:

Elizabeth Oltmans Ananat, Ph.D., Assistant Professor, School of Public Policy, Duke University --
“Economic and Social Determinants of the Educational, Occupational, and Residential Choices of Young Adults”

Phillip Atiba Goff, Ph.D., Assistant Professor, Department of Psychology, University of California, Los Angeles -- “Broken Windows, Broken Youth: The Effect of Law Enforcement on Non-White Males’ Development”

Sara Goldrick-Rab, Ph.D., Assistant Professor, Educational Policy Studies Department, University of Wisconsin-Madison -- “Rethinking College Choice in America”

Patrick Sharkey, Ph.D., Assistant Professor, Sociology Department, New York University -- “The Impact of Acute Violence and Other Environmental Stressors on Cognitive Functioning and School Performance”

Spin Cycle

Education Next apparently has provided a platform for school choice advocate George Mitchell to shill for voucher schools outside of the state of Wisconsin. Here is his latest spin on a study that shows the high school graduation rate to be 12 points higher in seven Milwaukee voucher schools compared with 23 Milwaukee public high schools.

The Milwaukee Journal Sentinel story by Erin Richards provides the crucial quote regarding causation from the study's author, John Robert Warren, a sociology professor at the University of Minnesota:

"We still don't know whether it's going to the voucher school that causes you to be more likely to graduate, or if it's something about the kinds of families that send their kids to voucher schools would make them more likely to graduate," he said.

Then there's the whole question of which and how the voucher and public high schools were chosen for purposes of comparison. More questions than answers. Unlike Mitchell, I neither see this report as providing "another piece of evidence suggesting that urban students benefit when afforded more educational options," nor "new data" to encourage President Obama and Education Secretary Duncan to take "a second look at the power of parent choice."

The study was funded by the voucher-advocacy group School Choice Wisconsin, run by Mitchell's wife, Susan. The Mitchells have split from national school choice leader Howard Fuller who is devoting his current efforts to furthering accountability and quality in the Milwaukee Parental Choice Program.

After the spin cycle, be sure to rinse.

For past perspective on Voucher Inc, please visit here.

Using Value Added to Assess Teacher Effectiveness

The Association for Public Policy Analysis and Management -- an organization not widely known outside of academia and technical policy circles -- puts on truly meaty conferences. I've attended three APPAM conferences to date, including the Annual Fall Research Conference going on in Washington, DC this week.

Education is merely one strand at APPAM, but the sessions feature some of the biggest names in educational research addressing some very policy relevant issues. The current conference features sessions on value-added modeling, school choice, teacher certification and teacher induction, teacher performance pay, financial aid, college persistence, and more.

The session I attended yesterday on "Using Value Added To Assess Teacher Effectiveness" was excellent. It featured four papers each of which I will undoubtedly oversimplify in this brief blog post. (I encourage you to seek out the papers and read them closely -- below I've linked to those that are available.) One by Dan Goldhaber and Michael Hansen (University of Washington) suggests that year-to-year correlations in value-added teacher effects are modest, but that pre-tenure estimates of teacher job performance do predict estimated post-tenure performance in both math and reading. A second by Julian Betts (UCSD) and Cory Koedel (University of Missouri-Columbia) suggests that bias does exist in value-added models due to student sorting, but that it can be overcome through the use of multiple years of value-added data; further, the study suggests that data from the first year or two of classroom teaching may be insufficient to make reliable judgments about teacher quality. A third by Michael Weiss of MDRC suggests that that teacher variability carries implications for measuring program effects within randomized controlled trials when those teachers are not randomly assigned. And a fourth by John Tyler (Brown University) and Tom Kane (Harvard University) found that teacher assessments made using classroom observation rubrics (such as Charlotte Danielson's) are closely aligned with value-added ratings of teachers.

Whispered Policies

Friday's Chronicle reports on a new study that points out how difficult it can be to identify which colleges and universities have no-loans policies designed to enhance affordability. Author Laura Perna and her colleagues find that the majority of elite institutions with these policies fail to advertise them in ways that are accessible to low-income students and families-- effectively maintaining their status as "bastions of privilege." The researchers then go on to make several helpful suggestions about how colleges could change their tactics to increase awareness and uptake of their progressive efforts.

But they could've gone one step further and discussed the incentives colleges have to maintain the status quo-- that is, to continue making their current and former students and staff feel good with liberal actions, garnering attention in elite venues such as the New York Times, without fundamentally changing their overall enrollment demographics or costing too much money. Call me cynical, but as a sociologist it strikes me that this is exactly how power is effectively maintained in the face of pressure for socially responsible actions from powerful institutions.

According to another recent study by economists Waddell and Singell, of the just-over 384,000 Pell Grant recipients attending 4-year institutions in 2000, only 0.3% were enrolled at Ivy League institutions (which disproportionately possess these no-loan policies). Across elite private institutions, Pell recipients rarely amount to more than 1% of the entering class. In 2000, there were only 108 Pell recipients in the freshman class at Harvard, and just 50 at Princeton. These are tiny, tiny numbers. So if no-loans policies actually resulted in massive increases in applications from low-income students, we could see many consequences for those schools. For one, their institutional aid budgets would have to grow-- if low-income students managed to get past the admissions hurdle. Second, depending on how exactly admissions dealt with the increased applicant pool (e.g. whether a 'thumb' was placed on the scale so as to ensure a reasonable proportion were admitted-- an action recommended by Bill Bowen), graduation rates might be affected. Third, you'd see a larger, more visible contingent of people on campuses from different family backgrounds, affecting social dynamics. Many of these outcomes could be interpreted as both positive and negative, depending on your perspective.

Simply put, right now colleges with small numbers of low-income undergraduates can afford to adopt no-loans policies. Based on the two studies discussed here, this is likely because their policies are only weakly communicated to the groups who'd be affected (I hestitate to call these the "targeted audiences" however) and the effects on enrollment are small and subtle. For example, Waddell and Singell conclude that such policies do not increase the overall number of needy institutions at institutions but do have some effect in skewing the composition of that group toward somewhat lower-income students who've traveled longer distances to attend college. Since positive publicity generated by laudatory articles in the elite press may well generate enough new alumni donations to offset current costs, the whole thing may be close to a "wash" --under current circumstances. More effective publicity and outreach to families who do not read the mainstream liberal elite newspapers or visit websites like finaid.org to get their information about college, might change the game. Under those new conditions, I have to wonder-- would no-loans policies continue to be so popular in elite higher education?

The Stories We Tell Ourselves

Once upon a time, college students could pay their tuition with a mix of family support, financial aid, and perhaps a little work. Today, family support and aid are woefully inadequate for a broad swath of undergraduates, and full-time work is common.

Is working while in college truly necessary? Are the earnings used for academic expenses related to postsecondary education, or are they frittered away on life's pleasures? Since a handful of studies indicate a negative association between working long hours and rates of degree completion, these questions have taken on broader significance.

Unfortunately, few studies track students' income and expenditures in systematic ways. To better understand spending patterns, and attempt to tease out the reasons for those patterns, one would ideally have longitudinal data collected for a large sample of students, and complemented by in-depth interviews with a sub-sample of students to delve more deeply into the reasons underlying decisions, and validate the measures employed. Now true confession: Together with Doug Harris, I am conducting just such a study right now, the Wisconsin Scholars Longitudinal Study. But that's not why I'm writing this-- we don't yet have data to report on.

But apparently someone else does. A few weeks ago, a news outlet reported the headline "Will Work for Beer," covering the release of a new study from the Bureau of Labor Statistics, published in the Journal of Population Economics. In that study the authors used national cross-sectional data and determined that the earnings students make from work are not enough to replace contributions from their parents, or cover tuition costs. According to the report, "We test several hypotheses regarding the financial motives for and academic effects of college student employment and find empirical support for the hypothesis that a decrease in parental transfers increases the work hours of four-year college students. We also find that an increase in the net price of schooling increases the number of hours worked by both four-year and two-year college students."

Ok. So the decision to work may have something (but not everything) to do with how much support parents provides and how expensive college is. Unsurprising. Not particularly newsworthy.

But the lead author didn't stop there. Instead, she waded into popular stereotypes about college students, telling the reporter that the results mean that the drive to work isn't coming from a need to really make ends meet-- instead, "students...work to have ‘beer money,' money for entertainment, money to pay other expenses, just not their tuition."

Huh?

Her conclusion took a gigantic interpretive leap from her data. Notably, it's not a conclusion found anywhere in the actual research paper. All her evidence suggests is that students' work isn't generating income equivalent to parental contributions or in line with college costs. This could mean many things, including that students have a hard time finding enough work to generate sufficient earnings. Of course it suggests they likely need to find other ways to make ends meet-- including loans. But it says nothing about what they use their work earnings for, how they prioritize expenses, what they go without, etc. With her statement to the press, the author did little more than simply impute meaning to meaningless results.

Why mention "beer money"? It's not uncommon for an academic paper to simply say what it shows-- and conclude that while we need to know more about explanations for patterns in the data, we just don't have the information in the dataset to tell us what we need to know. Why step outside those bounds, and lend fodder to the fire? In what way is this helpful-- to policymakers, to students, or frankly, to anyone?

Working students are often struggling students. There's good qualitative evidence on this, even if the quantitative evidence isn't yet available. Professors dislike them because they tend to fall asleep in class, having been up serving on the graveyard shift instead of studying. Their classmates often don't know them well, since student-employees have little time left for socializing. Their grades are lower than average, their stress levels high, and their chances of degree completion relatively low. So why do we feel the need to minimize their need to work, to mock them for it, to enforce a stereotype that their earnings are spent at bars? It seems nothing less than classist-- in the absence of providing students with sufficient financial supports to make working during college truly optional, we try and make ourselves feel better by telling stories that students work not out of true financial need, but rather a desire to imbibe.

Maybe that helps some fraction of folks sleep at night, but I seriously doubt it's grounded in any kind of truth.

Research: Attracting New Teachers to Urban Schools

New research led by Tony Milanowski of the University of Wisconsin-Madison provides more evidence that increasing teacher pay may not be the best approach to attract new teachers to high-need, hard-to-staff urban schools. A key finding of the study -- published in the International Journal of Education Policy and Leadership -- which explored job factors important to pre-service educators was that "working conditions factors, especially principal support, had more influence on simulated job choice than pay level."

'Policy implications' include:

• "[M]oney might be better spent to attract, retain, or train better principals than to provide higher beginning salaries to teachers in schools with high-poverty or a high proportion of students of color."
• "[I]nduction programs and curricular flexibility are important to new teachers. The finding that induction programs are attractive, combined with evidence that such programs can be
  effective in reducing teacher turnover (e.g., Ingersoll and Kralick, 2004; Smith and Ingersoll, 2004), suggests that urban districts may want to implement high-qualityinduction and mentoring programs, especially for new teachers in schools with high proportions of poor students or students of color."

Superteacher To The Rescue!

Given the recent spate of federally-funded studies showing no effect of a variety of educational innovations and interventions, my predicted answer to the question ('Can Teachers' Talent Translate Elsewhere?') posed in this Houston Chronicle story is "no."

I worry, however, that the basic premise of the federally funded Talent Transfer Initiative is faulty and builds upon the notion of teaching (as reinforced by popular culture) as an individual rather than as a collective pursuit. Can 'superteachers' walk into dysfunctional school cultures and work magic that can result in a quantifiable impact on student learning? Some surely can. (It's too bad we can't clone Jamie Escalante and Frank McCourt, isn't it?) More important to ask is, should we expect them to?

What is more desperately needed than an expensive scheme to redistribute 'superteachers' is a serious attention to teaching and learning conditions. My New Teacher Center colleague, Eric Hirsch, spearheads assessment of school culture and the training of school administrators to more effectively shape it. His and independent research (here and here) has identified that teacher effectiveness is facilitated by a positive school context, including support from leadership, the existence of a collaborative working environment, and time for professional learning.

It doesn't appear that the Talent Transfer Initiative envisions teaching and learning conditions as part of the solution, and that's terribly unfortunate. I wonder if the TTI is even collecting such data to investigate the relationship between these variables and teacher success, or lack thereof? Until we address these contextual issues in low-performing and hard-to-staff schools, we're not going to get the results that we expect and students deserve.

UPDATE (9:35 p.m.) -- Claus von Zastrow offers an excellent blog post on Public School Insights about this study as well.

The College Payoff

Cross-posted from Brainstorm...

One reason I was so excited to join Brainstorm was that it presented a chance to go toe-to-toe once in awhile with my colleague and friend, Kevin Carey. Over the years I’ve read Kevin’s work frequently, and often found myself respectfully disagreeing with him. What’s the best is that our points of disagreement are always worth arguing over—as we are both so clearly interested in seeing major changes when it comes to equity and educational attainment.

This past week presented an illustration. I wrote a critique of an American Enterprise Institute report Kevin co-authored, and he responded with a post taking on some of my points. Since I have plenty to say in turn, and since I think this is a discussion very much worth having, I want to continue the debate here.

My main point in the original post was that the AEI authors jumped to conclusions I don’t find particularly helpful. They want to do something about low graduation rates, but prematurely conclude the solutions lie in changing institutional practices. Kevin replied with “colleges make a difference” and who would want to argue otherwise?

Well, in some sense—me. To put a finer point on it, I argue they individually make a difference, but mainly on the margins and for uncomfortable reasons.

Here's the outline of what I'm thinking. First, I emphatically believe that there’s an economic payoff to years of college, and credentials in particular, and that most students learn at least something during the time they spend in college. I also buy the research of Jennie Brand & Yu Xie (and others) who find that students least likely to attend college are in fact most likely to benefit from attending. This is why I'm for greater equity in access and completion. Second, I definitely do not believe that the sources of observed differentials in student outcomes among colleges—things like levels of student engagement, graduation rates, and returns to the degree—are about institutional policies and practices of the kind Kevin's referring to. Why? Since at least to some degree, most of our colleges select kids for admission based on evidence that they will be engaged and “able to benefit” from the experience--and they do this in different ways and to differing degrees-- then variation in graduation rates is clearly going to result.

So that's why colleges themselves probably only matter on the margins. And here comes the "uncomfortable" part. We have to recognize that (like it or not) the primary functions of our colleges and universities are (1) facilitating the creation of social networks and (2) credentialing. We go to college to hang out with people who will later be our friends, spouses, colleagues, and Facebook buddies-- and these people will help us find jobs and make good connections throughout our lives. We also go so that future employers will find us more desirable-- whether or not they should.

Ultimately, I don’t think that detracts from the importance of higher education, and in particular from the goal of broadening access to higher education. It’s a gatekeeper, and more people need to get in. But it does—and should- detract from the sense that some colleges “do a better job” than others. What does that really mean when what they "do" is help you meet your socially advantaged counterparts and send smoke signals to employers? If that’s what you’re buying, and you understand that, ok. I don't think most people do.

Now back to Kevin’s points. Sure, some colleges have high dropout rates and that’s a shame. Part of the reason is that they’re enrolling students who—a decade or two ago—wouldn’t have attended college. Now, in a college for all culture, they go. Some get a degree- and in this sense, opening doors is serving them well. Others suffer enormous personal costs, financial investments and feelings of personal failure. These things are hard to measure, but are undoubtedly affecting the numbers we observe. We hardly pay attention to (or measure) important factors like individuals’ health and development, and yet we assume that any remaining variation in outcomes (e.g. engagement or graduation rates) not accounted for by observable factors can be credited to college practices—instead of attributing the variation to the vast array of important predictors of individual functioning that we just don’t measure. Why?

If we want to make better policies to increase attainment and close gaps we need to get a better handle on what the real problems are. What if the differences in graduation rates are explained by differences in how mentally and physically prepared students at different colleges are for postsecondary education? Right now, that doesn’t show up in the data. So Kevin says the remaining variation is in the colleges’ domain. Yet if, based on that, we direct policy interventions at the colleges when the real problem is health, we’ll fail to generate change. That’s simply not useful, and potentially a waste of money. Why wait for research? This is why. We need more numbers, and less conjecture.

How College Gets Under Your Skin

Cross-posted from Brainstorm, over at the Chronicle of Higher Ed.

I’ve been preoccupied by sleep lately. Not sleeping— though as I approach the end of my first trimester I sure could use some— but sleep itself. What it means to sleep a little or a lot, how it affects your daily interactions with others, etc. This is something I know a tiny bit about, having spent a solid year sleep-deprived after the birth of my first child, but not something I’ve devoted my academic time to.

Until now. I just spent two full days at the Cells to Society (C2S) Summer Biomarker Institute. C2S is also known as the Center on Social Disparities and Health at Northwestern University. It’s directed by developmental psychologist Lindsay Chase-Lansdale, and has additional star power in folks like Thom McDade, Emma Adam, and Chris Kuzawa. These are social science researchers who have mastered the hard sciences as well, and are using medical tools to get at how social practices and environments “get under the skin.”

What does that mean? Well, to explain I’ll tell you why I’m thinking about sleep. It all begins with an attempt to understand the reasons why so many low-income kids drop out of college. A big problem, to be sure— and one that we still don’t know enough about. I’m thinking that has to do with the limited number of ways in which we’ve approached the problem. It’s primarily treated as an educational issue, one we tackle with a combination of college practices and individual-level incentives like money.

But what if, in fact, higher rates of dropout had something to do with poorer mental or physical health? What if the conditions in which low-income kids experience college actually make them less healthy? We all understand stress, and most of us think it’s a regular part of life everyone deals with. But we have differing types and degrees of stress, and in turn differing responses and reactions. Some of us think being stressed out is about trying to fit in an optional French class to our busy schedules, because we’d like to hang out with that cute French boy. Others feel stressed because they do not have enough money to pay for lunch, and are working 2 jobs on top of 4 classes to try and make ends meet.

Looking at sleep patterns, and sleep quality, is one way to try and quantify the effects of college— and policies associated with college-going — on health. I have Emma Adam to thank for getting me to really start think of this as a research topic— one I plan to pursue. When you wake up well-rested your cognitive functioning is improved, and you can go out and learn. When you’ve been in bed for very few hours, or tossed and turned all night, it can be hard to drive to school, let alone master the material in class. At some point, you may just give up.

Lest anyone read this to mean that I think genetic differences underlie social class differences in college attainment— stop right there. Not at all. But there are complex ways in which one’s social environment can alter the biological state, even temporarily, which in turn affects academic achievement. I think it’s about time we start thinking, then, about how college gets under our skin.

Sorting, Selection, and Success

Cross-posted from Brainstorm, over at the Chronicle of Higher Education.

The latest report from the American Enterprise Institute, Diplomas and Dropouts, hits one nail on the head: plenty of students starting college do not finish a degree. Access does not equate with success, and partly as a result, U.S. higher education is perpetuating a lot of inequality.

What do we do about this? The authors identify a key fact: “analysis of graduation rates reveals wide variance among institutions that have similar admissions standards and admit students with similar track records and test scores.” They interpret this to mean that “while student motivation, intent, and ability matter greatly when it comes to college completion, our analysis suggests that the practices of higher education institutions matter, too.”

This is a pretty common argument made by many policy institutes and advocacy organizations, including but not limited to the Education Trust and the Education Sector. I understand their goal—to make sure that colleges and universities can’t hide behind the excuse of “student deficits” in explaining low graduation rates, and instead get focused on the things they can do something about. In some ways that mirrors efforts over the last fifty years to focus on “school effects” in k-12 education —witness the continuing discussion of class size and teacher quality despite evidence that overall variation in student achievement is much more attributable to within-school differences in student characteristics than to between-school differences (school characteristics). Like many others, I read those findings to say that if we really want to make progress in educational outcomes, we must address significant social problems (e.g. poverty, segregation) as well as educational practices. Don’t misinterpret me- it’s not that I think teachers don’t matter. It’s simply a matter of degree—where and how can we make the biggest difference for kids, and under what circumstances.

Unlike k-12, access in higher education isn’t universal and competitive admissions processes and pricing structures result in lots of sorting of kids into colleges and universities. As a result, they differ tremendously in the students they serve. In turn, as the AEI report admits, this necessarily shapes their outcomes.

The problem is, all this sorting (selection bias) has to be properly accounted for if you want to isolate the contributions that colleges make to graduation rates. (I’ll qualify that briefly to add that the role college enrollment management —tuition setting, financial aid, and admissions— plays in the sorting process is quite important, and is under colleges’ control.) But if you want to isolate institutional practices that ought to be adopted, you first have to get your statistical models right.

Unfortunately, I don’t think the AEI authors have done that. To be sure, they try to be cautious, pointing out colleges that look “similar” but have extremely different graduation rates (rather than modestly different ones). But how they reached “similarity” leaves a lot to be desired. It seems to rest entirely on level of selectivity and geographic region. Their methods don’t begin to approach the gold standard tools needed to figure out what works (say, a good quasi-experimental design). Important student-level characteristics (socioeconomic background, high school preparation, need for remediation, etc) aren’t taken into account. Nor are many key school-level characteristics (e.g. resource levels and allocations). In sum, we are left with no empirical evidence that the numerous other plausible explanations for the findings have even been explored.

I’m not surprised by this, but have to admit that I’m a bit bummed. Yes, I “get” that AEI and places like it aren’t research universities. Folks don’t want to spend long periods of time conducting highly involved quantitative research before starting to talk policy and practice. But I don’t see how this approach is moving the ball forward—sure it gets peoples’ attention, but it’s not compelling to the educated reader—the one who votes and takes action to change the system. Moreover, it doesn’t get us any closer to the right answers, or provide any confidence that if we follow the recommendations we can expect real change.

There have been solid academic studies of the causes for variation in college graduation rates (here’s one example). They struggle with how hard it is to deal with the many differences among students and colleges that are not recorded – and thus not detectable—in national datasets. If we want better answers, we need to start by investing in better data and better studies. In the meantime, I think skipping the step of sifting and winnowing for the most accurate answers is inadvisable. Though, sadly, unsurprising…

Effective Dissemination or Shameless Self-Promotion?

On Friday and Saturday I participated in an AERA/Hechinger Institute workshop for early career education researchers, designed to help us "learn the ropes" about disseminating and translating our work effectively. Bluntly-- I signed up for this thing because of my multiple forays into a world of blogging/Facebooking/Twitter and other forms of media interactions that have made me acutely aware of my communication limitations (e.g. how poorly I write sometimes, and how often I can't quickly convey what I really mean--see prior sentence as case in point). What I didn't sign up to do was to learn how to build my own reputation. (Sadly, the surprise among some readers is palpable...)

But I worried about this throughout the workshop. Getting quoted in the paper frequently can have unintended consequences, including making others think you want it to be all about you. This is probably especially true if anyone realizes that professors often have to reach out to the media in order to gain their attention-- despite reporters' stated interest in such efforts, they come across as too-eager-beaver. So why do this kind of workshop at all? Isn't it way too risky, putting myself out there like this?? In trying to think it over, I ventured back to what Deborah Lowenberg Ball said Saturday morning to our group, and to the cadre of folks attending Wednesday's Spencer reception as well. In a nutshell, she argued the following:

1. We need to become disciplined and effective communicators of education research. Right now, those in academia who speak out tend to sound like advocates, or border on unconfident or unclear about what we do and don't know works in education. Both contribute to a less-than-positive rep for Ed Schools.

2. If we don't take this on, others will- in fact they already are. Evaluation firms are a good example. They know how to talk about education but are rarely specialized experts in education. This does not mean they can't raise the big questions or go after answers, but it does mean that more often than not they miss many of the phenomena and problems particular to this social institution; simply because they spend less time with it.

3. Outreach is part of the university mission as a "public agent of education." If we in the academia refuse to engage in the struggle to share our expertise, we essentially cut off the world from what we actually do know about how to better educate kids.

So in sum-- after a long hard week at AERA, some of us spent time learning the ropes at Hechinger, figuring out how to speak to and with the media, policymakers, and practitioners not so we can enjoy seeing our names in print more often, but so that we can really strive to do our jobs. Those jobs include disciplined and effective outreach, and we'd fail if we didn't work at it.

The next time you see my name in print, I really hope you understand (a bit better anyway) my motivations. I thank Deborah for giving me, and so many others, a way to think through them.

Media Coverage of Conference Papers

The edublogosphere is rife with coverage of AERA papers right now. So is the edu-print media.

I'm not sure this is a good thing.

Remember, many academics treat the conference paper as a first, or maybe second draft. It's a chance to try out ideas in front of colleagues, not necessarily to present polished or even slightly polished work.

But when journalists write about conference papers, they often make it sound like these are vetted pieces of work. They're not. It's premature to draw conclusions from them, or make recommendations.

So here's my suggestion, particularly for the online media-- devote a column or section to "work in progress" and cover conference presentations there. That way your readers will be appropriately warned.

"No Effects" Studies

Education Week's got this article out about randomized trials producing "no effects." According to the article, these null findings are raising eyebrows and "prompting researchers, product developers, and other experts to question the design of the studies, whether the methodology they use is suited to the messy real world of education, and whether the projects are worth the cost, which has run as high as $14.4 million in the case of one such study."

Wow, is that ever a disappointing reaction. Here's why:

1. We should be psyched, not upset, that studies with null effects are being released. That is not always the case. Publication bias, anyone? I've often thought that studies demonstrating null effects need to be publicized even more widely than those that find positive or negative impacts. Too many places out there are at the behest of funders, and can't release null findings. Too many assistant professors don't get tenure because they "didn't find anything." Are you kidding me? If it's a current practice and you learn it doesn't produce any effects, either way, it needs to be out there. We should learn as much from null findings and "worst practices" as we do from "statistically significant" impacts and "best practices."

2. Saying that experimentation isn't suited to the "messy real world" is a cop out. It lumps many different kinds of experiments into one category-- the good, the bad, and the ugly. Field experiments, lab settings, cluster-randomized trials with volunteer districts, and student-level randomized experiments with participants selected via administrative data-- these are very different animals. Each approach has a differential potential for generalizable results (external validity) and varying levels of challenges to internal validity as well. I'll grant you, experiments that rely on volunteer samples probably can't help us much in education-- since in real life programs aren't applied to students, families or schools who volunteer--they apply to everyone. This is especially a problem when we try interventions to close achievement gaps-- African-Americans who volunteer for studies are very, very different from those who do not (Tuskegee anyone?).

3. Doing experiments well costs a LOT of money. Putting trials on tight budgets helps to ensure they aren't run well--PIs cannot build the kinds of relationships that promote treatment fidelity, cannot collect high-quality data, and cannot get inside the black box of mechanisms--and instead are stuck simply estimating average treatment effects. No drug works for everyone, and no drug works in the exact same way for everyone-- the medical community knows this, and uses larger samples to make identifying differential and heterogeneous effects possible. When is Education going to catch up?

4. One thing I do agree with this article on. The model IES is using needs some revisions. I heard William T. Grant president Bob Granger give a great talk at SREE recently, where he made the point that the usual 'try small things then scale them up' model isn't going anywhere fast. We need to know how current policies work as currently implemented-- at scale. Go after that, spend what's necessary to conduct experiments with higher internal AND external validity, and support researchers to do this who reject old models and try new things. I promise you, we'll get somewhere.

Time to Get Creative

One of the hardest things about being a scholar of higher education is working at a university. Inevitably we're seen as either pandering to the administration, or in direct warfare with it. What I can't understand is why leaders at universities don't identify and embrace their colleagues as (nearly) free sources of expertise. Those of us who study higher ed are generally a deeply committed body of folks who love nothing more than dreaming up creative ways to help colleges achieve.

Yet so often we're left out in the cold, only brought in at the last minute when it's time to give us a warning and tell us to shut up. Which, of course, makes us less effective, hurt, and generally pissed off.

My last two days were extremely long ones, extremely fraught ones...and today will be no exception. My task: figure out, in short time, highly nuanced and effective ways to increase the amount of need-based aid UW has WITHOUT promoting economic inequality. I'm glad I have a pool of talented colleagues with which to confer. Now only if I'd had more time....