To complete the questions below, you must read the article attached below written by Phillips et al. (2010) and answer questions about the research approach the authors used.
1. Which of the following (Case Study, Content Analysis, Correlation, Policy Evaluation, Qualitative Analysis) is the research design used for the study described in this article?
2. Which of the following (Quantitative, Qualitative, Mixed/Both) is the type ofempirical data analyzed for the study described in this article?
3. In 1-2 sentences, identify the participants (the sample) used for the study described in this article.
Be specific: indicate the type(s)/group(s) of participants, the total number of participants, and the demographic characteristics of the participants.
4. In 1-2 sentences, state the main finding(s)/results of the study described in this article.
5. In 1-2 sentences, state the conclusions of the study
and the implications for further study or understanding of the topic/issue.
Am J Community Psychol (2013) 51:289–298
DOI 10.1007/s10464-012-9538-2
ORIGINAL PAPER
The Evaluation of Arkansas Act 1220 of 2003 to Reduce
Childhood Obesity: Conceptualization, Design, and Special
Challenges
Martha M. Phillips • James M. Raczynski •
Delia S. West • LeaVonne Pulley • Zoran Bursac •
Laura C. Leviton
Published online: 28 June 2012
Ó Society for Community Research and Action 2012
Abstract This article describes the evaluation of the
Arkansas Act 1220 of 2003, a comprehensive legislative
proposal to address the growing epidemic of childhood
obesity through changes in the school environment. In
addition, the article discusses specific components of the
evaluation that may be applicable to other childhood
obesity policy evaluation efforts. The conceptual framework for the evaluation, research questions, and evaluation
design are described, along with data collection methods
and analysis strategies. A mixed methods approach,
including both quantitative (surveys, telephone interviews)
and qualitative (key informant interviews, records reviews)
approaches, was utilized to collect data from a range of
informant groups including parents, adolescents, school
principals, school district superintendents, and other
stakeholders. Challenges encountered with the evaluation
are discussed, as are strategies to overcome those challenges. Now in its 9th year, this evaluation has documented
substantial changes to school policies and environments
but fewer changes to student and family behaviors. The
evaluation may inform the methods of other evaluations of
childhood obesity prevention policies, as well as inform
policymakers about how quickly they might expect
implementation of such policies in their own states and
localities and anticipate both positive and adverse
outcomes.
M. M. Phillips (&) J. M. Raczynski D. S. West L. Pulley
Z. Bursac
Fay W. Boozman College of Public Health, University of
Arkansas for Medical Sciences, 4301 W. Markham Street, Mail
Slot 820, Little Rock, AR 72205, USA
e-mail: MMPhillips@uams.edu
L. C. Leviton
Robert Wood Johnson Foundation, Princeton, NJ, USA
Keywords Childhood obesity Evaluation
Evaluation design Evaluation methods Policy
Introduction
This article describes the design and implementation of a
9-year evaluation of an ambitious state policy to reduce
childhood obesity through a variety of changes in nutrition
and physical activity policies and practices in public
schools. National obesity levels have increased dramatically among adults as well as children and adolescents over
the past two decades. Recent studies indicate that 68 % of
adults in the United States are overweight or obese (Flegal
et al. 2010) and 15 % of children (ages 6–11 years) and
adolescents (ages 12–19 years) were overweight (Centers
for Disease Control and Prevention 2008). In Arkansas, the
statistics are even more alarming. Data from Arkansas’ six
years of annual body mass index (BMI) assessments in
public schools indicate that 21 % of Arkansas children are
obese (at or above the 95th percentile of BMI for age and
gender), and another 17 % are overweight (age-and-gender-specific BMI between the 85th and 95th percentiles)
(Arkansas Center for Health Improvement 2009).
School systems and legislators across the US are
responding to concerns about childhood obesity with policy
modifications, such as changing vending machine availability and contents within school concessions, measuring
and reporting children’s BMI to parents, changing cafeteria
selections and food preparation methods, and modifying
physical education requirements (Koplan et al. 2006).
Many of these policy changes are the result of either local
initiatives or statewide mandates; however, describing the
quantity and diversity of these school level changes proves
difficult, if not impossible, to catalog since local initiatives
123
290
are rarely reported in the literature and natural variation
often occurs with statewide initiatives. As of 2008, legislative bodies in the majority of the 50 states had considered
legislation pertaining to the areas of nutrition, physical
activity, and/or physical education in schools (Dixon et al.
2009; National Conference of State Legislatures 2010).
Moreover, federal legislation (‘‘S. 2507: Child Nutrition
and WIC Authorization Act of 2004,’’ 2004; ‘‘S. 3307–
Healthy, Hunger-Free Kids Act of 2010,’’ 2010) has
established requirements for school wellness policies,
increased physical activity in schools, and established
requirements for cafeteria food offerings that comply with
the Dietary Guidelines for Americans.
Arkansas was among the first states to address childhood
obesity through multi-component legislation, and the history of Arkansas Act 1220 of 2003 (‘‘Arkansas Act 1220 of
2003‘‘, 2003) has been described elsewhere (Raczynski
et al. 2009; Ryan et al. 2006). In brief, Act 1220 included
two specific mandates: the annual measurement of BMI for
all public school children and the reporting of the BMI and
associated health risks to parents; and the restriction of
student access to vending machines during the school day
in elementary schools. The Act also created at least three
important mechanisms to foster additional policy change.
First, the Act required the Arkansas Department of Health
to employ community health promotion specialists to assist
schools in implementing nutrition and physical activity
programs. Second, it created the statewide Child Health
Advisory Committee, with representation from specified
stakeholder groups, to review the evidence and develop
nutrition and physical activity standards concerning: foods
sold individually in school cafeterias; competitive foods
offered through vending machines, student stores, fundraisers, food carts, or concessions; continuing professional
development of food service staff; expenditure of funds
derived from competitive food and beverage contracts; and
physical education and activity. Finally, the Act required
school districts to establish district-level nutrition and
physical activity advisory committees with broad membership to help oversee the implementation of statewide
policies and to develop locally-relevant policies and practices for schools within that district.
The Arkansas legislation has served as a model for
childhood obesity prevention policies at the national, state,
and local levels throughout the country. Most notably, the
Act was cited in the 2004 reauthorization of the national
Child Health and Nutrition Act in a requirement that school
districts participating in federally-funded school meals
programs should adopt local wellness policies addressing
factors in school environments that affect student diets and
physical activity. Further, the passage of Act 1220 provided a unique opportunity to evaluate the implementation
of a broad-reaching public health initiative, with a goal of
123
Am J Community Psychol (2013) 51:289–298
understanding how such activities are implemented in
communities and to examine the natural variation in policy
implementation.
Evaluation of such public policy initiatives is necessary to
ensure that policies and programs are effective and resources
are best utilized. The evaluation of the Arkansas initiative
may inform policymakers, researchers, public health professionals, and school personnel about the facilitators and
barriers involved in implementation of such policies in their
own states and localities and the potential for improvements
in childhood obesity rates across our nation. This paper
describes the conceptualization, evaluation design, data
analysis approach, and dissemination processes of the
Arkansas Act 1220 of 2003 Evaluation Project which has
been funded by the Robert Wood Johnson Foundation.
Methods
Conceptual Framework for Evaluation of Act 1220
Figure 1 depicts the conceptual framework upon which the
evaluation is based. Briefly, the framework posits that, prior to
the enactment of Act 1220, there were existing nutrition and
physical activity policies within schools and school districts.
The passage of Act 1220 stimulated the enactment of new
policy recommendations by the Child Health Advisory
Committee and school district advisory committees; all were
expected to lead to changes in school environments. Policymakers expected that these changes would increase families’
and students’ knowledge about physical activity and nutrition,
produce more favorable attitudes toward healthy food and
physical activity, increase students’ physical activity, and
decrease their consumption of unhealthy, calorie-dense foods
in schools. The ultimate goal of Act 1220, as expressed by its
legislative supporters, is to reduce the percentage of Arkansas
youth who are overweight or obese.
It is important to note that the conceptual framework
does not include monitoring of compliance with portions of
the Act or with rules and regulations instituted later. This
decision was based on two primary factors. First, school
compliance with rules and regulations is the responsibility
of the Arkansas Department of Education. Second, to
include compliance as a specific component of the evaluation would likely have had a substantial effect on the
willingness of school personnel to complete surveys with
truthful responses, thereby jeopardizing the overall validity
of the evaluation.
Evaluation Design
The legislation, which applied to all public schools in the
State, in effect created a single, statewide intervention group
Am J Community Psychol (2013) 51:289–298
POLICY
INPUTS
Existing State & District
Policy
– BMI Measurement
– Vending machines
– Competitive foods
– Physical Education
– Physical Activity
– Nutrition Education
ACT 1220
– BMI Measurement
– Vending machines
Elementary
Jr/High
– Competitive foods
– Physical Education
– Physical Activity
– Nutrition Education
– School District Nutrition
& Physical Activity
Committees
POLICY
CHANGE
Local policies
adopted in
advance of
Act 1220
mandates
Policies adopted to
comply with
mandates resulting
from Act 1220
291
KNOWLEDGE
& INTENT
BEHAVIOR
CHANGE
– Parental knowledge
of childhood weight
issues
INDIRECT EFFECTS
(Family behaviors)
– Change in family physical
activity
– Parental intent to
change physical
activity, calorie
intake, types of food,
snacking patterns,
and/or TV watching
– Change in family food
choices and/or preparation
– Knowledge among
school personnel of
childhood weight
issues and school’s
role in improving
health status
Local policies
adopted in
excess of (more
stringent than)
Act 1220
mandates
Environmental change in
schools
-Increased participation in
healthy weight programs
DIRECT EFFECTS
-(Adolescent behaviors)
– Increased physical activity
– in school
– out of school
– Decreased time spent
watching TV, playing
computer games
– Modified eating behaviors
– Fewer snacks eaten
– Healthier snack
choices
– Increased participation
in school lunch program
– Fewer fast food meals or
snacks
– Healthier beverage
choices
OUTCOME
Increased
proportion
of youth
within
healthy
weight
range
Fig. 1 Conceptual framework underlying evaluation of Act 1220 initiatives
and eliminated the possibility of controlled experimental or
quasi-experimental study designs. Randomization of
schools to intervention and control conditions was not
possible, and evaluators determined that no other potential
control group (e.g., private schools in Arkansas, public
schools in another state) was sufficiently comparable and
appropriate. Because of the multifaceted nature of the Act
1220 policy implementation and because it is embedded in
open, complex systems, conventional controls are not feasible. Thus, the design for the evaluation of Act 1220 may be
most accurately described as monitoring of both implementation and outcome, using regular assessments to capture and report the evolution of the intervention, the
processes of its implementation, and the results, intended
and unintended, of those processes. The design also allows
for an examination of natural variations in policy implementation and outcomes among schools. The evaluation’s
primary stakeholders include: legislators; the Governor;
Departments of Education and Health staff; the public,
including parents and school personnel; and members of the
state Child Health Advisory and district-level Nutrition and
Physical Activity Advisory Committees who advise the
Department of Education and local school districts,
respectively.
The evaluation encompasses a multi-component assessment plan. Grounded in the conceptual framework, the
evaluation and measurement plan includes: (1) a baseline
assessment of schools and school districts in 2004, prior to
implementation of the Act’s policy changes, to characterize
policies and practices previously in place; (2) an ongoing
qualitative evaluation, to document, describe and understand
the processes involved in conceptualizing, enacting and
implementing the Act at the state and local levels, and subsequent policy initiatives that emanate from the Act; (3) a
quantitative impact assessment to document changes in
school environments and policies and in knowledge, attitudes and behaviors concerning childhood obesity, nutrition
and physical activity in both families and students; and (4)
monitoring of any changes in the BMI distribution of
Arkansas public school students, using the BMI data collected by schools. Thus, multiple informant groups are being
used to create a multi-dimensional picture of the implementation and outcomes of Act 1220.
Hypotheses and Research Questions
Hypotheses being addressed are that the implementation of
Act 1220 initiatives will result in changes in school environments; in knowledge, attitudes, and beliefs; and, most
importantly, in behavior change among students and their
families that may ultimately increase the proportion of
Arkansas youth within the healthy weight range, based on
guidelines for BMI-for-age-and-gender percentiles (Ogden
and Flegal 2010). Although affecting the weight status of
Arkansas’ youth (reducing the proportion of youth who are
overweight or obese) is the ultimate objective of Act 1220,
this type of outcome should likely occur over an extended
timeframe. Thus, the evaluation team monitors the
123
292
proportion of Arkansas youth within healthy weight ranges,
using the statewide BMI screening data obtained from
schools. However, we emphasize that the evaluation
focuses on documenting environmental changes at the
school level, and changes in knowledge, attitudes, beliefs
and behavior among students and families, that if implemented consistently and persistently, should logically lead
to changes in weight status. Additionally, the on-going
process and impact evaluation of the Act allows for the
evaluation of adverse outcomes which could not be determined from BMI screening data alone. With these overall
goals in mind, the primary and secondary questions to be
addressed in the evaluation are summarized in Table 1.
As noted, the Act 1220 evaluation uses both qualitative
and quantitative methods to obtain information from multiple sources. Records review, key informant interviews,
surveys, and telephone interviews all contribute unique and
complementary data that inform a comprehensive, multidimensional evaluation.
Qualitative Elements of the Act 1220 Evaluation
The evaluation obtains ongoing qualitative data from
record reviews and key informant interviews to characterize stakeholders’ understandings of the key issues of concern and enrich quantitative data. These data sources offer
insight into the viewpoints of key individuals engaged in
implementing Act 1220. Because policies related to Act
1220 were phased in over time and districts/schools are
continually revising policies, we are able to monitor
stakeholders’ perceptions of the policy development process and track the successes and challenges they encounter
in the execution of Act 1220. The evaluation team gathers,
reviews and summarizes all relevant records from the Child
Health Advisory Committee, the Department of Education
(policies and procedures), State Board of Education
(meeting minutes, other documents relevant to Act 1220
policies), legislative oversight committees (meeting minutes, proceedings), and the Arkansas Legislature (proceedings relevant to continued status of Act 1220 and
related activities). We also access other information, such
as responses to telephone hot-lines, staff activity logs, and
records from schools concerning the completion of the
BMI assessment (e.g., number of students assessed), to
document the process of establishing and implementing
Act 1220 policies.
The evaluation team completes annual key informant
interviews with members of special stakeholder groups,
including: members of the Child Health Advisory Committee, Department of Education, State Board of Education, and
legislative oversight committees; legislative sponsors of the
original Act 1220 legislation; other members of the Arkansas
legislature; principals and superintendents; school nurses;
123
Am J Community Psychol (2013) 51:289–298
Table 1 Primary and secondary questions to be addressed by Act
1220 evaluation
Primary questions
1. What impact did the enacted policies and procedures have, over
the 9 measured years, on:
(a) the school environments experienced by youth enrolled in
Arkansas’ public schools;
(b) the knowledge, attitudes, and beliefs of parents and students
regarding healthy physical activity and nutrition practices;
(c) health care providers engaged in providing support to
families;
(d) self-reported changes in physical activity and nutrition
among Arkansas youth and families?
2. What was the trend in weight status among Arkansas youth
enrolled in public schools over the period of the 9-year
evaluation?
3. What potentially negative and unintended consequences
occurred (e.g., eating disorders, weight-based teasing) occurred
as Act 1220 was being initiated?
Secondary questions
1. What were the baseline policies regarding nutrition and physical
activity in Arkansas public schools, prior to the initiation Act
1220 initiatives?
2. What policies were ultimately implemented by local school
districts and schools?
3. What was the level of compliance among local schools with the
stated policies?
4. What challenges did the Child Health Advisory Committee
(CHAC), the State Department of Education, and the local
school districts face in the implementation of the policies and
procedures stimulated by Act 1220? How were those challenges
addressed?
5. What were the reactions of stakeholders (families, students,
health care providers, school personnel) to the policies enacted
because of Act 1220, and did these reactions change over time?
6. What were the natural variations in policy implementation
among schools, with respect to demographic characteristics,
support needed for implementation, fidelity to recommended
protocols, and level and timing of policy adoption (earlier than
or beyond those required by Act 1220)?
7. What were the natural variations in the choices of strategies
chosen by families who attempted to make changes in their
physical activity and nutrition practices?
8. What changes actually occurred in BMI levels of children who
were identified as overweight or obese at baseline?
community-based physicians who may be treating students
as a follow-up to the BMI screening program; and members
of the local Nutrition and Physical Activity Advisory Committees/Wellness Committees. The team completes these
interviews by telephone, records them on audiotape, and
transcribes them prior to analysis to ensure anonymity and
accuracy in documenting responses. The team then reviews
transcripts and summarizes key themes, concerns, and
messages for each informant group. More stringent qualitative analyses using thematic content analytic methods are
employed for specific topical analyses as necessary.
Am J Community Psychol (2013) 51:289–298
Further, during the first five years of the evaluation,
project staff, with the assistance of a clipping service,
monitored media reports (e.g., articles, interviews, letters to
editors, informational pieces placed in newspapers and
aired on television and radio stations throughout the state)
related to the Act, BMI measurement, and childhood
obesity overall. These data provided a different perspective
on public sentiment and areas of concern.
Quantitative Elements of the Act 1220 Evaluation
Quantitative data derive from annual surveys of school
principals and superintendents as well as annual interviews
of cross-sectional samples of families that include both a
public school student and one of the students’ parents/
guardians as respondents. These surveys and interviews
document changes in policies, environments, knowledge,
attitudes, and behavior. Baseline surveys and interviews
were completed in the Spring of 2004, before policy
changes were made and before parents had received the
first child health report from the schools. Follow-up data
collection occurs in the Spring and early Summer of each
subsequent school year.
Surveys of school district superintendents and school
principals monitor changes in school and district policies
and programs relating to nutrition, physical education, and
physical activity. Questionnaires distributed to both groups
assess: (1) policies and procedures relating to physical
activity, physical education, child nutrition programs,
health and nutrition education, vending machines, competitive foods, and health assessments within the district;
and (2) the financial impact of vending machine and
competitive food changes.
Questionnaires are distributed by mail with a stamped,
self-addressed return envelope. A second questionnaire is
mailed to respondents failing to return the questionnaire
within four weeks. If the second questionnaire is not
returned, project staff call the principal or superintendent to
encourage completion and mail or fax another copy of the
questionnaire as necessary. These procedures have resulted
in return rates across evaluation years to date ranging from
64 to 78 % among superintendents and 72 to 79 % among
principals.
Telephone surveys of parents and youth enrolled in
Arkansas public schools are completed annually. Crosssectional samples are selected each year using a multistage
stratified random sample procedure, which has been described in detail elsewhere (Bursac et al. 2005). To acquire a
sample of parent and child participants, target schools are
randomly selected (n = 113 at baseline; n ranged from 477
to 496 in the 6 follow-up years) from within 45 strata representing geographic regions of the state (5 regions), grades
293
served (elementary, middle/junior, high schools), and
enrollment size (small, medium, large). For each target
school, listed phone numbers in the zip code areas included
in the school’s attendance zone are sampled to identify
households with a child attending the school. Goals for
completed interviews are one percent of each school’s student enrollment. Interviews are completed by a survey
research firm, using computer-assisted telephone interviewing technology.
The parent interview includes questions to document: (1)
parental knowledge of and attitudes concerning the BMI
measurement process, vending machine policy changes,
school policies concerning competitive foods, and other
nutrition and physical activity policies as may be recommended by the Child Health Advisory Committee; (2)
parental reports of weight-based teasing of the student,
which might be escalated by the BMI measurement and
reporting process; (3) parental knowledge, attitudes and
beliefs about health risks associated with a child’s weight
status; (4) family physical activity, dietary intake, and food
preparation methods; (5) family rules and habits concerning
television watching and computer use; (6) parental intent to
change family physical activity and dietary intake patterns;
(7) parental intent to enroll school-aged children in weight
control programs (including program type); (8) parental
intent to seek additional information from health care providers; and (9) parental report of child’s height, weight, age
and gender (to calculate weight status for age and gender)
and parental perception of their child’s current BMI category. Follow-up interviews have included questions related
to receipt of the BMI report and beliefs about the role of
schools in helping families maintain healthy weights for
children.
For each consenting household, the parent informant is
interviewed first about family patterns and then about the
child. In households with more than one child attending the
targeted school, a single student within the household is
randomly selected. If the index child is 14 years of age or
older, the interview is completed with the child when
consent and assent are provided; for children, younger than
14 years, a proxy interview is conducted with the parent.
The student-focused interviews include questions to document: (1) student perceptions of the process of BMI
measurement, including privacy and confidentiality of
information; (2) student reports of teasing; (3) physical and
sedentary activity within the previous 7 days; (4) dietary
intake and other dietary behavior, including snacking
behavior at home and school, participation in school meal
programs, fast food meals, beverage intake (regular and
diet soda, fruit juices, milk), and frequency of vending
machine use during school hours; and (5) recent changes in
physical activity or diet.
123
294
Extraction of Other Data
Characteristics of schools and school districts are useful in
explaining potential variation in either implementation or
outcomes. Data gathered from existing data sources (i.e.,
Department of Education databases) include enrollment,
grades served, student–teacher ratio, ethnic diversity, revenues from diverse sources, proportion of students receiving
free or reduced price meals, and school improvement status.
Data and Statistical Analyses
Data analysis strategies include both qualitative analysis of
records and key informant interviews and quantitative
analysis of survey data. Survey data from parents, youth,
principals, and superintendents are first analyzed with
descriptive and univariate analytical techniques (Chi-square
and t test), followed by multivariate methodologies (linear
regression, or logistic regression) as appropriate depending
on the characteristics of the outcome and predictor variables
being analyzed. Student, family, and school data are analyzed cross-sectionally, assessing behaviors, characteristics,
and policies annually, and highlighting changes in particular
behaviors, characteristics, or policies of interest over time.
The evaluation employs weighted analyses of parent and
youth interview data, with a functional sample size representative of approximately 450,000 students coming from
390,000 households, to generate estimates that can be generalized to the overall state public school population. To
complement these cross-sectional analyses, we have linked
school policy data over time by school and school district,
allowing the completion of longitudinal policy change
analyses, with school being the primary unit of analysis.
Indices of nutrition and physical activity policy are being
constructed to facilitate the analysis of policy implementation over time. Principal component and factor analytic
methods are being used to identify naturally occurring
clusters of policy and the change in those policy clusters as
implementation unfolded.
Human Subjects Approval
All data collections plans and protocols were reviewed and
approved by the University of Arkansas for Medical Sciences’ Institutional Review Board. A waiver of written
informed consent was approved for all data collection
processes.
Results
Annual findings from this ongoing evaluation are summarized in some detail in annual reports (Phillips et al. 2005,
123
Am J Community Psychol (2013) 51:289–298
2006, 2007, 2008, 2009, 2010, 2011). In brief, the evaluation has documented substantial changes to nutrition and
physical activity policies and practices at the school district
and individual school levels. For example, in 2004 only
26 % of schools indicated that they modified recipes to be
lower in fat or were working to provide more fruits, vegetables or fiber (Phillips et al. 2011). That percentage
increased gradually to 42 % in 2010 (OR = 2.537; 95 %
CI: 1.903, 3.383; Wald Chi-square = 40.29; p B 0.0001)
(Phillips et al. 2011). Similarly, the percentage of schools
offering whole milk, either white or chocolate, has
decreased significantly since 2004, while the proportion of
schools offering low-fat or skim milk options has increased
significantly (see Table 2). Schools have significantly
reduced access to competitive foods and beverages as well.
The percentage of schools with vending machines on
campus has remained essentially constant at 78–80 % since
2006; however, both beverage and food machines are significantly more likely to be located only in staff lounges
and, thus, not available to students (see Table 3). Vending
machines available to students are less likely to be located
in cafeterias, gymnasiums, and snack bars or school stores
than they were in 2004, and the contents of those machines
are more likely to include healthy options than before the
passage of Act 1220 (see Table 3).
Changes to physical activity policies and practices
include the establishment of policies prohibiting the use of
physical activity as punishment for bad behavior in physical education (77 % of schools reported such policies in
2004, compared to 83 % in 2010; OR = 1.351; 95 % CI:
1.010, 1.808; Wald Chi-square -4.10; p B 0.05) and/or the
punishment of bad behavior in another class by excluding
students from physical education (84 % in 2004, 93 % in
2010; OR = 2.675; 95 % CI: 1.825, 3.921; p B 0.0001) or
from recess (42 % in 2004, 54 % in 2010; OR = 1.970;
95 % CI: 1.522, 2.552; Wald Chi-square = 26.44; p B
0.0001). Further, the proportion of school districts requiring that elementary schools provide students with regularly-scheduled recess has risen from 58 % in 2004 to 70 %
in 2010, and the proportion of districts requiring that student fitness be measured on a regular basis has grown from
26 % in 2004 to 44 % in 2010, although the differences
across years were not statistically significant for either
comparison.
In general, concommitant changes to nutrition and physical activity behaviors in the families of students attending
Arkansas public schools have not been noted. Among adolescents with access to vending machines in schools, reported purchases from beverage machines on campus have
declined from 9.2 purchases per month in 2004 to 2.0 purchases per month in 2010 (parameter estimate = -6.43;
standard error = 0.92; t = -7.01, df = 330; p B 0.0001);
the percentage who report making no purchases in the past
Am J Community Psychol (2013) 51:289–298
295
Table 2 Percentage of Arkansas public schools reporting offering of milk options in school cafeteria, 2004–2010
Milk options offered
in school cafeteria
2004
2005
2006
2007
2008
2009
2010
Odds
ratio
(OR)a
95 % Confidence
interval for ORa
Wald
Chisquarea
p value*
Whole white
77.6 %
73.6 %
57.3 %
45.2 %
41.8 %
33.9 %
33.5 %
0.142
0.111, 0.183
232.05
B0.0001
Whole chocolate
36.3 %
29.8 %
22.5 %
15.5 %
15.8 %
12.9 %
13.8 %
0.275
0.208, 0.363
82.91
B0.0001
Low-fat white
Low-fat chocolate
91.7 %
69.3 %
94.2 %
75.7 %
94.3 %
80.0 %
96.0 %
84.1 %
95.6 %
84.0 %
95.3 %
83.4 %
94.6 %
82.6 %
1.438
2.069
0.920, 2.247
1.586, 2.698
2.54
28.80
NS
B0.0001
Skim white
26.1 %
31.8 %
39.4 %
44.5 %
41.0 %
42.2 %
39.6 %
1.853
1.459, 2.352
25.65
B0.0001
Skim chocolate
9.0 %
13.2 %
18.4 %
23.7 %
24.0 %
25.8 %
26.4 %
4.052
2.942, 5.580
73.46
B0.0001
a
For variable Year (dichotomous comparing baseline to most recent year) in multivariate regression analysis; adjusted for school level
(elementary, middle, high school), % of free or reduced price meal participants, urban/rural locale, and total enrollment
month increased from 22 % in 2004 to 65 % in 2010
(OR = 6.701; 95 % CI: 3.421, 13.123; Wald Chisquare = 30.76; p B 0.0001). However, adolescents have
not reported changes in the frequency of soda consumption
or of visits to fast-food restaurants (Phillips et al. 2011).
While awareness of childhood obesity and of the risks faced
by a child who is overweight or obese has increased among
parents, those same parents do not report making substantial
changes to family eating or physical activity behaviors
(Phillips et al. 2011).
Although desired positive impacts on family behaviors
have not been noted, neither has the evaluation shown any
evidence of widespread negative consequences, e.g.,
increases in weight-based teasing or increases in unhealthy
diet behaviors (Krukowski et al. 2008; Phillips et al. 2011).
The proportion of adolescents expressing concern about
their weight remained steady at approximately 25 %,
exercising for weight control at 60 %, and dieting at 40 %
(Phillips et al. 2011). The proportion of adolescents
reporting being victims of weight-based teasing decreased
over time (12 % in 2004; 6 % in 2010), as did the proportion reporting that they took diet pills (6 % in 2004;
3 % in 2010) (Phillips et al. 2011).
Discussion
Evaluation of public policy is important for a number of
reasons. Clearly, evaluation serves to identify the degree to
which a policy attains its objectives, benefits the target
population, and/or reveals unintentional harm that may
have occurred. Policy evaluation also supports the development of effective interventions and provides insight into
which approaches work best for certain groups of individuals or in certain situations. In addition, policy evaluations can contribute to the improvement of the specific
policy under study.
Each of these purposes has been addressed and achieved
at least to some extent by the Act 1220 evaluation. For
example, the evaluation team reports annually to the
Arkansas General Assembly regarding the ongoing
implementation of Act 1220 and findings from the most
recent assessments. These oral and written reports (Phillips
et al. 2005, 2006, 2007, 2008, 2009, 2010, 2011) have been
reassuring to most policymakers, documenting that changes in school environments are occurring, that there is
widespread parental support for changes to offerings in
school-based vending machines, and that feared negative
consequences, such as unhealthy weight control behaviors
and weight-based teasing, have not increased over baseline
levels (Krukowski et al. 2008; Phillips et al. 2011; Raczynski et al. 2009). These findings have been used by
legislators and policy advocates to inform subsequent
debates about changes to Act 1220 itself and to regulations
emanating from Act 1220 that affect nutrition and physical
activity policies and practices within schools. Other findings from the evaluation, e.g., the accuracy of parental
perception of child weight status and improvements in
accuracy after receiving a letter from the school providing
results from the BMI assessment (West et al. 2008), have
been disseminated to scientific and policy communities,
increasing the body of knowledge concerning the documented benefits and limitations of legislative approaches to
combating childhood obesity. Additional data are being
prepared for dissemination to a range of audiences to further inform debate and policy development in Arkansas
and in other states, as national efforts to combat the
growing problem of childhood obesity continue.
As with all evaluations, it is important to recognize both
the strengths and weaknesses of the evaluation design and
recognize the limitations that present from the program or
policy being evaluated. The evaluation of Act 1220 had
many strengths but also some weaknesses. Monitoring such
a far-reaching policy in real time over the course of 9 years
has already proved advantageous in offering a wealth of
new knowledge. For example, our evaluation has shown,
among other things, that school change can be expected to
occur incrementally over a period of three to four years and
123
296
Am J Community Psychol (2013) 51:289–298
Table 3 Percentage of schools reporting location and contents of vending machines in Arkansas public schools, 2004–2010
2004
2005a
2006
2007
2008
2009
2010
Odds
ratiob
95 %
Confidence
intervalb
Wald
Chi-square
statisticb
p valueb
84.6
NA
79.7
80.3
79.4
80.0
77.6
0.590
0.441, 0.789
12.66
B0.001
Location of vending machines on school campuses
Percentage of schools with vending machine
on campus (VM schools)
Percentage of VM schools with beverage machine
99.7
NA
99.7
100.0
99.5
99.4
100.0
NA
NA
NA
NA
Percentage of VM schools with food machine
60.2
NA
51.0
52.4
54.1
52.5
54.0
0.653
0.505, 0.845
10.55
NS
Percentage of VM schools with beverage
machine available only in staff/teacher lounge
36.0
NA
47.8
46.8
50.9
52.0
57.9
4.565
3.167, 6.577
66.31
B0.0001
Percentage of VM schools with food machine
available only in staff/teacher lounge
52.3
NA
61.7
69.3
66.4
71.5
75.4
3.205
2.098, 4.898
28.99
B0.0001
Beverage machine
13.3
NA
10.2
9.3
8.4
8.4
7.0
0.542
0.352, 0.837
7.65
B0.01
Food machine
6.1
5.0
3.7
1.8
2.6
1.7
0.317
0.110, 0.912
4.54
B0.05
24.1
23.3
18.3
18.8
13.8
0.322
0.230, 0.453
42.74
B0.0001
7.6
5.6
4.2
3.8
3.8
0.263
0.124, 0.559
12.08
B0.001
Vending machine locations available to students
Cafeteria
Gymnasium
Beverage machine
30.3
Food machine
10.8
NA
Snack bar/school store
Beverage machine
8.9
Food machine
14.2
NA
4.0
2.4
2.8
1.5
1.3
0.144
0.063, 0.329
21.22
B0.0001
5.8
4.0
6.1
1.8
2.4
0.186
0.081, 0.427
15.69
B0.0001
Hallways
Beverage machine
–
Food machine
–
NA
18.2
18.8
18.5
17.4
17.0
0.909
0.643, 1.286
0.29
NS
12.6
10.2
12.1
14.4
9.6
0.867
0.479, 1.568
0.22
NS
Contents of vending machines available to students
Chocolate candy
85.2
NA
73.9
70.2
75.0
66.7
70.4
0.442
0.196, 0.999
3.85
B0.05
Other candy
88.6
NA
76.5
76.8
69.6
67.6
74.1
0.319
0.132, 0.771
6.43
B0.05
Cookies
91.5
NA
90.8
98.8
91.3
88.3
87.5
0.368
0.110, 1.229
2.64
NS
Cakes, pastries
60.8
NA
59.3
57.0
52.7
55.4
50.0
0.568
0.283, 1.142
2.52
NS
Chips
91.5
NA
80.3
86.7
83.7
82.9
79.6
0.287
0.101, 0.817
5.47
B0.05
Sodas, other sugar-sweetened beverages
83.8
NA
83.0
81.4
73.5
58.4
55.6
0.190
0.120, 0.301
50.25
B0.0001
Fruit drinks
76.9
NA
81.2
69.6
63.2
58.4
44.0
0.234
0.155, 0.353
47.67
B0.0001
Low-fat, low-sugar cookies
23.3
NA
58.3
48.1
58.7
56.2
57.1
4.057
1.939, 8.491
13.82
B0.001
Low-fat crackers
26.1
NA
55.3
50.6
54.3
59.2
59.6
3.532
1.748, 7.133
12.37
B0.001
Low-fat, low-sugar pastries
16.5
NA
37.4
24.4
22.0
30.6
36.4
2.129
0.988, 4.590
3.72
NS
Low-fat chips
43.2
NA
84.6
77.8
76.9
82.4
73.2
4.765
2.192, 10.360
15.53
B0.0001
100 % Fruit juices
53.2
NA
70.9
60.6
56.2
65.0
62.1
1.329
0.901, 1.961
2.06
NS
Water
81.9
NA
92.9
91.6
92.1
94.6
95.5
5.664
2.624, 12.225
19.52
B0.0001
a
Vending data not available for 2005 because of inadvertent error in skip patterns in questionnaire
b
For variable Year (dichotomous comparing baseline to most recent year) in multivariate regression analysis; adjusted for school level (elementary, middle, high
school), % of free or reduced price meal participants, urban/rural locale, and total enrollment
then to reach a maintenance period (Phillips et al. 2011),
that the increased focus on weight may not be associated
with increased dieting behavior or weight-based teasing
(Krukowski et al. 2008) by teens, and that states might
expect to encounter delayed opposition to legislative initiatives, even when they were broadly accepted initially
(Phillips et al. 2008).
Conducting an evaluation of an intervention in a naturalistic setting brings some challenges. The major difficulty
has been the absence of a stable ‘‘intervention.’’ As in
many new policy evaluations, relatively few components
123
were well specified and made operational in advance
(Bardach 1978; Lipsey 1993). The framers of the legislation did not anticipate an evaluation component and, thus,
did not include evaluators in the planning process. Predictably, the evaluation team was presented with a challenge of designing an evaluation plan prior to knowing the
specific outcomes of the law which were determined over
the course of a year by the Child Health Advisory Committee. The evaluation team relied on the scientific literature and the scientific expertise of the team to devise a plan
that would account for the expected but unknown
Am J Community Psychol (2013) 51:289–298
components of the law and to capture within a baseline
assessment the range of variables that would enable the
evaluation of unknown policies from baseline to follow-up.
In addition, the implementation of policies was expected to
vary substantially across the more than 1,100 school
intervention sites across the state. Our response was to
develop the conceptual model, anticipating as much as
possible the range of policy changes that might occur, the
variety of school characteristics that might affect implementation and outcomes, and the full range of family and
student behaviors that might be affected by the Act 1220
initiatives. This conceptual model served then as the
foundation for the design of the evaluation, measurement
strategies, and assessment protocols.
The absence of a comparison group was a challenge, at
least to the extent that we wanted to test causal relationships between the law and changes in attitudes, behavior
and health status. We could not identify a state that, in our
estimation, was sufficiently similar to Arkansas but without existing school-based childhood obesity policies, to
serve as a comparison state. Further, it was likely that
such a state would not have remained policy-free for the
extended period of the evaluation and thus would not have
offered a viable control for the absence of obesity prevention policies. Had such a comparison state been identified it might have provided some information against
which to evaluate the impact of the Arkansas law, but
there would be a host of alternative explanations for any
differences between the states that were seen over time,
limiting the overall utility of such a comparison. Faced
with a statewide intervention for which no suitable comparison group could be identified, the evaluation team
chose to focus on understanding the natural variations
within the state in opinion, behavior, and policy that were
present at baseline and changes in those elements that
might occur in the years following the passage of Act
1220. The advantages of acquiring knowledge about the
implementation of this policy outweigh the limitations in
our ability to determine a specific causal relationship
between a specific policy and outcomes.
Mixed methods assessment and analytic strategies were
designed for adaptation to policy implementation as it
unfolded over time and to capture additional, previously
unanticipated, information that might suddenly become
relevant. Thus, the evaluation of Act 1220 is conceptualized as a staged evaluation, with additional components
integrated as necessary. Examples of this flexibility are
found in the addition of interviewing additional key
informant groups (e.g., school nurses, community health
promotion specialists, members of the local Nutrition and
Physical Activity Advisory Committees) at the first annual
follow-up assessment, reflecting the new constituencies
that became available and relevant during the year.
297
Conclusions
Evaluation of public policy is essential to the creation of
effective policy and interventions. Both formative and
outcome evaluation strategies serve to assure that scarce
resources are applied effectively, critically important target
groups are reached, and unintended harm does not occur.
Those who would mount rigorous evaluations of policy
interventions are challenged, however, when implementing
rigorous methods in the real environments of naturalistic
experiments within which these policies operate. Policy
interventions often take place in complex, multi-system
settings that include multiple influences interacting much
as intersecting gears in an engine—moving one another in
complex ways to achieve final outcomes. Evaluations in
such settings must be carefully designed and thoughtfully
implemented to be flexible enough to respond to the
changing environment, rigorous and consistent enough to
be credible and effective in documenting impact, and sufficiently comprehensive and pragmatic to address issues of
concern for multiple stakeholder groups.
The ongoing evaluation of Arkansas’ Act 1220 is an
example of just such a policy evaluation. Guided by scientific expertise and theory, and designed to be both consistent and flexible, the evaluation is able to provide data to
answer questions posed by decision-makers inside and
outside the state, to inform the implementation of the Act’s
initiatives and those that have emerged because of or in
conjunction with the Act, and to monitor the occurrence of
both positive and potentially harmful outcomes in a timely
manner. We expect that findings related to the natural
variation in implementation of school policy changes and
those related to lessons learned from highly successful
schools will be most informative for those seeking to
reduce the burden of childhood obesity in their communities. Additional information about current and future findings from this project can be found on the project’s website
at www.uams.edu/coph/childhoodobesity.
Acknowledgments This work was supported by the Robert Wood
Johnson Foundation [grant numbers 30930, 51737, 60284, 61551].
The preparation of this manuscript has also been supported by the
Arkansas Prevention Research Center (U48 DP001943).
References
Arkansas Act 1220 of 2003 20-7-133-135 (2003).
Arkansas Center for Health Improvement. (2009). Year six assessment of childhood and adolescent obesity in Arkansas (Fall
2008–Spring 2009). Little Rock, AR.
Bardach, E. (1978). The implementation game: What happens after a
bill becomes law. Cambridge: MIT Press.
Bursac, Z., Phillips, M., Gauss, C. H., Pulley, L., West, D., &
Raczynski, J. (2005). Arkansas Act 1220 evaluation: Multi-stage
stratified surveys with PPS sampling. Paper presented at the
123
298
American Statistical Association, Section on Health Policy
Statistics, Alexandria, VA.
Centers for Disease Control and Prevention. (2008). Overweight and
obesity. Retrieved 22 May, 2008, from www.cdc.gov/nccdphp/
dnpa/obesity/index.htm.
Dixon, L., Waldron, B., Linton, T., & Nelson, H. (2009). Balance:
2008 Mid-Year report. Princeton, NJ: Robert Wood Johnson
Foundation.
Flegal, K., Carroll, M., Ogden, C., & Curtin, L. (2010). Prevalence
and trends in obesity among US adults, 1999–2008. JAMA,
303(3), 235–241.
Koplan, J. P., Liverman, C. T., Krock, V. I., & Wisham, S. L. (2006).
Progress in preventing childhood obesity: How do we measure
up?. Washington, DC: Institute of Medicine, National Academies Press.
Krukowski, R. A., West, D. S., Siddiqui, N. J., Bursac, Z., Phillips, M.
M., & Raczynski, J. M. (2008). No change in weight-based
teasing when BMI screening for childhood obesity in conducted
in schools. Archives of Pediatric and Adolescent Medicine,
162(10), 936–942.
Lipsey, M. W. (1993). Theory as method: Small theories of
treatments. New Directions for Program Evaluation, 57, 5–38.
National Conference of State Legislatures. (2010). Promoting healthy
communities and preventing childhood obesity: Trends in recent
legislation. Washington, DC.
Ogden, C., & Flegal, K. (2010). Changes in terminology for
childhood overweight and obesity. National Health Statistics
Reports, no. 25. Hyattsville, MD: National Center for Health
Statistics.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2006). Year two evaluation: Arkansas Act 1220 of 2003 to Combat
Childhood Obesity. Little Rock, Arkansas: Fay W. Boozman
College of Public Health, University of Arkansas for Medical
Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2007). Year three evaluation: Arkansas Act 1220 of 2003 to combat
childhood obesity. Little Rock, Arkansas: Fay W. Boozman College
of Public Health, University of Arkansas for Medical Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2008). Year four evaluation: Arkansas Act 1220 of 2003 to
123
Am J Community Psychol (2013) 51:289–298
combat childhood obesity. Little Rock, Arkansas: Fay W.
Boozman College of Public Health, University of Arkansas for
Medical Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2010). Year six evaluation: Act 1220 of 2003 of Arkansas to
combat childhood obesity. Little Rock, AR: University of
Arkansas for Medical Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2011). Year seven evaluation: Act 1220 of 2003 of Arkansas to
combat childhood obesity. Little Rock, AR: Fay W. Boozman
College of Public Health, University of Arkansas for Medical
Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2005). Establishing a baseline to evaluate Act 1220 of 2003: An
Act of the Arkansas general assembly to combat childhood
obesity. Little Rock, AR: Fay W Boozman College of Public
Health, University of Arkansas for Medical Sciences.
Phillips, M., Raczynski, J., Walker, J., & Act 1220 Evaluation Team.
(2009). Year Five Evaluation: Arkansas Act 1220 of 2003 to
combat childhood obesity. Little Rock, AR: Fay W Boozman
College of Public Health, University of Arkansas for Medical
Sciences.
Raczynski, J., Thompson, J., Phillips, M., Ryan, K., & Cleveland,
H. (2009). Arkansas Act 1220 of 2003 to reduce childhood
obesity: Its implementation and impact on child and adolescent
body mass index. Journal of Public Health Policy, 30(Suppl 1),
S124–S140.
Ryan, K. W., Card-Higginson, P., McCarthy, S. G., Justus, M. B., &
Thompson, J. W. (2006). Arkansas fights fat: Translating
research into policy to combat childhood and adolescent obesity.
Health Affairs, 25(4), 992–1004.
S. 2507: Child Nutrition and WIC Authorization Act of 2004. (2004).
Retrieved 4 August, 2011, from www.govtrack.us/congress/
bill.xpd?bill=s108-2507.
S. 3307—Healthy, Hunger-Free Kids Act of 2010. (2010). Retrieved
4 August, 2011, from www.govtrack.us/congress/billtext.xpd?
bill=s111-3307.
West, D. S., Raczynski, J. M., Phillips, M. M., Bursac, Z., Gauss,
C. H., & Eaton, B. (2008). Parental recognition of overweight in
school-age children. Obesity, 16(3), 630–636.
Order an Essay Now & Get These Features For Free:
Turnitin Report
Formatting
Title Page
Citation
Outline
