Program evaluation is a critical tool for understanding and improving organizational activities and systems. This report updates the 1999 CDC Framework for Program Evaluation in Public Health (CDC. Framework for program evaluation in public health. MMWR Recomm Rep 1999;48[No. RR-11];1-40) by integrating major advancements in the fields of evaluation and public health, lessons learned from practical applications of the original framework, and current Federal agency policies and practices. A practical, nonprescriptive tool, the updated 2024 framework is designed to summarize and organize essential elements of program evaluation, and can be applied at any level from individual programs to broader systems by novices and experts for planning and implementing an evaluation. Although many of the key aspects from the 1999 framework remain, certain key differences exist. For example, this updated framework also includes six steps that describe the general process of evaluation planning and implementation, but some content and step names have changed (e.g., the first step has been renamed Assess context). The standards for high-quality evaluation remain central to the framework, although they have been updated to the five Federal evaluation standards. The most substantial change from the 1999 framework is the addition of three cross-cutting actions that are core tenets to incorporate within each evaluation step: engage collaboratively, advance equity, and learn from and use insights. The 2024 framework provides a guide for designing and conducting evaluation across many topics within and outside of public health that anyone involved in program evaluation efforts can use alone or in conjunction with other evaluation approaches, tools, or methods to build evidence, understand programs, and refine evidence-based decision-making to improve all program outcomes.

Next-generation sequencing (NGS) of amplicons is used in a wide variety of contexts. In many cases, NGS amplicon sequencing remains overly expensive and inflexible, with library preparation strategies relying upon the fusion of locus-specific primers to full-length adapter sequences with a single identifying sequence or ligating adapters onto PCR products. In Adapterama I, we presented universal stubs and primers to produce thousands of unique index combinations and a modifiable system for incorporating them into Illumina libraries. Here, we describe multiple ways to use the Adapterama system and other approaches for amplicon sequencing on Illumina instruments. In the variant we use most frequently for large-scale projects, we fuse partial adapter sequences (TruSeq or Nextera) onto the 5' end of locus-specific PCR primers with variable-length tag sequences between the adapter and locus-specific sequences. These fusion primers can be used combinatorially to amplify samples within a 96-well plate (8 forward primers + 12 reverse primers yield 8 × 12 = 96 combinations), and the resulting amplicons can be pooled. The initial PCR products then serve as template for a second round of PCR with dual-indexed iTru or iNext primers (also used combinatorially) to make full-length libraries. The resulting quadruple-indexed amplicons have diversity at most base positions and can be pooled with any standard Illumina library for sequencing. The number of sequencing reads from the amplicon pools can be adjusted, facilitating deep sequencing when required or reducing sequencing costs per sample to an economically trivial amount when deep coverage is not needed. We demonstrate the utility and versatility of our approaches with results from six projects using different implementations of our protocols. Thus, we show that these methods facilitate amplicon library construction for Illumina instruments at reduced cost with increased flexibility. A simple web page to design fusion primers compatible with iTru primers is available at: http://baddna.uga.edu/tools-taggi.html. A fast and easy to use program to demultiplex amplicon pools with internal indexes is available at: https://github.com/lefeverde/Mr_Demuxy.

Next-generation sequencing is a powerful tool for virological surveillance. While Illumina(R) and Ion Torrent(R) sequencing platforms are used extensively for generating viral RNA genome sequences, there is limited data comparing different platforms. The Illumina MiSeq, Ion Torrent PGM and Ion Torrent S5 platforms were evaluated using a panel of sixteen specimens containing picornaviruses and human caliciviruses (noroviruses and sapoviruses). The specimens were processed, using combinations of three library preparation and five sequencing kits, to assess the quality and completeness of assembled viral genomes, and an estimation of cost per sample to generate the data was calculated. The choice of library preparation kit and sequencing platform was found to impact the breadth of genome coverage and accuracy of consensus viral genomes. The Ion Torrent S5 510 chip runs produced more reads at a lower cost per sample than the highest output Ion Torrent PGM 318 chip run, and generated the highest proportion of reads for enterovirus D68 samples. However, indels at homopolymer regions impacted the accuracy of consensus genome sequences. For lower throughput sequencing runs (i.e., Ion Torrent 510 and Illumina MiSeq Nano V2), the cost per sample was lower on the MiSeq platform, whereas with higher throughput runs (Ion Torrent 530 and Illumina MiSeq V2) there is less of a difference in the cost per sample between the two sequencing platforms ($5.47-$10.25 more per sample for an Ion Torrent 530 chip run when multiplexing 24 samples). These findings suggest that the Ion Torrent S5 and Illumina MiSeq platforms are both viable options for genomic sequencing of RNA viruses, each with specific advantages and tradeoffs.

The Community Antiretroviral (ARV) Therapy Support Group (CASG) program aims to address low retention rates in Mozambique's HIV treatment program and the absorptive capacity of the country's health facilities. CASG provides patients with the opportunity to form groups, whose members provide peer support and collect ARV medications on a rotating basis for one another. Based on the promising results in one province, a multi-site level evaluation followed. We report on qualitative findings from this evaluation from the patient perspective on the role of social relationships (as facilitated through CASG) in conferring time, financial, educational and psychosocial benefits that contribute to improved patient retention. These findings may be helpful in informing what aspects of social relationships are critical to foster as CASG is implemented within a greater number of Mozambican health facilities, and as other countries design and implement related models of care and treatment with a support group component.

BACKGROUND: Determining patterns of nucleotide and amino acid substitution is the first step during sequence evolution analysis. However, it is not easy to visualize the different phylogenetic signatures imprinted in aligned nucleotide and amino acid sequences. RESULTS: Here we present PoSE (Pattern of Sequence Evolution), a reliable resource for unveiling the evolutionary history of sequence alignments and for graphically displaying their contents. Substitutions are displayed by category (transitions and transversions), codon position, and phenotypic effect (synonymous and nonsynonymous). Visualization is accomplished using MATLAB scripts wrapped around PAML (Phylogenetic Analysis by Maximum Likelihood), implemented in an easy-to-use graphical user interface. The application displays inferred substitutions estimated by baseml or codeml, two programs included in the PAML software package. PoSE organizes patterns of substitution in eleven plots, including estimated non-synonymous/synonymous ratios (dN/dS) along the sequence alignment. In addition, PoSE provides visualization and annotation of patterns of amino acid substitutions along groups of related sequences that can be graphically inspected in a phylogenetic tree window. CONCLUSIONS: PoSE is a useful tool to help determine major patterns during sequence evolution of protein-coding sequences, hypervariable regions, or changes in dN/dS ratios. PoSE is publicly available at https://github.com/CDCgov/PoSE.

Annotated whole genome sequences of three isolates of the Akhmeta virus (AKMV), a novel species of orthopoxvirus (OPXV), isolated from the Akhmeta and Vani regions of the country Georgia, are presented and discussed. The AKMV genome is similar in genomic content and structure to that of the cowpox virus (CPXV), but a lower sequence identity was found between AKMV and Old World OPXVs than between other known species of Old World OPXVs. Phylogenetic analysis showed that AKMV diverged prior to other Old World OPXV. AKMV isolates formed a monophyletic clade in the OPXV phylogeny, yet the sequence variability between AKMV isolates was higher than between the monkeypox virus strains in the Congo basin and West Africa. An AKMV isolate from Vani contained approximately six kb sequence in the left terminal region that shared a higher similarity with CPXV than with other AKMV isolates, whereas the rest of the genome was most similar to AKMV, suggesting recombination between AKMV and CPXV in a region containing several host range and virulence genes.

We followed the dynamics of capsid amino acid replacement among 403 Nigerian outbreak isolates of type 2 circulating vaccine-derived poliovirus (cVDPV2) from 2005 through 2011. Four different functional domains were analyzed: 1) neutralizing antigenic (NAg) sites, 2) residues binding the poliovirus receptor (PVR), 3) VP1 residues 1-32, and 4) the capsid structural core. Amino acid replacements mapped to 37 of 43 positions across all 4 NAg sites; the most variable and polymorphic residues were in NAg sites 2 and 3b. The most divergent of the 120 NAg variants had no more than 5 replacements in all NAg sites, and were still neutralized at titers similar to those of Sabin 2. PVR-binding residues were less variable (25 different variants; 0-2 replacements/isolate; 30/44 invariant positions), with the most variable residues also forming parts of NAg sites 2 and 3a. Residues 1-32 of VP1 were highly variable (133 different variants; 0-6 replacements/isolate; 5/32 invariant positions), with residues 1-18 predicted to form a well-conserved amphipathic helix. Replacement events were dated by mapping them onto the branches of time-scaled phylogenies. Rates of amino acid replacement varied widely across positions and followed no simple substitution model. Replacements into the structural core were the most conservative and were fixed at an overall rate approximately 20-fold lower than rates for the NAg sites and VP1 1-32, and approximately 5-fold lower than the rate for the PVR-binding sites. Only VP1-143-Ile, a non-NAg site surface residue and known attenuation site, appeared to be under strong negative selection.IMPORTANCE The high rate of poliovirus evolution is offset by strong selection against amino acid replacement at most positions of the capsid. Consequently, poliovirus vaccines developed from strains isolated decades ago have been used worldwide to bring wild polioviruses almost to extinction. The apparent antigenic stability of poliovirus obscures a dynamic of continuous change within the neutralizing antigenic (NAg) sites. During seven years of a large outbreak in Nigeria, the circulating type 2 vaccine-derived polioviruses generated 120 different NAg site variants via multiple independent pathways. Nonetheless, overall antigenic evolution was constrained, as no isolate had fixed more than 5 amino acid differences from the Sabin 2 NAg sites, and the most divergent isolates were efficiently neutralized by human immune sera. Evolution elsewhere in the capsid was also constrained. Amino acids binding the poliovirus receptor were strongly conserved, and extensive variation in the VP1 amino terminus still conserved a predicted amphipathic helix.

Public health response to vaccine-derived poliovirus (VDPV) that is transmitted from person to person (circulating VDPV [cVDPV]) differs significantly from response to virus that replicates in individuals with primary immunodeficiency (immunodeficiency-associated VDPV [iVDPV]). cVDPV outbreaks require a community immunization response, whereas iVDPV chronic infections require careful patient monitoring and appropriate individual treatment. To support poliovirus outbreak response, particularly for type 2 VDPV, we investigated the genetic distinctions between cVDPV2 and iVDPV2 sequences. We observed that simple genetic measurements of nucleotide and amino acid substitutions are sufficient for distinguishing highly divergent iVDPV2 from cVDPV2 sequences, but are insufficient to make a clear distinction between the two categories among less divergent sequences. We presented quantitative approaches using genetic information as a surveillance tool for early detection of VDPV outbreaks. This work suggests that genetic variations between cVDPV2 and iVDPV2 may reflect differences in viral micro-environments, host-virus interactions, and selective pressures during person-to-person transmission compared with chronic infections in immunodeficient patients.

Poliovirus (PV) is currently targeted for worldwide eradication and containment. Sanger-based sequencing of the VP1 capsid region is the current standard method for PV surveillance; however, the whole genome sequence is sometimes needed for higher resolution global surveillance. In this study, we optimized whole genome sequencing protocols for poliovirus isolates and FTA cards using NGS, aiming for high sequence coverage, efficiency, and throughput. We found that DNase treatment of poliovirus RNA followed by random RT, amplification, and the Nextera XT DNA Library Preparation Kit produced significantly better results than other preparations. Average viral reads per total reads, a measurement of efficiency, is as high as 84.2% +/- 15.6%; PV genomes covering >99-100% of the reference length were obtained and validated with Sanger sequencing. A total of 52 PV genomes were generated, multiplexing as many as 64 samples in a single Illumina MiSeq run. This high-throughput, sequence-independent NGS approach can facilitate the detection of a diverse range of PV, especially for those in vaccine-derived polioviruses (VDPV), circulating VDPV, or immunodeficiency-related VDPV. In contrast to previous studies on other viruses, our results showed that filtration and nuclease treatment did not produce discernable increases in sequencing efficiency of PV isolates. However, DNase treatment after nucleic acid extraction to remove host DNA significantly improved sequencing results. This NGS method has been successfully implemented to generate PV genomes for molecular epidemiology of the most recent PV isolates. Additionally, the ability to obtain full PV genomes from FTA cards will aid in facilitating global poliovirus surveillance.

BACKGROUND: Poxviruses constitute one of the largest and most complex animal virus families known. The notorious smallpox disease has been eradicated and the virus contained, but its simian sister, monkeypox is an emerging, untreatable infectious disease, killing 1 to 10 % of its human victims. In the case of poxviruses, the emergence of monkeypox outbreaks in humans and the need to monitor potential malicious release of smallpox virus requires development of methods for rapid virus identification. Whole-genome sequencing (WGS) is an emergent technology with increasing application to the diagnosis of diseases and the identification of outbreak pathogens. But "finishing" such a genome is a laborious and time-consuming process, not easily automated. To date the large, complete poxvirus genomes have not been studied comprehensively in terms of applying WGS techniques and evaluating genome assembly algorithms. RESULTS: To explore the limitations to finishing a poxvirus genome from short reads, we first analyze the repetitive regions in a monkeypox genome and evaluate genome assembly on the simulated reads. We also report on procedures and insights relevant to the assembly (from realistically short reads) of genomes. Finally, we propose a neural network method (namely Neural-KSP) to "finish" the process by closing gaps remaining after conventional assembly, as the final stage in a protocol to elucidate clinical poxvirus genomic sequences. CONCLUSIONS: The protocol may prove useful in any clinical viral isolate (regardless if a reference-strain sequence is available) and especially useful in genomes confounded by many global and local repetitive sequences embedded in them. This work highlights the feasibility of finishing real, complex genomes by systematically analyzing genetic characteristics, thus remedying existing assembly shortcomings with a neural network method. Such finished sequences may enable clinicians to track genetic distance between viral isolates that provides a powerful epidemiological tool.

To assess the dynamics of genetic reversion of live poliovirus vaccine in humans, we studied molecular evolution in Sabin-like poliovirus isolates from Nigerian acute flaccid paralysis cases obtained from routine surveillance. We employed a novel modeling approach to infer substitution and recombination rates from whole-genome sequences and information about poliovirus infection dynamics and individual vaccination history. We confirmed observations from a recent vaccine trial that VP1 substitution rates are increased for Sabin-like isolates relative to the wild-type rate due to increased non-synonymous substitution rates. We also inferred substitution rates for attenuating nucleotides and confirmed that reversion can occur in days to weeks after vaccination. We combine our observations for Sabin-like evolution with the wild-type circulating VP1 molecular clock to infer that the mean time from the initiating vaccine dose to the earliest detection of circulating vaccine-derived poliovirus (cVDPV) is 300 days for type 1, 210 days for type 2, and 390 days for type 3. Phylogenetic relationships indicated transient local transmission of Sabin 3 and possibly Sabin 1 during periods of low wild polio incidence. Comparison of Sabin-like recombinants with known Nigerian VDPV recombinants shows that while recombination with non-Sabin enteroviruses is associated with cVDPV, the recombination rates are similar for Sabin-Sabin and Sabin-non-Sabin enterovirus recombination after accounting for time from dose to detection. Our study provides a comprehensive picture of the evolutionary dynamics of oral polio vaccine in the field. IMPORTANCE: The global polio eradication effort has completed its twenty-sixth year. Despite success in eliminating wild poliovirus from most of the world, polio persists in populations where logistical, social, and political factors have not allowed for vaccination programs of sustained high quality. One issue of critical importance is eliminating circulating vaccine-derived poliovirus (cVDPV) that have properties indistinguishable from wild poliovirus and can cause paralytic disease. cVDPV emerges due to the genetic instability of the Sabin viruses used in oral polio vaccine (OPV) in populations that have low immunity to poliovirus. However, the dynamics responsible are incompletely understood because it has historically been difficult to gather and interpret data about OPV evolution in regions where cVDPV has occurred. This study is the first to combine whole-genome sequencing of poliovirus isolates collected during routine surveillance with knowledge about polio intra-host dynamics to provide quantitative insight into polio vaccine evolution in the field.

Shiga toxin-producing Escherichia coli (STEC) are a common cause for food-borne diarrheal illness outbreaks and sporadic cases. Here, we report the availability of the draft genome sequences of 228 STEC strains representing 32 serotypes with known pulsed-field gel electrophoresis (PFGE) types and epidemiological relationships, as well as 12 strains representing other diarrheagenic E. coli pathotypes.

INTRODUCTION: Research has shown that partnerships between public health agencies, service providers, and other key stakeholders can expand resources and facilitate focus on community health issues more effectively than can any agency or organization acting alone. There is, however, little empirical evidence drawn from actual public health emergency responses to support this claim. The US response to novel influenza A (H1N1) virus provided the Centers for Disease Control and Prevention (CDC) the opportunity to explore whether, and the extent to which, state, local and territorial health departments strengthened partnerships with key partner agencies and sectors. METHODS: Participants included the CDC Public Health Emergency Response (PHER) grantees comprised of 62 state, territorial and local health departments. PHER grantees completed an assessment instrument in May 2011, including questions asking them to rate their partnership strength (on a four-point ordinal scale) with six types of partners before and after the H1N1 response. Grantees additionally reported if and how PHER funding contributed to enhancing the strength of these partnerships. RESULTS: Sixty-one PHER grantees (61/62, 98%) completed the assessment instrument's partnerships section. PHER grantees reported that their partnerships with retail pharmacies were most strengthened (mean increase = 1.11 (on a four-point ordinal scale), SD = .82). This was followed by schools (K-12) (mean increase = .90, SD = .58); private medical providers (mean increase = .81, SD = .68); immunization authorities (mean increase = .80, SD = .61); main education authorities (mean increase = .75, SD = .68); and businesses (mean increase = .74, SD = .61). Mean PHER grantee increases in the strength of each partner type were statistically significant for all partner types (P < .01). Grantees reported that PHER funding contributed to enhancing the strength of their partnerships with schools most frequently (46/46, 100%), and businesses least frequently (31/37, 83.8%). CONCLUSIONS: This inquiry provides evidence that state, territorial, and local health department partnerships with key sectors, agencies, and programs were strengthened after the H1N1 response. It further demonstrates that the CDC's PHER funding contributed to the health departments' reports of increased partnership strength.

Since 2001, the Centers for Disease Control and Prevention's Public Health Emergency Preparedness cooperative agreement has supported state, territorial, and local public health departments in preparing for and responding to public health emergencies. This conceptual article describes complexities identified and lessons learned in developing community preparedness performance measures for the Centers for Disease Control and Prevention's public health preparedness program. Challenges arose in (a) defining community; (b) measuring meaningful community engagement; and (c) determining a strategy for collecting, aggregating, and analyzing data from diverse state, territorial, and local health departments. This article contributes to prior work describing conceptual challenges in developing standardized measures of performance at the federal level and suggests ways to potentially mitigate general performance measurement challenges as well as measurement complexities specific to community preparedness. It may be informative for those state, territorial, and local health departments currently implementing (or contemplating implementing) community preparedness activities and for individuals more generally engaged in performance measurement.

The Centers for Disease Control and Prevention's Sustainable Management Development Program (SMDP) partners with low- and middle-resource countries to develop management capacity so that effective global public health programs can be implemented and better health outcomes can be achieved. The program's impact however, was variable. Hence, there was a need to both engage in a strategic planning process and collect useful data to inform the process. We therefore designed a qualitative evaluation and findings that emerged concerning our program’s contribution to individual career advancement and professional growth; the need for institutional support and a champion to move public health management capacity development efforts forward in low- and middle-resource countries; and interest in diverse professional learning opportunities contributed to program improvement and suggested new strategic directions for CDC's global public health management service delivery. Our inquiry provides a concrete example of how qualitative methods, specifically key informant interviews, can provide useful data for strategic planning within public health settings. It may be useful to readers who are interested in conducting strategic planning within public health and other related areas including health care, mental and behavioral health, and the social sciences.