Assessing Public Speaking Impact

This training helps professionals measure public speaking outcomes, covering methods to assess and improve presentation effectiveness for speakers and communicators.

A digital instructor presents all 14 pages. Hold “Ask” at any point and ask out loud — the answer comes from this course. No sign-up needed.

28 minFree to watchDownloads

What you’ll learn

01Measuring Public Speaking Outcomes: Overview and FrameworkWelcome. In this course, we will build a practical framework for measuring public speaking outcomes. Our goal is not more applause, but better evidence. Outcomes span four domains: cognitive, which is what listeners understand; affective, which is how they feel; behavioral, which is what they do next; and audience-level results, such as retention over time. To measure, we must separate real outcomes from weak proxies like attendance, satisfaction surveys, and anecdotes. We will use a four-level logic: reaction, learning, behavior, and results, and note its limits, especially that results take time and are affected by many factors. You will also learn core vocabulary: formative versus summative, criterion-referenced versus norm-referenced, and reliability versus validity. For example, two coaches may both score a speech as a three, yet judge different aspects of it. That is an inter-rater disagreement worth documenting, not hiding. By the end, you should identify measurable outcomes, compare instruments, and document evidence with confidence. Next, we move to defining outcome domains and competency models.dl.acm.org ijee.ie dlnext.acm.org+21 min
02Defining Outcome Domains and Competency ModelsLet's turn to defining outcome domains and competency models. Your first decision is scope. Map three observable domains: content and organization; delivery and vocalics; and language and clarity. Notice that each domain names something a rater can see or hear. Next, translate your competence framework into low-inference behavioral indicators, not trait labels. Instead of calling a learner confident, document what supports that judgment: steady eye contact, varied vocal emphasis, purposeful gestures. Then separate speaker-internal outcomes, such as skill and self-efficacy, from audience-internal outcomes, such as comprehension, retention, and action intent. These require different evidence. For example, a workplace workshop may track action intent, while a classroom course may track comprehension. In practice, choose domains by program level and audience type, whether classroom, workplace, community, or conference. Expect constraints: limited time, audience variability, and occasional disagreement between raters. A lightweight competency map helps here. Link each program goal to one measurable indicator, so scoring stays consistent and defensible. From Competency Statements to Observable Indicators.smu.edu eric.ed.gov powerspmhs.weebly.com+22 min
03From Competency Statements to Observable IndicatorsNow let's translate competency statements into observable indicators. An indicator works only if a trained observer can see or hear it in one viewing. So anchor behaviors you can point to: eye contact, pacing, transitions, evidence quality, and handling of questions. Next, align the rubric language to the specific speaking context and audience. A classroom persuasive speech and a conference keynote will not use the same descriptors. Before rating begins, decide whether you will score against criteria or against other speakers. Criterion referenced asks whether the speaker met the standard. Norm referenced asks how the speaker compares with the group. Choose one, and document it, because switching midstream undermines your data. Finally, test each indicator for traceability. Every line on the rubric should trace back to an intended learning outcome, not to an incidental behavior that simply happened to appear. Here is a concrete example. If your stated outcome is audience adaptation, then eye contact alone is a weak indicator. A stronger one is whether the speaker adjusts vocabulary and examples for that specific audience. That distinction will matter when we look at validity, reliability, and rater bias in speaking assessment.smu.edu eric.ed.gov powerspmhs.weebly.com+22 min
04Validity, Reliability, and Rater Bias in Speaking AssessmentNow let's look at validity, reliability, and rater bias. These three ideas are related, but they are not interchangeable. Validity means the ratings reflect true speaking ability, not a rater's habits. Reliability means multiple raters agree on the same performance. A high reliability score sounds reassuring, but it does not guarantee valid judgments. Raters can agree on a biased standard. Watch for common biases: the halo effect, where one strong impression colors everything else; leniency or severity, where some raters are consistently easy or hard; central tendency, where raters avoid the extremes; and order effects, where earlier or later performances are scored differently. Bias linked to accent, dialect, gender, or major can also contaminate ratings. For example, in a research seminar study, one rater scored female presenters higher than male presenters, and raters favored students from their own academic major. Also account for construct-irrelevant variance, such as anxiety, topic familiarity, room acoustics, or recording quality. These factors change scores without changing speaking ability. So check both agreement and fairness. Next, we turn to rater training, calibration, and guardrails.link.springer.com tandfonline.com ejournal.upsi.edu.my+22 min
05Rater Training, Calibration, and GuardrailsNext, let's look at rater training, calibration, and the guardrails that keep scoring defensible. Start with short calibration cycles. Bring raters together around shared sample speeches. Each rater scores independently first, then the group discusses where scores diverged and why. That consensus discussion is where interpretation of the rubric actually aligns. For reliability, use double scoring and blind review. In a workshop, two coaches can score the same recorded talk without seeing each other's marks. If the gap is large, you have concrete evidence to revisit the criteria. Be realistic about training effects. They fade over time. Studies show consistency can drop in delayed post-training phases, so schedule periodic refreshers and monitor ratings continuously. Track severity, leniency, and bias during and after each cycle. One rater may consistently score cohesion harshly while being lenient on fluency. Finally, document rater decisions. Written rationales let you audit results and defend them when a learner or stakeholder asks how a score was reached. That documentation turns reliability from a claim into evidence. Let's move on to rubrics, rating scales, and observation protocols.link.springer.com tandfonline.com ejournal.upsi.edu.my+22 min
06Rubrics, Rating Scales, and Observation ProtocolsLet's turn to the instruments themselves: rubrics, rating scales, and observation protocols. Match the tool to the decision. Use analytic rubrics when you need diagnostic coaching feedback, because they separate dimensions like organization, language, and delivery. Use holistic rubrics when a fast program-level judgment matters more than detail, such as ranking entries in a competition. Behaviorally anchored scales describe what each performance level looks like. For example, a four might read, uses eye contact to guide the audience; a two might read, reads mostly from notes. Checklists and simple counts flag whether a specific behavior occurred, such as citing three credible sources. Write descriptors that name observable actions. The Oral Communication rubric does this well, defining level four as seamless transitions and level two as weak transitions. Concrete language like that reduces interpretive drift between raters. Then choose an observation protocol. Live coding captures natural conditions; recorded review allows replay and re-scoring. Time-sampling notes behavior at set intervals, while event coding logs each instance of a target behavior, such as filler words or supporting examples. Before full deployment, pilot your instrument on a small set of speeches. Compare rater scores, revise ambiguous descriptors, and document your reliability. That step protects you from discovering disagreement after you have already scored an entire cohort. Next, we look at direct measures through performance tasks and audience response.smu.edu eric.ed.gov powerspmhs.weebly.com+22 min
07Direct Measures: Performance Tasks and Audience ResponseLet's turn to direct measures, which capture what learners actually do and what audiences actually grasp. Start with performance tasks and simulations that mirror real speaking contexts. A workshop learner might deliver a two-minute project pitch to a small panel, then handle three unscripted questions. That task samples far more than a written test can. Next, collect immediate audience checks. Ask listeners to state the main claim, recall one supporting detail, rate clarity, report attitude shift, and indicate whether they intend to act. Each item points to a different outcome. Then document behavioral data: timing against the planned length, filler frequency, gesture use, slide dependence, and responsiveness during Q and A. One caution. Audience variability and inter-rater disagreement will affect your numbers, so brief your observers before you score. Audience response systems add live comprehension and engagement signals. Triangulate these with observer ratings. No single source suffices. A strong survey, weak delivery, or an engaged room with poor recall each tells an incomplete story. Using Audience Response and Comprehension Checks Well.dl.acm.org ijee.ie dlnext.acm.org+22 min
08Using Audience Response and Comprehension Checks WellNow let's look at using audience response and comprehension checks well. The guiding principle is simple. A poll is not just a participation device. It is a measurement moment. So design short post-talk quizzes that target recall, inference, and misconception. For example, include one item asking for a definition, one asking students to apply a concept to a new case, and one built around a known misconception. Then read the response distributions carefully. A split room usually signals genuine disagreement or a hard question. A collectively wrong majority signals a shared misconception worth addressing. Add very short answers alongside fixed options. In one large class, free-text responses surfaced concepts the multiple-choice options never anticipated. Keep engagement, confidence, and comprehension as separate measures. High participation does not guarantee understanding, and confident answers can still be wrong. Finally, keep polling ethical. Seek consent, protect anonymity, and make participation low-pressure so learners respond honestly rather than defensively. That distinction is what makes the data trustworthy. This leads us to technology-assisted measurement, where we examine automation and its limits.revistas.um.es journals.cilip.org.uk dlnext.acm.org+22 min
09Technology-Assisted Measurement: Automation and Its LimitsNow let's turn to technology-assisted measurement, looking at what automation does well and where its limits sit. Start with the pattern from a recent three-level meta-analysis of sixty-seven studies. Human-machine correlations are strongest for delivery, at approximately point seven eight. They drop to roughly point five for grammar and vocabulary. So the correlation varies by construct. Before you adopt any tool, match it to your intended outcome. Delivery skills? Automated scoring can work well. Grammatical range? Treat the score as partial evidence. Second, hybrid scoring is promising. In a TOEFL iBT study, human-machine composites matched or exceeded human holistic reliability. But note the caveat: composites built entirely from human analytic scores were most reliable, and reliability decreased as machine scores replaced human ones. Third, stability differs by feature. Acoustic-prosodic features show high test-retest reliability. Semantic measures, such as information density, are far less stable, even with identical prompts. Finally, watch for fluent but off-topic content. Acoustic-only systems can be misled by it. Flag it, and flag abnormal speech, for human review. Next, indirect measures: self-report, feedback, and perceived competence.sage.cnpereading.com2 min
10Indirect Measures: Self-Report, Feedback, and Perceived CompetenceNow let us turn to indirect measures, where learners report on their own experience rather than being scored directly. These tools capture speaking anxiety, self-efficacy, and perceived improvement, and they complement your observer ratings. Use validated instruments when you can. The Personal Report of Public Speaking Anxiety, or PRPSA-18, has solid reliability and discriminates higher and lower fear of speaking. The Self-Statements During Public Speaking scale separates positive and negative self-talk, and speaking self-efficacy scales, such as the EFL Speaking Self-Efficacy Scale, measure confidence in specific speaking tasks. For your own programs, design brief peer and audience feedback forms with clear, low-inference items. Instead of asking whether a speaker was engaging, ask whether the main point was clear within the first minute. Then handle the limitations honestly. Self-report data are vulnerable to social desirability, acquiescence, and recall bias. And treat gaps between self-rated and observer-rated performance with caution, because they can be meaningful, or simply measurement noise. Next, we connect these measures into a coherent plan in Designing an Evaluation Plan: Baselines, Growth, and Comparison.2 min
11Designing an Evaluation Plan: Baselines, Growth, and ComparisonNow let's turn design choices into a workable plan. First, choose a design. Pre-post suits short workshops, repeated measures suits multi-week programs, cohort comparison suits ongoing courses, and a single case suits one speaker. Next, set baselines and realistic growth targets matched to program length and starting points. For example, a two-day workshop might target one measurable gain in structure, not a full transformation. Then decide sampling, timing, and dosage across the program cycle. Use comparison groups where feasible, and interpret non-experimental results cautiously; audience variability and missing data limit what you can claim. Align the plan with resources and rater capacity. When several raters score the same speech, document agreement and resolve disagreements. Finally, obtain informed consent and IRB approval when required, and report limitations transparently. In short, design small, measure clearly, and state what the data can support. Coming up next, analyzing and interpreting speaking outcome data.2 min
12Analyzing and Interpreting Speaking Outcome DataNow let's turn to analyzing and interpreting your speaking outcome data. Start with descriptives. Compute means, check distributions, and calculate change scores from pre to post. Document those first. Then report effect sizes with confidence intervals, not just p-values. Confidence intervals tell you how precise your estimate is. For example, an effect size of zero point four with a wide interval invites caution. Judge practical significance against similar speaking studies and your program goals. And remember, sample size shapes effects. Cheung and Slavin found small samples often inflate magnitudes, so treat effects from under one hundred learners carefully. Address missing data, rater disagreement, and outliers transparently. Visualize growth with before and after charts or rubric heatmaps. Finally, avoid over-interpreting small samples, p-hacking, and causal claims from correlation. This prepares us for reporting results and closing the improvement loop.2 min
13Reporting Results and Closing the Improvement LoopNow let's close the loop by reporting your results well. Reporting is not the end of measurement. It is where the improvement cycle begins. First, tailor the detail to your audience. Learners need specific, kind, actionable feedback. Instructors need patterns across a cohort. Administrators need trends and resources. Funders need outcomes tied to the original goals. A workshop example: learners receive a short rubric summary, while the programme lead receives an aggregate comparison of pre and post scores. Second, turn findings into decisions. Identify which speakers need targeted coaching, and document what you will change next time. Third, report ethically. Protect anonymity, prevent score misuse, and state uncertainty plainly. Fourth, communicate limitations, methods, and scoring honestly. If two raters disagreed on a dimension, say so, and explain how you handled it. Finally, schedule review cycles to revisit your instruments, targets, and evaluation questions. That keeps measurement current and fair. Let's move to your capstone: building and defending a measurement framework.2 min
14Capstone: Building and Defending a Measurement FrameworkLet's bring everything together in your capstone. Assemble one complete framework for your program: outcomes, measures, timeline, raters, and analysis. Connect every measure to an intended outcome, and state your validity, fairness, and feasibility rationale in a sentence or two. For example, if an outcome is audience understanding, one measure might be a brief recall check, with two trained raters and a stated agreement target. Next, anticipate stakeholder questions about reliability evidence, rater effects, cost, and scoring safeguards. Defend results using double scoring, calibration cycles, and documented rater decisions. Note the common constraints here: time is limited, audiences vary, and raters will not fully agree. Training improves self-consistency more reliably than it forces unanimity, so plan for drift and re-calibration across the term. Finally, peer-review with a critique checklist, then revise instruments and targets before full deployment. Capture your framework in one page your team could actually run. That is the real deliverable. Thank you for your sustained, careful work across this course. You now have the tools to measure speaking outcomes defensibly, and to explain those results to the people who need them. Keep building the evidence.smu.edu eric.ed.gov powerspmhs.weebly.com+22 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.15 pages · 3.5 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.15 pages · 16.4 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.15 pages · 3.4 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.