Good/Bad News

Cite any reference to the article below as:

Deubel, P. (2001, Spring). The good and bad news of software use for mathematics proficiency test preparation. Ohio Journal of School Mathematics, 43, 55-60.

The Good and Bad News of Software Use
for Mathematics Proficiency Test Preparation

By
Patricia Deubel, Ph.D.

Problem

In March 2000, 126,252 public school eighth graders took the Ohio Ninth Grade Proficiency Test (ONGPT). Approximately 46% passed all tests taken, a decrease of 3% from last year’s performance. Pass rates for this year’s eighth graders were lower on all five tests, with the biggest drops in writing (7%) and reading (5%). Mathematics results showed an overall decline of 1% from last year's passing rate (ODE, 2000b). Typically, mathematics has been the most difficult part of the ONGPT for students to pass. The problem can potentially get worse. Ohio's Senate Bill 55, the academic accountability package passed by the legislature in 1997, requires that beginning in 2003 the state's proficiency tests shall measure student knowledge of core academic areas through a tenth-grade level. The first class to be impacted by this new law is the Class of 2005 (ODE, 1998).

In the Ohio Department of Education (ODE, 2000a) report, The State of Education in Ohio, State Superintendent Zelman said that proficiency test results suggest that Ohio needs better approaches in mathematics instruction. In terms of the use of software to prepare students for the proficiency tests, however, the ODE's assessment center does not recommend products to school districts for preparation for the state's proficiency tests. School districts are told: Let the buyer beware (Gillespie, 1999).

Standards adopted by The National Council of Teachers of Mathematics (NCTM, 1989) and goals of the Ohio Schools Technology Implementation Task Force (1999) include that teachers should use electronic products to improve student learning. Students should have access to technology that will help them meet high standards of achievement. Has software use, however, made any difference on students passing the mathematics test of the ONGPT?

To address that question, Grade 8 mathematics, special education, and proficiency intervention teachers (N =113, which was an 88% response rate) in 35 middle schools across 13 mid-sized districts in the Urban Schools Initiative were surveyed prior to the March 2000 tests. The survey addressed teacher beliefs on individual and organizational factors related to the use of software during class time, what software was used, how it was used, and software's instructional and technical merit for proficiency test preparation. Mathematics achievement was measured for over 6000 Grade 8 students. The study received support from the ODE Urban Schools Initiative, which is now the Office of Comprehensive School Improvement. This article presents major findings of the study. The news is both good and bad.

Good and Bad News of Software Use

The good news is that software use during math class time appears to have increased among all 13 participating school districts from 1994-2000. Reported use from teachers showed a rise from 3% in 1994-95 to 52% in 1999-2000. The bad news is that software use during class time was not an integral part of the mathematics curriculum in most settings for those districts. Forty percent of participants had never used software in their instruction, which does not necessarily mean that teachers were resistant, incompetent, lacked expertise, or were technophobes, according to Cuban (O'Neil, 2000). Typical among the 52 reasons that teachers gave for not using software for instruction were statements such as:

	There is one computer in my room with the Optimum program loaded. Students use the program during study hall for practice. I don't use it in class because the program doesn't provide enough feedback to students to make good use of class time. I would be thrilled to use more computer software and even previewed some this fall, but the district doesn't provide funds for purchase.
	I don't have the knowledge or training to use computers or software effectively. I also don't have clear goals, whether to teach the 8^th grade curriculum or teach the test.
	Proficiency software was moved out of the classroom and into the labs. I have not taken students as of yet to the labs.
	I have no software on my computer…I have spoken with our computer teacher to see if I can get programs from the computer lab sent up to my classroom. Apparently there are issues with licenses and/or wiring.
	I do not have one computer per student. Also, computers do not teach. They are weak instruments that can be used to help clarify concepts and provide drill and practice. (But if a kid can do something, how much D & P is needed?) In this system, the 8^th graders are so far behind, one does not have time to develop skills before the prof. test is given. Hence, we cram on everything. It is like telling a farmer to plant his corn June 20 and expect it to yield as well as corn planted May 15^th. Get serious! This is not a complaint or excuse. It is simple reality.
	Given all the time most of my 8^th graders spend in front of a computer screen both at home and in our computer labs, most have no intuitive math comprehension or grasp of basic skills. Our learning problems in [school name omitted] are not regarding what tools we use to teach, but rather how we show a need to be educated to people who are involved only in daily basic survival techniques.
	The software we have consists of practice problems and scoring. The value of this software is strictly as a change of pace to our normal proficiency review cycle.
	We do not have access to them in our classroom. The lab has one program, but the computers do not have enough memory to get the newer programs.
	Everyone is on different levels, so we do individual assignments.
	I do not feel that the software we have available is of much benefit and certainly pales in comparison to that which takes place in my classroom.
	Having only one computer is the major reason and the lack of materials. This is also my first year of teaching math and I feel that I was not ready for that step.
	I have asked for such software, but not received it.
	Need more information and willing to use; would be helpful.

Top test improvement strategies that all teachers used for proficiency test preparation included small group tutoring (81%), whole class instruction (75%), individual tutoring (71%), communicating high expectations (56%), followed by using computer software (50%). Even among the 52% of teachers who used software during 1999-2000, most only used it occasionally all year (43%), which meant at most a few times a month (50%).

The occasional use of software during class time actually had a significant negative impact on students passing the test. Software may not have been used in a focused manner to help individual students to master specific skills in which they were weak. More students who had not used software during class time passed the test than those who did use software during class time, a significant difference (Table 1).

Software use made a significant difference on passing the test for students who did not use software during class time, but had used software in proficiency intervention classes that met in addition to regular math classes. Unfortunately, most schools did not have the luxury of offering such classes. The implication is that the personalized, regular and focused effort on using software over time to combat individual weaknesses and to build skill fluency led to achievement gains (Table 2).

Table 1: Chi-Square Analysis of Test Results for Class Time Software Use _________________________________________________________________
		Actual			Expected
		Passed	Did Not	Total	% Passed in Group	To Pass	Not Pass
_________________________________________________________________
Software		1129	1706	2835	40%	1264	1571
No Software		1991	2172	4163	48%	1856	2307
Total		3120	3878	6998	45%
_________________________________________________________________
Chi-Square:	43.714	Significant at .01 level			df =1
Probability:	3.8E-11
_________________________________________________________________ Note: Data for one school (N=294 grade 8) could not be included because results were reported for grades 8 and 9 combined. The 45% who passed the test included 16% who used software and 29% who did not use software.

Table 2: Chi-Square Analysis of Test Results for Class Time Software Use and Extra Proficiency Intervention Using Software ___________________________________________________________________
		Actual			Expected
Software Use		Passed	Did Not	Total	% Passed in Group	To Pass	Not Pass
___________________________________________________________________
Class Use, No Extra		682	962	1644	41%	761	883
Class Use, W. Extra^a		27	84	111	24%	51	60
No Class Use, No Extra		1150	1428	2578	45%	1193	1385
No Class Use, W. Extra		753	559	1312	57%	607	705
Total		2612	3033	5645	46%
___________________________________________________________________
Chi-Square:	104.808	Significant at .01 level			df =3
Probability:	1.4E-22
___________________________________________________________________ Note: This table includes data from 27 of 35 participating schools, which provided test results with a breakdown for students who had extra proficiency intervention using software. ^{aThis category includes results from 3 teachers in 2 schools. In school 1, 9 of 18 passed (50%). In school 2, 18 of 93 passed (19%). Caution is needed in generalizing the category result because the sample size is too small in comparison to other groups.}

So why haven't teachers used software more often in their instruction? Results indicated that administrative support, teacher instructional style, teachers' perceived priority of learning about computers and software, computer availability and access, availability of technical assistance when needed, and software quality were significant factors in teachers' decisions to use software in their instruction.

Surprisingly, source of computer learning, whether by staff development or learning on one's own, and lack of time to learn about computers and software were not factors that distinguished software users from non-users. Only 40% of teachers felt staff development met their needs to learn to use technology in the classroom. Likewise, only 19% of each group believed they had adequate time to learn about computers and new software. Anxiety was not an issue of concern. Only 12% of all teachers expressed anxiety when using computers and software.

Other good news is that teachers rated overall software quality as good for preparing students for the ONGPT math test. Ten of the top 12 software packages they used have correlation to NCTM, national and state standards, or to the mathematics learning objectives of the ONGPT (Table 3). The most frequently used program, Math Blaster (1997), was not among those, however. In fact, Soloway and Norris (1998) criticized Math Blaster for its gaming format. Students get to play a shoot-'em-up game having nothing to do with what they just learned as a reward for success. They proposed alternative software called Tenth Planet Explores Math (1996-2001). The fraction module, for example, provides multiple linked representations, and allows students to manipulate objects and to keep computer journals of their investigations. Problems are framed in real-world applications. The Tenth Planet fractions module makes much better use of the computational media than does Math Blaster, but the cost is four times as much for this thoughtful learning curriculum. Most software was used for remediation and drill and practice, but teachers clearly pointed out the need for drill and practice software for students who were failing the ONGPT math test because they lacked the skills tested on the exam.

Table 3: Top 12 Software Titles Used During Class Time ________________________________________________________________________
	All Districts (13, N=60)		>50% Districts (7, N=39)		<=50% Districts (6, N=21)
Title	Used	Best	Used	Best	Used	Best
________________________________________________________________________

Math Blaster (1997)	43%	3%	28%	0%	71%	10%
Passing the Ohio Ninth Grade Proficiency Test (1997)	37%	8%	33%	10%	43%	5%
Optimum (1997)	25%	5%	23%	5%	29%	5%
Math Skills and Concepts SuccessMaker (1999)	18%	5%	10%	5%	33%	5%
Hot Dog Stand (1996)	13%	0%	15%	0%	10%	0%
Math for the Real World (1996)	13%	2%	8%	3%	24%	0%
GoFigure (1997)	12%	5%	10%	8%	14%	0%
Math Munchers Deluxe (1997)	12%	0%	13%	0%	10%	0%
OPT for Success (1998)	12%	3%	13%	5%	10%	0%
Desktop Tutor (1996)	10%	2%	10%	0%	10%	5%
Mystery Math Island (1995)	10%	0%	13%	0%	5%	0%
SkillsBank (1998)	10%	0%	8%	0%	14%	0%
________________________________________________________________________ Note: All products, except Math Blaster and Hot Dog Stand, have been correlated to either national or state standards. The columns headed as Best indicate the percentages of teachers who chose the product as best for addressing the 16 learning objectives on the math test of the ONGPT. Districts were divided into two groups: those in which more than 50% of surveyed teachers used software and those in which 50% or fewer used software.

The bad news is that schools may not be getting the total return on investment of technology dollars. As one teacher in this study stated, software use was a pleasant break for kids. Ten of 36 software titles on the survey were titles that districts said were available, but were not used during math class time, although they may have been used during other times (e.g., tutoring sessions during or after school or during intervention classes). In fairness to teachers, some abandoned using available software because of its weaknesses. Some software dated back to 1991, or could not run on available computers. For some, building wiring and software licensing issues prevented access to software they wanted to use.

Valuable Software

Valuable educational software has stated learning objectives that are adhered to, is motivating to students, has well designed navigation icons, and contains multimedia features. It is also available above Version 1.0, which often has not been debugged and should be avoided (Abramson, 1998). Although overall software quality was rated good, findings revealed that software did not always have those valuable qualities, or the ability to foster higher order thinking skills.

Weaknesses in software's instructional and technical merit revealed that teachers want a management system that allows them to modify software to individualize instruction. The management system should keep track of student progress, identify areas of weakness and strength, suggest paths to improve, and have the ability to automatically adjust for student needs. The software should have more than one entry level, more than one level of difficulty, and provide some repetition to assist in retention. Software should allow students to save data, so that if they are not finished with a lesson, they do not have to begin again. Students should be able to change answers before software grades assessments.

Valuable software has an extensive database of problems, so that upon repeated use of software, students encounter a different set of problems. Problems should reference real-life applications and software should accommodate more than one solution method. Feedback should be tutorial in nature, not just indicate if responses were right or wrong. Help and audio features should be under user control. Teachers want software security against student errors or their intentional attempts to disrupt software operation. Software should meet NCTM, national and state standards, and learning objectives for the ONGPT. All of these criteria provide a set of guidelines to help districts in selecting software.

Conclusions

Clearly, much can be said about improving mathematics instruction using technology. Money continues to be an issue for many urban schools, particularly those that do not qualify for a lot of federal aid. One middle school principal stated their district needed computers, but fell in the middle between the haves and have-nots and must purchase their own technology. Although district and school budgets paid for the majority of software, teachers in 8 of 13 districts indicated some use of outside funding sources, including grants, parent associations, or Ohio School Net.

Schools cannot keep up with purchases of the latest software to run on older computers. Teachers cannot be expected to have contingency lesson plans for technology that fails when they need it. Therefore, they continue to use textbooks, overhead projectors, and chalk because they are reliable and flexible, as this study demonstrated. Let the buyer beware does not appear to be a sufficient answer when purchasing software, if the goal is to raise achievement of students with technology in Ohio's classrooms.

Reference List

Abramson, G. W. (1998). How to evaluate educational software. Principal, 78(1), 60-61.

Desktop Tutor (Version 1.0) [Computer software]. (1996). Hollandays Publishing Corp., PO Box 161, Clayton, OH 45315.
http://home.att.net/~hollandays/index.html Email: hollandays@worldnet.att.net

Hot Dog Stand: The Works [Computer software]. (1996). Sunburst Communications, Inc., 101 Castleton Street, Pleasantville, NY 10570. http://www.sunburst.com Accessed December 21, 1999.

Gillespie, C. (1999, September 24). Firm offers a trial Ohio proficiency test on Net. Cleveland, OH: The Plain Dealer, p. 5B.

GoFigure [Video and Computer software]. (1997). WVIZ Channel 25 and Ohio University Public TV, 4300 Brookpark Road, Cleveland, OH 44134. http://wviz.org/Connections/1998/03/gofig.html

Math Blaster [Computer software]. (1997). Knowledge Adventure, 4100 West 190^th Street, Torrance, CA 90504. http://www.knowledgeadventure.com Accessed December 29, 1999.

Math Concepts and Skills SuccessMaker (Version 5.2) [Computer software]. (1999). Sunnyvale, CA: Computer Curriculum Corporation. http://www.ccclearn.com/products/successmaker/index.html

Math for the Real World [Computer software]. (1996). Knowledge Adventure, 4100 West 190^th Street, Torrance, CA 90504. http://www.knowledgeadventure.com Accessed December 22, 1999.

Math Munchers Deluxe [Computer software]. (1997). The Learning Company, 500 Redwood Blvd., Novato, CA. 94947. http://www.learningcompanyschool.com/school/products/mmd.htm Accessed April 24, 2000.

Mystery Math Island [Computer software]. (1995). Lawrence Productions, Inc., 1800 South 35^th Street, PO Box 458, Galesburg, MI 49053-0458. http://www.voyager.net/lawrence/software/mathematics/software_mystery.html Email: marketing@lpi.com Accessed January 3, 2000.

National Council of Teachers of Mathematics Curriculum and Evaluation Standards. (1989). Reston, VA: National Council of Teachers of Mathematics. http://www.enc.org/reform/journals/ENC2280/nf_280dtoc1.htm Accessed October 2, 1998.

Ohio Department of Education. (1998). News release: Ninth-graders break overall, individual records on ninth-grade tests. http://www.ode.ohio.gov/www/ims/prof9_news_oct97.html
Updated January 26, 1998. Accessed July 22, 1999.

Ohio Department of Education. (2000a, February). The State of Education in Ohio.

Ohio Department of Education. (2000b, June). News Release: Class of 2000 Maintains Steady Performance on Ninth-Grade Proficiency Test. http://www.ode.state.oh.us/comm/news/06-05-00.htm Updated June 2000. Accessed June 6, 2000.

Ohio Schools Technology Implementation Task Force report: Senator Robert R. Cupp, Chairman. (1999, March 3). pp. 1-35. http://www.ohioschoolnet.k12.oh.us/extlink/frameset.asp?exturl=/resources/document/documents/doclib_490.pdf&returl=/ Accessed July 15, 1999.

O'Neil, J. (2000). Fads and fireflies: The difficulties of sustaining change. Educational Leadership, 57(7), 6-9.

OPT for Success (Version 2.0) [Computer software]. (1998). Pearson Publishing, Inc., Kamico Instructional Media, 5917 Maury's Trail, Austin, TX 78730-2767. http://www.pearson-publishing.com Email: info@kamico.com

Optimum (Version 1.0) [Computer software]. (1997). Photonics Graphics, 700 W. Pete Rose Way Suite 360, Cincinnati, OH 45203. http://www.photonicsgraphics.com Email: photonics@photonicsgraphics.com

Passing the Ohio Ninth Grade Proficiency Test (Version 2.0) [Computer software]. (1997). Englefield & Arnold, Inc., 2121 Bethel Road, Suite D, Columbus, OH 43220. http://www.eapublishing.com

SkillsBank (Version 4) [Computer software]. (1998). SkillsBank Corporation, 500 Redwood Blvd., Novato, CA 94947. http://www.skillsbank.com

Soloway, E., & Norris, C. (1998). Using technology to address old problems in new ways. Association for Computing Machinery, Communications of the ACM, 41(8), 11-18.

Tenth Planet Explores Math [Computer software]. (1996-2001). Tenth Planet, 625 Miramontes St., Half Moon Bay, CA 94091. http://www.tenthplanet.com Accessed January 24, 2001. Email: info@tenthplanet.com

Contributor:

Patricia Deubel holds a Ph.D. in Computing Technology in Education from Nova Southeastern University in Florida, a Masters degree in Curriculum and Instruction from Ashland University in Ohio, and a B.S. in mathematics from Ohio University. She has over 25 years of experience in mathematics and computer education teaching, teacher training and staff development, and curriculum development. She has presented computer workshops at the state and local levels and currently teaches mathematics at The Ohio State University in Mansfield. Her recent articles have been published in the Journal of Instruction Delivery Systems and HyperNexus: Journal of Hypermedia and Multimedia Studies. For further consultation regarding the outcomes of this study email: pdeubel@aol.com

Last Modified: 02/03/02