111 Washington Avenue, Suite 101

Albany, NY 12210

(518) 427-1060 Voice & TTY

nysilc@nysilc.org

www.nysilc.org

NYSILC Focus Group Voting Machine Testing Report

NYSILC Focus Group Voting Machine Testing

March 10, 2008 - The New York State Independent Living Council (NYSILC) conducted focus group testing for the Sequoia/Dominion Image Cast, Premier AutoMark and Liberty Mark voting machines during the month of February 2008 at its office located in Albany, NY.

The primary benefit was to provide the vendors with feedback so that they could make final improvements to their ballot marking devices before rolling them out to the counties in advance of voter and poll worker education efforts, as well

as the voting activity scheduled for Primary and Election Days in 2008. The secondary benefit was to have people with disabilities test and help identify what ballot marking devices would best suit their needs from the models selected by the counties per the court order.

NYSILC utilized a basic Voting Machine Survey to collect information from the focus group participants. See Attachment I for a copy of this survey. Item I. A., “voting access,” varied slightly based on the exact features found for each machine. The collective results from the surveys have been summarized in this report. NYSILC believes that the collective data and comments provide objective results. The reader can draw their own conclusions from the information presented in the report.

I. Sequoia Dominion Image Cast (February 14 and 15, 2008).

The testing took place over a two day period. The vendor preferred time slots of up to 45-minutes per person. This allowed them ample time to interview the participants after voting to receive feedback about the use of their machine. The vendor recorded all comments directly into a laptop computer. At the end of the second day, the vendor facilitated a group discussion and shared the collective results of the feedback received from the testers. The vendor identified a list of immediate changes they would make, a second list of recommended changes that would require further investigation before decisions could be made about potential changes, and a third list identified possible changes to the system in the future that would require an investment of more time, resources, information, research, etc.

At the same time, NYSILC collected information from the testers using its survey instrument. The collective results have been summarized below.

A. Focus Group: Eleven individuals.

Disability types:

Physical = 3.

Sensory = 4. (One blind, three low vision)

Cognitive = 1.

Mental Health = 0.

Multiple = 3.

None = 0.

· Registered to vote?

o Yes = 11 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%

· Frequent voter?

o Yes = 9 = 82%.

o No = 1 = 9%.

o Not sure = 1 = 9%.

· Familiar with accessible voting technology?

o Yes = 7 = 64%.

o No = 2 = 18%.

o Not sure = 2 = 18%.

B. Voting Access Interfaces.

Identify the component(s) of the machine you used while voting:

(Check all that apply)

Monitor/magnification = 7.

Monitor/screen contrast = 3.

Audio feedback/headset = 9.

Handheld tactile device = 9.

Sip and puff device = 2.

Rocker panels = 1.

C. Overall Preference.

On a Likert scale from 0-10 (0=worst, 10=best), how would you rate the overall performance of this voting machine?

Range score: 2.5 to 8.

Average score: 6.36.

D. Time Study.

Ballot Description: Extensive. Actual primary ballot. Seven races. One race asked the voter to select (10) of (24) candidates. Another race prompted to voter to chose (3) of (7) candidates.

· Voting Function: Range (8-25 minutes), Average (18 minutes).

· Ballot Print/Mark Function: Range (2-3 minutes), Average (2 minutes).

· Verify Ballot Function: Range (2-4 minutes), Average (3 minutes).

· Total Vote Time: Range (11-33 minutes), Average (23 minutes).

E. Independent and Private Use.

· Were you able to vote privately and independently?

o Yes = 8 = 73%.

o No = 2 = 18%.

o Not sure = 1 = 9%

The two individuals who stated “no” cited the need for more privacy. They felt that the system, which allows ballot marking on one end and scanning on the other, is too wide open and needs some level of separation between the two activities. The vendors talked about a pop up screen, but until it materializes, the testers rated their experiences based on what was present that day.

Another tester responded “not sure” because the vendor showed them a privacy sleeve, but could not actually demonstrate how it worked because the printing tray shorted out and had to be dismantled. As a result, they could not show how the ballot went into the privacy sleeve and was fed confidentially into the scanner. Therefore, the tester replied “not sure.”

F. Overall Comments.

• The handheld tactile device worked well, but there were places where the prompts could be better or places where there were no prompts at all (i.e., like for write in candidates).
• The doubled-sided ballot printing tray broke during use and had to be dismantled. Fortunately, the ballot being used was only one-sided and printed for the remaining testers.
• The scanning device had some difficulty with the first ballot, but eventually worked successfully.
• A write in candidate vote was registered as an undervote during the verification process. It needs to be adjusted to record it as a vote.
• Audio was rudimentary and slow.
• Eliminate reference to “chevron” which is confusing and stick with more familiar reference to “arrows” (up, down, left, and right).
• Left arrow should say “contest” and right arrow should say “candidate,” using Grade 2 Braille, which is standard for adults. Note: A national expert identified during the group discussion that Grade 1 Braille is more appropriate for all users.
• Instructions for the use of all interfaces need to be standardized. Right now, the vendor only provides the user with instructions for the use of the handheld tactile device. They need to provide instructions for how to use magnification and contrast, the audio and headset, sip and puff device, and rocker panels.
• More time could be saved during the ballot printing function if it could mark a pre-printed ballot.
• No instructions were provided in how to verify a ballot. The process would be more private if the voter received instructions from the system. Also, due to the separation of the scanner unit from the ballot marking device, there is a need for a curtain to increase privacy.
• Audio/headset with sip and puff device was okay but not needed. Sip and puff device was utilized with sight of choices on monitor. While the sip and puff device worked, the tube would come out of the testers mouth on occasion and have to be reinserted. At the start of the session, it also increased the build up of saliva which blocked the tube.
• Second sip and puff user found the tethered line with a headset to be too intrusive and the control of the device unstable. Tester recommends they scrap the present device and replace it with a sip and puff device that is a stationary tube, attached to a flexible and adjustable arm, so a voter can position themselves under it and have more control.
• The screen contrast, audio feedback and headset and handheld tactile device features were functional. The text magnification worked, but would have been better if it was somewhat larger in size. Also, the magnification and contrast buttons were unmarked and on the cart. They should at least be mark in bold print for the voter on the cart. Some might argue that they should be buttons under the control of the voter on the handheld tactile device.
• The pale color used to highlight the candidate choice being considered is not obvious enough. It should be brighter. Note: A national expert identified during the group discussion that, to better address color blindness concerns, the frame of the cell should be highlighted in bold black.
• There are too many symbols on the machine. The symbols on the handheld tactile device need to be more direct. What are the orange buttons at the top for? Tester was not aware of the contrast feature. It needs to be identified at the beginning of the session. Perhaps it was missed due to the lack of labeling and/or instructions.
• Was able to read from screen with magnification and able to vote using audio feedback/headset and handheld tactile device. Could not read the actual printed ballot. Font too small.
• For audio instructions, it should tell the voter how many candidates are in the list they are about to select from. Difficulty skipping to the next contest from manual write-in mode.
• Audio instructions were too long and distracting. For handheld tactile device distinguish buttons by shape AND color in a consistent manner through out instructions.
• Would be a lot easier if the monitor had touch screen capability. Need more basic instructions for how to use the handheld tactile device. Found the red/green two button panels easier to use than the handheld tactile deice. Need instructions for how to use the red/green two button panels.

G. Overall Observations.

The system eventually accommodated all users. It appeared to be a functional system but could benefit from several suggested improvements made by the focus group testers. The vendors were extremely receptive to making the requested changes.

It should also be noted that the increased voting times appeared to be due to a variety of factors: instructions provided to users at first interface, the lack of a touch screen, and the need to navigate through the ballot race by race, candidate by candidate, frame by frame, and a more extensive ballot – all of which is time intensive.

II. Premier AutoMark (February 20, 2008).

The testing occurred on one day during half hour time slots per person. The vendor has tested its device with the disability community over the past few years. Its product has also been used in elections in other states across the country. As a result, the vendor allowed NYSILC to set up the focus group testing and use its survey instrument to collect information. The results have been summarized below.

A. Focus Group: Fourteen individuals.

Disability types:

Physical = 5.

Sensory = 1.

Cognitive = 2.

Mental Health = 2.

Multiple = 1.

None = 3.

· Registered to vote?

o Yes = 14 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%

· Frequent voter?

o Yes = 14 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%.

· Familiar with accessible voting technology?

o Yes = 13 = 93%.

o No = 1 = 7%.

o Not sure = 0 = 0%.

B. Voting Access Interfaces.

Identify the component(s) of the machine you used while voting:

(Check all that apply)

Monitor/magnification = 11.

Monitor/screen contrast = 6.

Touch screen = 13.

Audio feedback/headset = 5.

Tactile buttons/panel = 4.

Sip and puff device = 1.

Rocker panels = 0.

C. Overall Preference.

On a Likert scale from 0-10 (0=worst, 10=best), how would you rate the overall performance of this voting machine?

Range score: 7 to 10.

Average score: 8.43.

D. Time Study.

Ballot Description: Standard. Sample ballot of “famous names.” Eight races. Multiple parties. Two propositions.

· Voting Function: Range (3-12 minutes), Average (7 minutes).

· Ballot Print/Mark Function: Range (1-2 minutes), Average (1 minute).

· Verify Ballot Function: Range (1-2 minutes), Average (1 minute).

· Total Vote Time: Range (6-15 minutes), Average (9 minutes).

E. Independent and Private Use.

· Were you able to vote privately and independently?

o Yes = 11 = 79%.

o No = 1 = 7%.

o Not sure = 2 = 14%

The one person who stated “no” cited the need for more privacy. They felt that the system was too open and required a booth or some kind of covering to guarantee privacy.

For the two individuals who responded “not sure,” one did not offer up any written comments to help identify why. The second person felt that they would have more independence over the autocast feature with the sip and puff device if they put the appropriate directive (sip or puff) on the screen and in audio with the corresponding autocast option (cast marked ballot into drop box, return marked ballot to voter).

F. Overall Comments.

• I thought the machine was very easy to use and understand.
• Very easy to use.
• Write-in candidate’s name was not listed during the audio verification. It was marked on the ballot sideways.
• Worked very well.
• Very accessible. Quick and easy to use.
• Machine functioned well.
• Concerned about positioning of machine. Should face wall for privacy. Ideally, machine should be on an adjustable table that could be raised or lowered in a range of 4 inches to better accommodate voters. Need a crank and a 4 inch adjustable range for the scanner. Voter could benefit from more than the 1 minute tutorial.
• Would like unit at proper height. Would like a privacy screen. Wants to make sure polling site is accessible.
• Screen magnification and touch screen nice options. It was simple and easy to follow.
• Voter likes the various options. Easy to use. It should have a quick demo at the beginning that goes through the different features.
• It was very accessible and easy to use.
• Loved using the new machine. Touch screen worked well.
• Replace current sip and puff device (tether line with headset) with a stationary tube, attached to a flexible and adjustable arm, so a voter can position themselves under it and have more control. Note: Also need to make changes to autocast options related to the sip and puff. See II. E., paragraph two.

G. Overall Observations.

The biggest surprise is that most people tend to consider a touch screen as a general user accommodation. It was fascinating to see how many people with disabilities attempted to use the touch screen whenever possible as an accommodation of first choice, then move to a different interface if they couldn’t manipulate it. In II. B., thirteen of fourteen testers used the touch screen in combination with other accessible interfaces. Many were able to use this option favorably and thus keep voting times down as well.

The one major change the vendor may want to consider would be to move away from a tethered sip and puff line with a head set and switch to a more stationary model. This would include changes to autocast options related to the sip and puff for audio and screen instructions. The unit should also be positioned properly for privacy and adjustable height.

III. Liberty Mark (February 26, 2008).

The testing occurred on one day during half hour time slots per person. The vendor has reached out to the disability community on occasion in the past to make improvements to its product. It looked to NYSILC to provide some independent feedback from people with disabilities about the usability of its machine. Subsequently, NYSILC set up the focus group testing and used its survey instrument to collect information. The results have been summarized below.

A. Focus Group: Twelve individuals.

Disability types:

Physical = 3.

Sensory = 3. (One low vision, two blind)

Cognitive = 1.

Mental Health = 2.

Multiple = 0.

None = 3.

· Registered to vote?

o Yes = 12 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%

· Frequent voter?

o Yes = 12 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%.

· Familiar with accessible voting technology?

o Yes = 12 = 100%.

o No = 0 = 0%.

o Not sure = 0 = 0%.

B. Voting Access Interfaces.

Identify the component(s) of the machine you used while voting:

(Check all that apply)

Large font on ballot = 1.

LCD screen (Magnification/contrast) = 1.

Push buttons on ballot = 6.

Audio feedback/headset = 7.

Tactile buttons/panel = 6.

Sip and puff device = 1.

Joystick = 0.

Two/four button panels = 2.

C. Overall Preference.

On a Likert scale from 0-10 (0=worst, 10=best), how would you rate the overall performance of this voting machine?

Range score: 1 to 9.

Average score: 6.17.

D. Time Study.

Ballot Description: Standard. Partial election ballot. Six races. Multiple parties.

· Voting Function: Range (2-15 minutes), Average (8 minutes).

· Ballot Print/Mark Function: Range (1 minute), Average (1 minute).

· Verify Ballot Function: Range (1-4 minutes), Average (2 minutes).

· Total Vote Time: Range (4-19 minutes), Average (11 minutes).

E. Independent and Private Use.

· Were you able to vote privately and independently?

o Yes = 7 = 58%.

o No = 3 = 25%.

o Not sure = 2 = 17%

For the three who responded “no,” two felt that it was due to the audio directions not being clear enough. One of two also believed that making the blind voter search for the ballot (tape) at the top of the machine was not good. They felt it set up a situation where the poll worker was likely to intervene and was not independent or private. The third required assistance with the tactile buttons on the panel and with the insertion of the ballot (tape) into the verifier which interrupted their privacy and independence.

For the two who responded “not sure,” one noted that it took the vendor several attempts to get the ballot verifying device to read back their choices, so they were not confident that it would work in an election situation. It resulted in their “not sure” response. The second person that responded “not sure” did not offer up any written comments to help identify why.

F. Overall Comments.

• A light reflected a glare on the ballot that was difficult for one tester.
• The buttons on the ballot were difficult to push.
• The unit is on a stand that allows the voter to have it tilted toward them. If a person is in a wheelchair, it should be tilted toward them to maximize their reach.
• Audio instructions were incomplete for a blind voter. It was difficult to navigate the ballot based on the lack of prompts. Speech needs improvement. Instructions must be clearer, more detailed and speech pattern needs to be smoother. It was hard to know how to work the buttons.
• Had difficulty with audio instructions. Needed to ask for instructions and clarifications. Speech not smooth. Also, for the two controls for rate of speech, it seemed to change only the voice pitch, not the actual rate of speech.
• Keyboard good for write-in candidates. Braille labeling improved since machine used previous time.
• The verifying device speech was easier to listen to. However, it was harder to feed the correct end of the ballot (tape) into the device. Some kind of orientation of the ballot needs to be devised for the blind voter for the verifying process.
• Angle of face of machine as presented too steep to read Braille. Machine at flat or horizontal level is preferred for blind voter.
• It is difficult to vote privately and independently when people are next to you talking. It is distracting. There should be more privacy. Also, the vendor should come up with a ballot box that the voter deposits their ballot (tape) into after they have verified it.
• The verified vote device took several attempts to read my ballot back correctly. It should work the first time.
• Full face ballot font was increased which made it easier to use.
• Assistance needed with tactile buttons on the panel and with the ballot verifying device.
• The audio instructions were difficult to follow.
• The unit could benefit from a privacy screen and the ballot verifying device should have a ballot box where the voter deposits their tape (ballot).
• The unit was simple and basic to use.
• The full face ballot format made it familiar in format to what people have voted on. The tactile directional buttons on the panel were somewhat difficult to push, but manageable. The paper ballot cutting feature worked by resting at the edge of the voting machine and not falling to the floor. However, the lightweight paper ballot may be difficult to deposit in a ballot box (which was missing). Liked the sip and puff device with joystick option even though I didn’t use it.
• Need more cues and instructions for the audio voter like how many candidates to choose for a contest.
• Improve marking for significant buttons like the “cast ballot” button.
• Add Braille markings for the buttons to increase and decrease the rate of speed for speech.
• Ballot should have some kind of notch or hole punched into it to indicate what end goes into the verifying device for the visually impaired voter.
• Impressed with the functional sip and puff device. Able to navigate and cast ballot successfully. Easy to do write in candidates as well. Tester cautions use of sterile sip and puff tips and recommends poll workers wear latex gloves during use.

G. Overall Observations.

The model that the vendor set up for testing almost appeared to be an enhanced version of previously displayed models at the SBOE. The full face ballot display was in large print and easy to read, the sip and puff device was a stationary model preferred by previous testers and doubled as a joystick, the voting unit was presented on an accessible tilt stand that provided greater reach to a voter in a wheelchair, and additional accessible interfaces were provided through the options of either a two button or four button selection panel. Also, the voter verified ballot tape was improved to hang at the edge of the voting machine, voting choices were no longer abbreviated and the font size was increased and the audio verifying device used on site worked for most testers after set up and difficulty was by-passed beyond the first user.

However, the system could benefit from the improvements recommended by the testers.

Attachment I

VOTING MACHINE SURVEY

Initials: Vote Time: Ballot Print Time: Verify Ballot Time:

Type of Disability: (Check one)

( Physical, Sensory (Hearing/Visual), Cognitive, Mental Health, Multiple, None)

I. Voting Access.

A. Identify which component(s) of the machine you used while voting:

(Check all that apply)

1. Large font on ballot.

2. LCD screen (Magnification/contrast).

3. Push buttons on ballot.

4. Audio feedback/headset.

5. Tactile buttons/panel.

6. Sip and puff device.

7. Joystick.

8. Two/four button panels.

B. Where you able to vote privately and independently? (Y) (N) (Not sure)

(Circle one)

C. Overall comments?

II. Overall Questions

A. On a scale from 0-10 (0=worst, 10=best), how would you rate the overall performance of this voting machine? (Circle one)

0 1 2 3 4 5 6 7 8 9 10

B. Are you registered to vote? (Y) (N) (Not sure)

(Circle one)

C. Are you a frequent voter? (Y) (N) (Not sure)

(Circle one)

D. Are you familiar with accessible voting technology? (Y) (N) (Not sure)

(Circle one)