1/10/2011 Group | Ivan Zverev, Arthur Leung, Yang Yang APSC479 SWIMMING CUEING INTERFACE DEVICE APSC 479 Swimmer Cueing Interface Device Group 1051 Arthur Leung Yang Yang Ivan Zverev Project Sponsor David Neufeld Submitted To: Jon Nakane, Chris Waltham Engineering Physics Project Laboratory The University of British Columbia January 10th, 2011 ii Executive Summary The goal of this project is to develop a wearable and waterproof device that enables a blind or visually impaired swimmer to swim laps in a swimming pool lane. The project aimed to provide a small and lightweight device that would not interfere with regular swimming strokes while providing the user with detailed instructions. This project serves to advance the current model of communications with visually impaired swimmers, which consists of tapping the swimmer with a soft-ended stick. This device will allow the user to be more autonomous, and use lanes that are not separated from the rest of the pool. To accomplish this goal, we have designed a waterproof device with wireless communications which will send directions to the user through haptic and audio cues. A. NET C# framework is provided for host-side control and signal interaction with the swimmer device, which is handed over to the project sponsor along with an SDK to allow for easy development and system expansion. The project allows for further research to be done regarding the effectiveness of audio and haptic cues, and to identifying the best mix of such cues. The team has aimed to use out of the box parts and solutions wherever possible. The resources for these devices were provided by the project sponsor and the Project Lab, and will be returned to the providers at the end of the project. Additionally, electrical measuring equipment, breadboards and PCB milling machines from the project lab were used. Because of time constraints, we were not able to perform adequate testing of the competed device before this report was completed. An addendum report will be submitted on January 17, 2011 which will include data from dry and wet tests of the completed device, as well as tests in a swimming pool. Additional information will also be included in the Addendum report, such as photographs of the device and PCB, and any changes to the device that may be made during or after the tests. iii Table of Contents Executive Summary ....................................................................................................................................... ii Table of Figures ............................................................................................................................................ iv Table of Tables ............................................................................................................................................. iv Introduction .................................................................................................................................................. 1 Objectives ................................................................................................................................................. 2 Discussion...................................................................................................................................................... 3 Mechanical Design .................................................................................................................................... 3 Waterproof Enclosure ........................................................................................................................... 3 Connectors ............................................................................................................................................ 4 Haptic Motors ....................................................................................................................................... 6 Electrical Design ........................................................................................................................................ 6 Overview ............................................................................................................................................... 6 PCB Design ............................................................................................................................................ 7 Testing ....................................................................................................................................................... 8 The Schematic ........................................................................................................................................... 9 Board Layout ....................................................................................................................................... 11 Device Software (Pegasus) Development ............................................................................................... 12 Overview ............................................................................................................................................. 12 Software Platform (Hermes) Development ............................................................................................ 12 Overview ............................................................................................................................................. 12 Methods and Testing Protocol ............................................................................................................ 13 State Diagram ...................................................................................................................................... 16 Demonstration Software ........................................................................................................................ 17 Conclusion ................................................................................................................................................... 19 Project Deliverables .................................................................................................................................... 20 List of Deliverables .................................................................................................................................. 20 Prototype Device ................................................................................................................................. 20 Software .............................................................................................................................................. 20 Financial Summary .................................................................................................................................. 20 Ongoing Commitments ........................................................................................................................... 20 Recommendations ...................................................................................................................................... 22 iv Table of Figures Figure 1. A snapshot of the Otterbox 8000 waterproof container. .............................................................. 3 Figure 2. A side view of the Otterbox SolidWorks model ............................................................................ 4 Figure 3. Scans of the OtterBox 8000 SolidWorks model from various angles. .......................................... 4 Figure 4. Specification of HR 30 12 socket waterproofed connectors ......................................................... 5 Figure 5. The SolidWorks Model for the milled Otterbox. ........................................................................... 5 Figure 6. The waterproof connector attached to the OtterBox. .................................................................. 5 Figure 7. A picture of the haptic motors used in the project assembly. ...................................................... 6 Figure 8: Photograph of Rev 2 of the PCB, used to test Xbee range ............................................................ 8 Figure 9. Photograph. A second photo of Rev 2 of the PCB, used to test Xbee range ................................. 9 Figure 10. Schematic for the 4th revision of the PCB. ............................................................................... 10 Figure 11: Board layout diagram for the final Rev of the board (Rev 4) ..................................................... 11 Figure 12. Hierarchical representation of the projects within Hermes. Projects depend on the projects immediately below them, with the exception of Hades, which depends on all connected projects. ....... 13 Figure 13. A screenshot of the Hades debugger. The log is currently used only to record the quantity of queued commands, and exits/entrances from Waiting states................................................................... 15 Figure 14. An UML state diagram depicting the state transitions for the Hermes application on one thread. Waiting is the only state depicted here that occurs on an individual thread, allowing Hermes to avoid freezing when confirmation takes excessively long. ......................................................................... 16 Figure 15. A screenshot of the Ariadne program. ...................................................................................... 17 Table of Tables Table 1. Specifications for the haptic motors used in the project assembly. The data was obtained from http://pagermotors.com............................................................................................................................... 6 Table 2. A snapshot of the Hermes.Interal.API public methods that developers can use to create their macro-languages. ........................................................................................................................................ 14 Introduction The swimmer cueing interface device (SCID) system is sponsored by a visually impaired engineering technologist with a background in electrical engineering and technical writing. The sponsor is legally blind and continues to lose his remaining vision. While medical treatment may restore some vision, the sponsor is promoting the development of assistive technology addressing problems faced by the blind community. This project focused on providing a device to explore ways to address the swimming challenges faced by the blind and visually impaired. The current communication method involves a guide following the visually impaired swimmer on land, and tapping his/her shoulder with a long, soft-ended stick if the swimmer is going off course. This is a very inefficient system, as the swimmer is both limited to a side lane and is constantly unaware of his/her surroundings, waiting for a tap to signify any changes in the situation. The trust between swimmer and assistant must be great, and the assistant has to continuously follow the swimmer, while limited to the crude method of tapping to warn him of multiple dangers (lane end, wrong direction, approaching swimmers). The objective of the project was to develop a device visually impaired swimmers could use to navigate swimming pool lanes. Ideally, the device would guide the swimmer through haptic (tactile) and audio cues to the correct position and direction in a lane. It may also be used to notify the swimmer when the lane end is reached, and alert the swimmer to potential collisions with other swimmers or objects in the pool. In terms of hardware of this device, Arduino boards and some parts such as pager motors were provided by the sponsor. Customized PCB fabrications have been provided by the engineering physics project lab. Of particular interest to the project was the field of haptic (tactile) and auditory interface design. Both general and specific applications to the blind community had been considered, as well as derivative applications to the sighted community. The project would provide directional, navigational, collision avoidance and other cueing to the swimmer via haptic and audio notifications. Similar to a turn-by-turn audio GPS in a car, the device directs the swimmer to swim in the correct side of the lane and in the correct direction. It also helps the user avoid collisions with other swimmers or objects detected in the pool. The swimmer device subsystem consists of a wearable device with auditory and haptic cueing capabilities, as well as wireless links for data communication with a ‘host’ system for transmitting cues to the swimmer, based on video monitoring of the pool. The project scope includes developing a wearable haptic audio cueing device for the swimmer, as well as the host- side wireless infrastructure for data transmission. The scope also includes creating the software that will allow a human/AI ‘operator’ to send cues to the swimmer over the mentioned wireless interface. The scope did not include the video analysis software and any AI algorithms for translating sensory data into haptic commands. The project provides a platform that can be useful in determining the optimal communication mechanism to be employed in this application. As such, it is beyond the project’s scope to determine this optimal mechanism; the scope only includes creating a wearable device that can be used to investigate this topic. The 2 project is designed for future development by a specialist in the field of Human Computer Interface (HCI), with a basic knowledge of .NET C#, or any language based on Borland C. By the end of the project build schedule, a final demonstration of the system will be done in which a human operator on the host computer provides real-time cueing commands to a person wearing the device. In this report, we have provided related technical aspects of the SCID in terms of electrical circuitries, software developments and mechanical designs in the discussion session. The corresponding finished work and tested results are stated in the conclusion part of this report. Future possible development and related research direction are concluded in the recommendation session. Objectives The project team set the following objectives to address the challenge of creating a reliable, inexpensive and portable swimmer cueing device.  Develop a wearable device capable of providing haptic and audio cues to the swimmer. The haptic and audio cues need to be easily identifiable, and wearing the leads and audio components should not interfere with the swimmer’s stroke.  Ruggedize the device to withstand the environment of a swimming pool. This means water proofing the circuitry and connectors.  Ensure that the communication data link will be reliable and fast enough to transmit cueing signals to the device at near-real time, since some collision situations may appear moments before the collision is set to occur.  Develop a software suite that handles communication between a host-side workstation and the swimmer cueing device. This suite should implement an easy, graphical interface to send cues to one of potentially many swimmers. Additionally, the system needs to be easily extendable to facilitate further investigative development, specifically in the area of the cue ‘language’. As part of this objective, extensive SDK documentation needs to be compiled, complete with examples on extending the most important aspects of the software, especially if they are likely to change. 3 Discussion Mechanical Design The mechanical components of the device consist of two major parts: 1) Central processing body that contains a waterproofed enclosure for the electrical circuitries and a output connector 2) Haptic cueing vibrators (pan cake motors) to direct the swimmers Waterproof Enclosure The core of the mechanical design consists of a waterproof enclosure. The dimensions of the enclosure are set to be 5cm x 5cm x 4cm, which is originally set to be capable to store 3 PCB boards within it. The dimensions of the PCB board are measured as 4.5cm x 4.3cm x 0.6cm. However, due to the limitation of the hardware and tooling support of the project lab, a purely customized enclosure is unlikely to be made. The goal was then set to look up some finished waterproof protection products on the market. Although a few options of waterproof cases were available, a few negative points were brought to attention. 1) Our assessment of for pelican boxes as a candidate found that the gasket seal is part of the entire internal rubber shell for padding. Any tooling done for cable feeds would have to break this padding as well. 2) The GSR Lexan resin case was examined as an alternative to Pelican boxes.. However, the sides of the case did not seem to provide enough room for cable feeds. 3) iPod Nano cases were found with built in speakers and click wheel interface that seemed ideal for our project design. If we used this case, a more compact final SCID design with all components on one PCB might be designed to fit in it. However, the case did not provide a ready solution for a hermetic seal. Ultimately, a waterproof enclosure from Otterbox, known as an innovator of protective solutions for the leading global handheld manufacturers, wireless carriers and distributors, was chosen to be the most compatible case for this project. The product is named OtterBox 8000, which is designed to withstand submersions up to 100 feet. It is specialized on providing superior protection for small digital cameras, cell phones, GPS units, etc. The case also comes with a belt clip for added convenience to swimmers carry the SCID. The blue print as well as snap shots of the SolidWorks model for the OtterBox 8000 is illustrated below. Figure 1. A snapshot of the Otterbox 8000 waterproof container. 4 The features of OtterBox 8000 are listed as following:  Exterior Dimensions: 3.354" x 4.542" x 5.921"  Interior Dimensions: 1.935" x 2.831" x 5.188"  Weight: 9.6 oz. (272.15 grams) Connectors In our design, the central processing system and the feedback haptic cueing system are separated. A waterproofed connector was therefore required. Hirose HR30 12 socket front panel circular connector mated with associated plug was used. The figure below shows a HR30 12 socket connector with its dimensions. Figure 2. A side view of the Otterbox SolidWorks model Figure 3. Scans of the OtterBox 8000 SolidWorks model from various angles. 5 Figure 4. Specification of HR 30 12 socket waterproofed connectors This connector, though, accommodates a 3 mm maximum panel thickness, which means the wall thickness of Otterbox 8000 case exceed the limit. Therefore, the OtterBox would nott work directly without modification. Considering the case’s recommended tight cutting tolerances, a milling machine was used for thinning the wall thickness in order to fit the connectors. The SolidWorks model of the modified enclosure is shown below to illustrate the thinning process. The HR30 series connector has a clearance that is smaller than other both circular and D-sub waterproof connectors in our research. We chose the 12 socket one to support the 6 tactors for the haptic vibrators. This eliminated the problems of the tactors sharing a common ground with less than 12 pos. Figure 5. The SolidWorks Model for the milled Otterbox. Figure 6. The waterproof connector attached to the OtterBox. 6 Haptic Motors Other than the central processing system of the device, we use 6 pager motors to direct the visual impaired swimmers to avoid upcoming barriers. The motors are placed in the three directions of left, right and center of a pair of swimming goggles. The swimmer can be notified of specific actions with predetermined combinations of motor vibrations. Following is a snap shot of a pager motor that we used. The specification of the motor is illustrated below in Table 1. Table 1. Specifications for the haptic motors used in the project assembly. The data was obtained from http://pagermotors.com Voltage Rating 1.5v DC Current Rating 70 +/- 20 mA Resistance Arm. 7.0 +/- 1 ohms Insulation 1 M ohms min Dimensions diameter 0.394" (10.0mm) width 0.138" (3.50mm) lead length: 0.67" (17mm) Weight 0.99 grams (0.035 oz) Comments The entire backside of this motor is covered in adhesive material and once exposed will stick to most surfaces with little reinforcement Electrical Design Overview The goal of the electrical component was to create a device that can receive wireless commands that may be then used to control a set of (at least) 6 vibro-tactile motors, or send audio cues to the user through a set of headphones. The device then must fit into the chosen waterproofing enclosure, and have cabling for headphones and tactile motors. Figure 7. A picture of the haptic motors used in the project assembly. 7 For the device microcontroller, we chose to use an Arduino because of its availability and good documentation. The Arduino serial ports could be used to communicate with the wireless transmitter/receiver, while the PWM outputs could be used to control the tactile motors. The analog inputs and outputs could be used for further peripheral devices that may be desired by the sponsor. The board we chose was the Arduino Pro Mini from Sparkfun website, as it would be small enough so the device could be miniaturized if required. In addition, the small size allows for the addition of further Arduino boards if more processing power or outputs are required, as there are only six PWM outputs on the board. Through the research done for our proposal, we found the optimal choices for wireless communication were radio and Xbee for range and underwater attenuation. Xbee was chosen because of its ease to configure and reliability, and because the Xbee incorporated both a wireless transmitter and receiver for two-way communication. While the Arduino could drive motors directly off of its PWM outputs, we chose to use a source driver. This ensures that the Arduino is decoupled from the motors, and prevents issues with current draw or spikes. The device we chose to use was the MIC2981b, since it is a high current device and allows for the use of a separate battery for the motors if required. To send the audio cues to the user, we had to choose between a streaming or stored cues device. We found that the Arduino does not have enough processing power to drive both the motors and stream audio from the Xbee, so we chose to use a stored cues device; however we also designed a system to us a streaming audio chip, along with another Arduino Mini with the sole purpose of streaming audio. The stored audio cues device we chose was the SOMO- 14D, as it uses a SD chip to store the data and could be easily used. Finally, several other peripheral devices were required, such as an accelerometer, and a potentiometer. The accelerometer was desired to determine the facing of the swimmer, whether they were facing up or down. This would allow for simpler coding to change the directions to the swimmer after they have flipped. The potentiometer was desired to adjust the strength of the tactile cues, and possibly also the volume of audio cues. The output from both of these devices could be routed to the analog inputs of the Arduino. PCB Design To route between the Arduino and the various other components, we chose to mill a custom PCB. This PCB was designed using the EAGLE Cadsoft program, and the gerber files were then imported into the software used by the PCB milling machine (the LPKF Protomat S62). Headers were designed on the PCB to allow for easy mounting and removal of the various components, and components were mounted on both sides of the PCB to reduce space required. The first versions of the PCB were very compact (see Figure 8), and this resulted in very little space requirement in the enclosure; however, there was difficulty soldering the components on the board. Since the Otterbox enclosure was much larger than the PCB, we 8 decided to expand the PCB to ensure that our device would be properly soldered. Voltage regulators were added because the Xbees and Arduino required different supply voltages, and this gave the added advantage of protecting the circuit from incorrectly applied voltages. Initially, a surface mount accelerometer was used—because of the difficulty in mounting, however, we switched to a through-hole device obtained from the project lab. The new chip included a gyro and an Analog Devices triple axis ADXL335 accelerometer. In addition to enlarging the PCB, a header was added for motor outputs in place of test points. This simplified cabling in the device inside the Otterbox. A headphone jack was added to connect a set of headphones, and an input header was added to allow for easy future access to the Arduino analog inputs. This allows for the sponsor to easily add any peripheral devices that may be required. Testing Because of availability, our early PCB tests used the Series 1 Xbees which are rated for 75ft range. These tests were done to ensure that the attenuation due to the device being placed in the water would not limit the Xbee range. The desired range is for one Xbee to at least span the length of a swimming pool, which we assume to be approximately 50m or 164ft. While it was unlikely that the Series 1 Xbees would have sufficient range, the tests were useful to determine the degree of attenuation to be expected when the device is placed underwater. The transmitting end Xbee (wired to a laptop) was placed under approximately 4 inches of water, and the device was moved away starting at a range of 5ft. A command was sent from the laptop to set one of the Arduino outputs high, which would then power one of the motors. From these early tests, it was found that the range of the Xbees were significantly shortened by placing one chip under water. In the control test, it was found that the Xbees had a range of 75-80ft in a straight hallway, above water. However, the range with one end underwater was between 10 and 30ft. We therefore determined that the range of the Xbees is approximately halved with water attenuation. During our later round of testing, we used the stable second revision of the PCB (see Error! Reference source not found.). This chip did not have a mount for the audio chip, and LEDs were mounted to the device in place of motors. Since the cabling for the enclosure was not finished, this design allowed us to test the operation of the device underwater by placing the entire device in the transparent enclosure, then sending a command to turn the LED on or Figure 8: Photograph of Rev 2 of the PCB, used to test Xbee range 9 off. We tested the Series 2 Xbees using this setup, which were rated for double the range of the Series 1 Xbees. While the range was significantly better than the Series 1 Xbees, it was still unreliable and had a maximum distance of between 35 and 60ft. Because of the significant attenuation, we ordered a set of Series 2 Pro Xbees, which are rated for very long ranges; up to approximately 360ft indoors and a mile outdoors. We tested these Xbees with the same setup as the Series 2 Xbees, and the range exceeded expectations. From the indoor testing, the range was over 150ft down a straight hallway. The signal was still very reliable at the maximum range, with all commands being received and no dropped packets or delay. From this test we determined that the Series 2 Pro Xbees would be sufficient for our requirements. The Schematic Error! Reference source not found. shows the schematic for the final version of the circuit board. As discussed above, two voltage regulators were required to regulate the power to the various devices. We have designed the device to use a 9V battery, as these can be easily obtained for users. The source driver and Arduino both require a 5V supply, while the audio chip, accelerometer, and Xbee require 3.3V. Figure 9. Photograph. A second photo of Rev 2 of the PCB, used to test Xbee range 10 Figure 10. Schematic for the 4th revision of the PCB. Outputs 3, 5, 6, 9, 10, and 11 on the Arduino are PWM outputs, and so these are routed to the first 6 inputs of the source driver. The outputs of the source driver are then routed to 100kohm resistors to limit the current supplied to the motors, preventing overheating. The outputs were then routed to a header, which also includes a connection to ground which will need to be split in the cabling to connect to the ground of each motor. Besides the Source Driver, the remaining devices were mounted to the PCB using headers. In the board design, since the two headers for each device were not linked as once device, the headers could be easily flipped or rotated to allow for the device to be mounted on the top or the bottom of the PCB. Some headers were included only as a structure mount for the components, and do not have a function electronically, such as the left header for the Xbee. The Dout and Din pins of the Xbee were connected to the Rx and Tx pins of the Arduino, respectively. The Y-Axis output of the accelerometer is routed to the analog input of the Arduino, A0, and the remaining analog inputs are routed to a header to allow for the addition of other devices such as the potentiometers. The data pin of the audio chip was connected to capacitor and the Digital Output 8 of the Arduino. In addition, the Next, Play, Previous, and Reset pins of the chip were routed to a header, so that switches can be added to manually control the audio chip in necessary. The speaker + and – outputs from the audio chip were routed to an audio jack, as well as the header with the spare analog inputs. This allows for multiple ways to connect the headphones. 11 Board Layout Figure 11: Board layout diagram for the final Rev of the board (Rev 4) In the second revision of the PCB, components were layered one on top of another to reduce the footprint. This resulted in some difficulty in assembly, and so in the later versions we chose to have only a single component at each section of the PCB (see Figure 11); however components were still mounted on both sides of the board to simplify routing. To account for the possibility of high currents, the ground and power traces were enlarged. The signal traces were kept at a minimum of 24mils to account for the imprecision of the milling machine. Some routing decisions we made included the inclusion of three grounds in the header for the Play, Previous, and Next pins of the audio chip, routing a trace through an unused pin on the Arduino, and combining the header for the Arduino analog inputs and the speaker output pins from the audio chip. The three grounds in the audio chip header are useful to allow us to easily create switches at each of the pins, which are needed to manually control the chip for testing purposes. One trace was routed through an unused pin of the Arduino because we were unable to route between the output from the accelerometer and the input to the Arduino 12 without otherwise crossing another trace. This will not be a problem because the affected pin can be removed from the header before soldering. Finally, we chose to combine the speaker outputs and the Arduino inputs to the same header to reduce the number of headers required; it would not be difficult to mount separate connectors to the header, one leading to headphones and another leading to potentiometers or other peripheral inputs. In addition, these speaker inputs will only be used for testing as there is also a speaker jack connected to the same speaker outputs. Device Software (Pegasus) Development Overview Pegasus is the device side of the software, and controls the Arduino system. The code was written in the Arduino editor using Arduino-C. This subsystem was designed to be as lightweight and derivative as possible and to leave as much processing as possible to the server side of the system. It consists of a variable length command reader method that registers the sent command, and a collection of nested switch statements that act like a dictionary to identify the command and act out the appropriate response. The system also features timing code for using the embedded audio capabilities of the Pantheon device. Software Platform (Hermes) Development Overview The software platform, codenamed Hermes, was developed in .NET by Ivan Zverev and exposes the device capabilities for future tools developers. Hermes should not be confused with a final tool used for navigation in the pool. The purpose of the platform is to simplify and standardize the development of such navigation tools, as well as any other tool that uses the device, by handling the device communication and exposing easy to use methods that cover the scope of what the device can achieve. It should be explicitly stated that the Hermes platform does not provide a solution for the particular application of guiding a swimmer through a pool1. Rather, it gives the developer an easy to use encapsulation of the device capabilities. The developer’s only responsibility is to create a ‘macro-language’ – methods that consist of calls to methods defined in Hermes.API.Internal. The developer does not need to worry about connections or timeout protocols, since that is handled internally. This gives the platform great flexibility in supporting solutions for environments outside the scope of the initial challenge. For example, while the Hermes.API.Interal (detailed in Table 2) exposes commands like SetTactorState and GetTactorState, the developer can combine these commands into a macro-language consisting of commands like “TurnAround”, “ChangeLanes”, “SlowDown” and “Turn”, which allows the developer to create a simple language. 1 The team was not directly responsible for creating the macro-language that would be used in the swimming pool application. 13 Hades Hermes.API.Internal Hermes.Internal.Protocol Hermes.Internal.Transport Methods and Testing Protocol Hermes was developed using Visual Studio C# Express and uses the .NET Framework 4.0. The project was to be ported to Java by a member of another course, but time constraints stalled the porting work. As of January 7th, 2011, the work remains incomplete. Hermes is a modular hierarchical platform, optimized for replacement of layers. This allows future developers to replace entire sections of the platform if they so choose. The project is split into four main components: API.Internal, Internal.Transport, Internal.Protocol and Hades. The hierarchical dependency is depicted in Figure 12. The Transport layer is responsible for the low level messaging between the Serial port and the software. This layer handles connections to a serial port, writing data to the device and handling raised events for received data from the XBee module. The Protocol layer name is a misnomer because the layer does not handle the Protocol (although it did at one point). Instead, the Protocol layer handles the command queues for multiple instances of the Pantheon device class (which will be discussed later). The QueueManager class handles populating and accessing the queue. The QueueManager makes sure that all commands are processed but rather than immediately sending them to the Transport layer, the QueueManager waits for the device to confirm that the previous command has been processed before sending the next one. When the QueueManager receives a command, it creates a new background thread to process the command. This allows the program to continue operating while the command waits for its turn to be processed. When a command is finally sent, QueueManager sets a confirmation Boolean to true, indicating to other classes that it is waiting for a response from a command. When a valid response (or device layer exception) is returned by the transport layer, QueueManager sets the confirmation Boolean to false, returns the response to the initial source of the command and loads the next scheduled command for processing. The main visible layer for outside developers is the API.Internal layer. This layer defines all the functionality of the device and is the layer developers should use to construct their macro functions. A basic overview of the current interface is provided in the table below. Figure 12. Hierarchical representation of the projects within Hermes. Projects depend on the projects immediately below them, with the exception of Hades, which depends on all connected projects. 14 Table 2. A snapshot of the Hermes.Interal.API public methods that developers can use to create their macro-languages. String SetTactorState(String DeviceID, TactorNames Name, TactorStates State); /// /// Sets the state of a specific tactor to a specific intensity /// String SetTactorArrayState(List TactorInfo); /// /// Sets the state of a list of tactors by providing a list of tactors and the associated strengthsw /// TactorStates GetTactorState(String DeviceID, TactorNames Name); /// /// Returns the intensity of a specific tactor on the user device. /// List GetTactorArrayStates(String DeviceID, List TactorNames); /// /// Returns the intensity of multiple tactors /// void SetVolume(String DeviceID, VolumeLevel Volume); /// /// Sets the volume of the device's audio commands /// void PlayCommand(String DeviceID, int CommandAddress); /// /// Play a command file on the user device /// void StopCommand(String DeviceID); /// /// Stop the playback of the command /// AccelerometerValue GetOrientation(String DeviceID); /// /// Find out which way the device is currently oriented. /// Int16 GetNetworkID(String DeviceID); /// /// Returns the network ID. Currently not used. /// Int64 GetTimeOnline(String DeviceID); /// /// Returns the amount of time, in milliseconds, that the device has been operational. /// DeviceStatus GetDeviceStatus(String DeviceID); /// /// Gets all queryable elements of the Arduino. /// void FindDevices(); /// /// Find all devices that respond to the server station. /// This layer is also responsible for maintaining the actual symbols used to translate this command. Hermes uses Language Integrated Query (LINQ), a new feature in .NET 4.0 to obtain command information from XML documents quickly and painlessly. Commands are stored in 15 an XML document, as well as information about parameters, the methods description and the expected outcome. Hermes uses a LINQ query to gather this information when a call is made by the user to a certain command. This allows the method that sends commands to be generic for all types of commands without sacrificing flexibility, since the query can obtain any information and the XML document can be edited to provide more information if needed. Figure 13. A screenshot of the Hades debugger. The log is currently used only to record the quantity of queued commands, and exits/entrances from Waiting states. Finally, the topmost layer is the Hades debugger, a Windows Form Application made to debug and test the Hermes protocol. Hades uses a simple GUI to publish information about the queue and allow the user to type in commands to see how the system responds (see Figure 13). This program is invaluable to determining if the device is responding and what the responses are. Hades is a pseudo-third party program, because it doesn’t use any functions that are hidden from third party developers, but was developed by Ivan Zverev alongside Hermes. The system was tested extensively during the construction of the device, with each change getting tested to avoid as many bugs as possible. The tests included range tests with the hardware, command responses, queue jam simulations (achieved by sending multiple commands instantaneously), as well as port jamming (multiple write attempts to the same port simultaneously). This testing helped identify bugs, which led to numerous releases throughout the course of the build period. Release notes that accompanied software releases will be included in the Addendum report. Although the testing helped eliminate bugs, very little metric testing was done. Additional metric testing, specifically time delay between sending a command with an empty queue, average delay on a queued command and command processing time are tests that would help 16 optimize the system. These tests will be conducted and their results will be included in the Addendum mentioned earlier in this report. State Diagram The state diagram in Figure 14 is meant to demonstrate the basic states that the software should reside in, as well as queues that the system uses to leave a particular state. It should be noted that Scheduling can happen at any moment, and any state can enter scheduling without interrupting its hierarchical chain, because command processing is done on individual threads. This decision was made to avoid lock ups in the GUI when a command sent to the device takes a long time to respond. do/ Create Command Scheduling / [User sends a command] do/Queue Command Sending/ do/Write Command Writing/ Queue isn't Waiting entry/Queue wasn't empty exit/Queue is not waiting for a response Waiting / Queue is Waiting for Confirmation Idling do/ProcessCommand Reading / [Response received] / [Respose Timeout Exceeded] do/ Clear Queue waiting flag Purging / Number of allowed timeouts exceeded / Queue empty Figure 14. An UML state diagram depicting the state transitions for the Hermes application on one thread. Waiting is the only state depicted here that occurs on an individual thread, allowing Hermes to avoid freezing when confirmation takes excessively long. From the diagram it can be observed that the most entered states are Idling and Waiting. Idling represents a free state, essentially the Initial state of the software, where no commands are in queue, no responses are being processed and no commands are being sent. The only way to leave this state is to schedule a command to be sent, while the only ways to return to this state are to handle all received messages or purge the queue in case of an unresponsive system. Waiting is similar to Idling in that the system isn’t actively processing any requests, but is different because the system is expecting a response for a sent message, and may potentially have other messages in the queue. Any time the system sends a message to the device, the 17 software enters this state until a ‘DataReceived’ event has been triggered. This is the most common and expected way of exiting the Waiting state, but the state may also be terminated by a connection timeout, in which case the system will enter the Writing state and try to write again, or the Purging state, if the maximum number of connection amounts was reached. It should be noted that because Hermes is a multi-threaded application, strictly speaking there are situations where it can enter a multitude of the aforementioned states. The most common is Waiting and Idling, since the Waiting state is handled on its own thread, to avoid lockups in the GUI. For simplicity’s sake, the diagram treats this multiple state as Waiting, defining Idling as a free state with no child Waiting threads. By design, the Waiting state is concurrent with the Scheduling and Sending states, implying that while the Waiting state has a 1...1 multiplicity, its lifetime is 1...n, since any number of commands can be scheduled before the system leaves a Waiting state. The Waiting state has a 1...1 multiplicity with itself, since only one command can be written from the serial port at any time2. Demonstration Software The demonstration software, codenamed Ariadne, is an interactive application meant to demonstrate the hardware and Hermes software package without requiring a pool. The view the user sees at startup is shown in Figure 15. The software is structured like a primitive video game. Two users have the task of guiding a swimmer through a virtual pool with various objects. One user is blindfolded and controls the swimmer with the ‘W’,’S’,’A’ and ‘D’ keys, while the second user observes the on screen position of the swimmer and obstacles and sends queues using the Pantheon device. The objective is to complete the 8 levels in the game in the least amount of time. Figure 15. A screenshot of the Ariadne program. 2 It should be explicitly stated that these multiplicities are derived from using one end device. The Waiting state has a 1...m multiplicity if m end devices are connected. However, since the significant portion of the testing was completed with one device, the author provides the multiplicities for the 1...1 server-user situation. 18 Users may flip the swimmer using the ‘X’ key, but only the areas designated by the dashed yellow lines. The game ends if the blue swimmer (player character) collides with another swimmer or the pool boundaries. Like Hades, Ariadne is a pseudo third-party application that demonstrates the capabilities of the hardware device, and the ease of extendibility of the Hermes platform. To create this application, the developer doesn’t need to know any of the internals of Hermes, and is only required to create a macro-language for his particular application. 19 Conclusion In this project, we designed a waterproof device to direct visual impaired swimmers using haptic and audio cues, through wireless communication with an operator who will send the necessary commands to the cuing device using a host laptop. We have completed an implementation of the device using off the shelf components, and have provided the schematic, photographs, and datasheets for the design to reproduce or modify the device. In addition to the hardware for the device, firmware for the device has been provided as well as software for the host laptop which allows for flexible implementation of the system. This device will allow for research into the effectiveness of audio and haptic cues to direct visually impaired swimmers, and research to determine the optimal placement of haptic motors. As noted in the Executive Summary, dry and wet tests of the completed device were not performed due to time constraints. However, an Addendum report will be submitted on January 17, 2010 which will include the results of testing of the device, complete with haptic motors and audio cues which were absent from the system during the tests discussed in this report. Additional information will be provided, such as finalized schematics and PCB or mechanical designs if changes are made during the testing process. 20 Project Deliverables The project deliverables include one version of the working prototype device, the complete source code for the Hermes platform, the Hades debugger and the Ariadne demo application. The team will also deliver the documentation for the software, the hardware documentation and specification, as well as this final report. List of Deliverables Prototype Device The device will be submitted in an assembled and functioning state. Although originally prpmised, the device will not be in an ergonomic container, but will be enclosed inside an OtterBox 8000. The device will also not feature streaming audio support, instead relying on stored audio commands. No battery will be provided with the device. Software The software will be submitted in two versions. The first version, marked Unverified, will be a snapshot of the latest written code, although the code has not been tested as excessively. The Verified version will be a snapshot from an earlier functioning release that was used for all the testing. Hades will be included in both versions. No major changes are planned for the program as of January 8th, 2011. Ariadne will be included with one version of the source code, since the program did not receive any changes after testing. Financial Summary # Description Quantity Vendor(s) Cost Purchased By: Funded By: 1 XBee Series 2 Pro Transceiver 2 Sparkfun $62.48 Project Lab Project Lab 2 Arduino 5V 20Hz Microprocessor 2 Sparkfun $37.66 Ivan Zverev Project Lab/Sponsor 3 916 MHz receiver/transmitter pair 1 Sparkfun $10.00 Ivan Zverev Project Lab/Sponsor 4 434 MHz receiver/transmitter pair 1 Sparkfun Ivan Zverev Project Lab/Sponsor 5 Somo-14D 1 Robotshop $24.68 Ivan Zverev Project Lab/Sponsor 6 Line Driver 8 DigiKey $25.00 Project Lab Project Lab/Sponsor Ongoing Commitments The team did not get sufficient testing done for the device in the intended conditions. Because of this, the team is committed to finishing a round of tests that they will use to optimize the radio parameters and identify any structural deficiencies. The following list shows the list of tests and additions that will be conducted after the completion of this report: 21 1. Command response delay tests in aquatic conditions 2. Command response delay tests as a function of battery charge 3. Signal range tests in aquatic conditions 4. Signal range tests as a function of battery charge 5. Battery life tests 6. Multiple device interference tests 7. Software fixes based on any detected bugs The tests and analysis will be compiled into an Addendum report and submitted to the Project Lab and Project Sponsor on Monday, January 17th, 2011. 22 Recommendations While the project reached most of the goals set out in the project charter at the beginning of the term, a few loose ends should be tied up to improve the quality of the resulting project. Because the project was a prototype, the casing of the project was never made ergonomic or wearable. To make the project viable for further applications both in the target area and other haptic research, the device should be housed in a smaller container. This is needed because the Otterbox is simply too large to be handled effectively, and most of the container space is unused because of the relatively small size of the circuitry. The team had a few approaches planned for creating a better box, but did notdidn’t have the time to produce a sample for testing and analysis. The team was not able to create a live streaming audio solution with the chip that was selected, due to the complexity of the problem and the lack of experience the team had with audio electronics. This was part of the project charter, and the team recommends finishing this portion of the solution. The team was able to implement a solution that stores audio cues that can be triggered from the server. While a serviceable solution, this does notdoesn’t allow the server to be sure that the right audio cue was played (at least not as sure as they would be if they had just spoken the audio cue). The team recommends looking for alternative wireless communication solutions if a streaming option is to be made viable. The reason for this is that the data rate of an XBee Pro (10kbps) is simply too small to sustain a working and reliable audio link. The team suggests switching the radio profiles to long range, upper scale Class 1 Bluetooth devices, such as the LM5403, which has sufficient range and data rate. These devices have to be installed on both sides of the link (for both the server and the user) to enable two way communication, which makes the cost of the system pricier than originally anticipated. Other Bluetooth solutions lack the tested indoor range to make them a viable solution for this project. For the streaming audio chip, the team recommends a chip comparable to the VS1053/1055 series, although a chip with fewer abilities may also work. The important aspects are the ability to process encoded audio file formats like .mp3 and .oggopp, which can help reduce the requirement for bandwidth and ease the load on the RF system. To continue expanding on the electrical design of the SCID, additional tactile motors can be added to the system, but this will require an additional Arduino board. Testing should be done to determine the optimal positioning of the tactile motors, as well as the required audio cues and whether streaming audio will be required. Finally, the device can be further miniaturized by designing a smaller waterproof container and compressing the PCB if necessary. 3 Available at http://www.newark.com/lm-technologies/lm540/adapter-bluetooth-usb-class-1/dp/13T9470, with the main product website available at http://www.lm-technologies.com/business/product_item.php?item_id=20 23 The most important issue will be to test the user response to the tactile motors, to find the optimal placement and types of cues necessary to give efficient directions to a visually impaired swimmer. This can be done with dry testing or testing in a swimming pool. For dry testing, the device would be used to direct a blindfolded person through a room with various obstacles. An operator would then observe the test subject and issue commands to him through a laptop which will send the required cues to the device wirelessly. From this test, the optimal cues can be found to direct a person in a particular direction, and an average reaction time can be found for reference. Testing in the swimming pool would use a similar setup; however it will test the effect of distractions such as the splashing of water and the movement required to swim. This data can then be compared to the data from the dry test to determine whether these factors will affect a person’s ability to detect the tactile motors. If it is determined that additional tactile motors will be required, or if streaming audio will be needed, then another Arduino board will need to be added. The recommended board would be another Arduino Pro Mini to keep the device size minimised. The additional Arduino will likely require an additional Xbee to transmit and receive commands, however it may be possible to route both Arduinos to the same Xbee and use a modulating and demodulating system. For this project it was not necessary to decrease the PCB size because of the enclosure that we chose; however, the Otterbox is not favourable in terms of wearability and was only chosen because its space allowed for expansion if needed for further research. If a smaller waterproof enclosure could be found or designed, the device could be made much more space efficient and comfortable to wear. During the PCB revision process, we found that it is possible to compress the size of the PCB to a board that is not much wider than the Arduino Pro Mini, which is the largest component board. The second revision of the PCB had dimensions of 6cm x 3.4cm, but it was sometimes difficult to solder components to the board. Nonetheless, it is likely that the PCB can be miniaturised without raising the difficulty of assembly too much, especially for one more experienced with soldering.