Final description of our project

As the result of the course we developed a remote control system of the Lego MINDSTORM NXT robots from a mobile phone. We used Spike standard model of NXT as a robot to be controlled and Nokia 6233 as a remote controlling device, but we tried to write universal application which can be used with minor changes to control any NXT robot using any mobile devices with MIDP2.0 and JSR82 (Bluetooth API) support. We used LeJOS as an operating system for our NXT device which allowed writing both parts of the system using Java language.
The main features of our system are:
An ability to control any robot motors.
An ability to subscribe a phone to receive data from any robot’s sensors in real-time mode.
One can make robot to play some sound while executing any command.
Simple Bluetooth protocol. Almost all commands are described by one or two bytes.
Easy expandable for any other NXT robots.
The portability was our main goal. As I already told in my blog, we have very simple set of commands. For instance, all movements of the robot are performed by the following command pair “rotate/stop motor”, “motor index”. All command constants are stored in NXTCommand class both at mobile phone and in the robot. By using this pair one can increase forward motor speed (motor will start rotating forward if it is stopped or will slow down if rotates backward); do vice versa (increase rotating backward); stop rotation of a motor; rotate any motor to any desired angle forward or backward. Mobile phone application has a set of such small commands mapped to each number key. So, for instance when controlling Spike robot, if you press key 2 (it means forward) phone will send two commands: increase forward speed of motor B and increase forward speed of motor C. If you press pound button (it is a tail kick) phone will send 3 commands: rotate motor A to a certain angle forward, wait a little bit and rotate motor A to a certain angle backward.
Thus, if you want the application to control other type of NXT robot all that you need to change mappings of commands to the buttons in mobile phone application. Also adding a new type of command is not very challenging. You just need to keep in mind those limitations of LeJOS which I described earlier.

The second big feature except control of motors is an ability to receive data from sensors to a mobile phone. This is done in three steps (see our final presentation at Evgenia's diary). First of all when a user enters Sensors menu, phone sends request to a robot asking what sensor types it has. Robot sends back 4 sensor types in the order in which those sensors are connected to the sensor ports (Sensor from port 1 goes first, then goes sensor from port 2 and so on). Mobile application shows a menu to a user where he can choose which sensors he wants to receive data from. After user presses OK button, phone sends a request to subscribe for a certain sensor’s data. Robot handles this request by starting a new thread which monitors sensor values and sends them periodically to the phone.

Unfortunately our robot can not detect by itself which sensors it has (it is probably possible but we had no enough time to implement it), so sensor types should be manually changed in robot’s application code when porting it to another robot type. Also we have implemented listeners only for 3 types of sensors: ultrasonic, touch and sound sensors, because it is all that Spike has. But new listeners can be easily implemented by analogue with other ones just in few minutes.

About disadvantages of the system: Bluetooth channel is very unstable and sometimes some data is lost during transmission. This problem begins very relevant, when robot is moving and spends a lot of power for the motor rotation. So when you want to subscribe to some sensor, stop the robot and do not keep you mobile device far from your robot. Sometimes even some critical exception can be thrown because of broken connection or lost data. There is no graceful handling of broken Bluetooth connection neither in robot application nor in mobile phone. Also, robots application has no exit point. It is done so, because LeJOS doesn’t support correct exit from any program and it is just restarts in order to finish a running application.

The thing that could be added to this system is a separate thread for some own robot’s logic execution. For instance, this thread could contain code which would prevent robot run into an obstacle even if a user intentionally gives to robot such commands. Or robot could make some action by itself if user doesn’t send any commands for a certain period of time (for instance play some yawn sound).

Source code and binaries you can take from here.

Final seminar

This Friday was the last seminar and actually it was the most interesting one. At this seminar every group showed what they had done. We had a contest between 4 rescue robots, a robo-soccer game and also a robot which finds a route in a maze. Of course we also showed our Spike, which willingly carried out all my orders, sent from the phone. It was the most exciting part of the course – to see robots working. But think that I have noticed is that every team had some failures in construction or logic of their robots. All projects were a little bit unfinished, just few more testing was required to make solid robot. I think that all of teams left some work to the last studying week, where there were no enough time because of other courses. That is why I think, that there should be one session before final, where every team should show their working “beta-releases” and after that, if any bugs are found fix them before final seminar. That is especially useful for robots which take part in contests (rescue and soccer). Anyway, two of four rescue robots got very high score even without this improving step, but I think that rescue stand could be made more difficult for student’s contests (more obstacles, bigger gaps in line, etc). Also it is good idea to share your progress during the course not by showing presentations, but rather by bringing your robot and showing everybody its construction features and what he is already able to do. I think this will reveal main advantage of the course – the ability to see and touch what you are developing.

After looking at LeJOS source codes we found that when you are trying to pair with NXT device controlled by LeJOS it always sends "1,2,3,4" as a code. So if you pair a mobile phone with NXT using standard phone menu and entering 1234 as a code, you will then freely connect after that to robot.

My collegues started to implement mobile phone application and a program for robot. We decided that robot will be able to do very simple unary commands, such as "start motor rotating forward, use motor A" So each command will have two parts: 1) what to do with motor and 2) which motor to use. So if you want to make robot move forward, you should send in our case to commands: "start rotating forward motor B" and "start rotating forward motor C". This approach will allow using the same LeJos program to controll different NXT robots and all that you will have to do is to change command combinations which are sent by mobile phone. Also we are going to sent data from sensors back to phone, so that you can see values at sensors while you are controlling your robot.

5th seminar 30.11

Last week we have started writing a Java application for mobile phone, which will communicate with NXT device. I tried to write a simple library for bluetooth utilization on mobile phones, so that my colleagues could continue with developing a remote control application. My goal was to make my Nokia phone connect to the NXT and send some data to it. I've used BTReceive program from LeJos examples for our NXT device. This application simply waits for connection and then prints all received signals to the NXT's display.

I have made a simple interface for device searching, investigated, what bluetooth services NXT has and tried to establish connection with it. So my progress so far is the following: I can find NXT devices in range and establish connection with them. But still my code doesn't works well, because sometimes phone complains that he wants to pair with NXT device first befor connecting. I still did not find how to do this. The second problem is that BTReceive application installed at our NXT device doesn't show correctly numbers that I'm sending to it. Further investigation of Bluetooth settings is required.

At the seminar we told about all our progress and problems. Actually we already did a lot, because setup of Bluetooth is the unpredictable part of our work: if could not manage it then we would not be able to do anything.

Also we told about limitations which we discovered in LeJOS after deeper digging into it. For instance, there is no "swicth" operator at all, there can be only 64 fields in a class, there are problems with big numbers (arithmetic operations with long data type are not supported) and the 4th sensor port doesn't work with some types of sensors. The main limitation of LeJOS which should be always kept in mind is there is no garbage collector in it. So you should not create any object instances repeatedly (in loops or in listener methods), otherwise you will get an out of memory. But still LeJOS seems to be powerfull tool because of threads and synchronization support, which makes it usual for Java-programmers.

It is a last seminar before the final one so we have a lot of work to do during holidays...

4th seminar 23.11

At this seminar we shared our achievements in robot development. Most of groups were building their robots and they showed us their robot design. As we have almost nothing to build, we described what we are going to do (see previous post) and our first experience in NXT progrsmming for the LeJOS operating system. We wrote a very symple program just to test how motors and sensors are working. Our robot walks forward until it sees any obsticles and if so, it stops and kicks the obsticle by his tail. Also he makes a kick when you touch his sensor at tail. Well, as our project is programming-oriented, we almost first whi started programming and I did not hear from any group that they have used LeJOS.

Last Friday we had no class, but we’ve stared working on our project, so I still have what to write about. The topic of our project is “Robots programming using mobile devices”. We do not need to construct something by ourselves so we took already built Lego Mindstorm NXT robot, called Spike. We found later that there are several mistakes in its structure, so we had to remake it a little. Anyway, fixing wrong structure of robot without any additional parts is not as interesting as creation of robot from scratch. But we really have no enough time for playing with it. I would rather prefer to get a box with such Lego to play with at Christmas holidays :)

Let’s go back to our project work. First time we decided to make a programming environment for a pocket PC, which allow writing code in some language and then send it to the NXT device via Bluetooth. We have some guy at our department who developed some Java IDE for pocket PC devices. Also, there is a Java Virtual Machine for Lego NXT devices which allows programming Lego robots in Java. So we had an idea to extend Java IDE for pocket PC so it could be used for NXT programming. We assumed that the main difficulty will be to send compiled code to the NXT brick via Bluetooth. The idea was great, but after taking a look on how LeJOS programs are compiled and sent in Windows platform we found that the proper compilation of a program is even more challenging than Bluetooth communication. The problem is that there is a couple of third-party software which is used to create binary file in addition to the Java compiler. Porting this process to the pocket PC platform is not trivial task.

Thus, the first idea failed. It is good, but again we have no time to implement such project. So we decided to reduce our project from remote programming to the remote control of NXT robot. We are going to create a flexible mobile application which can be used to control NXT robot with LeJOS installed. The application should allow binding any action on mobile phone keypad. We found that there are still no such solutions available for the LeJOS platform (I believe that there are some but authors do not distribute them).

The work on our project is just started. We installed and configured all required software for programming in Java ME; Installed LeJOS to our robot; Wrote several simple programs for the robot to get some idea how to program NXT brick. Actually, I have some experience in working with Bluetooth on mobile phones, so I hope we should not get any big problems with it.

3rd lecture 9.11

It was not a lecture but rather a workshop where we showed our presentations about what we had done during last week. It was a little bit frustrating because we've got the letter saying that we have to make a presentations only one day before the class and before this letter we had not known that we should make any progress in area, that we have chosen. We even did not choose the concrete topic we are going to do. So we made a small presentation where we wrote our ideas of what can be done in 'remote robots programming using mobile devices' and what are approaches that already available in the Internet. Luckily, lecturer gave us few ideas and now we are almost ready to state the topic of our project.