Getting Data From The Web – Arduino + Ethernet

Friday, June 10 th , 2011

Yesterday we covered how you would go about controlling pins of your arduino over the internet using the Arduino Ethernet Shield set up as a server. Today we are going to take a look at using the shield as a client to get information off of a web page, and report back. I used this method a few months ago when I made the Nixie Twitter follower counter dubbed Twixie.

The ethernet shield can be used to access any non-password protected site with ease, but getting the information back to you is the hard part. For Twixie I created a special php page that queried the twitter API and displayed only the twitter count. This made it so I didn’t have to tell the arduino what to look for, or scour endless lines of HTML looking for a single number. For our example im going simplier. I created a PHP file that just outputs a random alphanumeric string. I did this because getting everyone setup with an API account somewhere is beyond the scope of this article, and unneeded to prove the concept and get you started. But, the idea is that you could easily take that PHP file (any web accessible file) and taylor it to display whatever you need.

In the client mode, the ethernet shield is able to access a webpage and return what is read. But the reading is done one byte at a time, and it reads the entire thing. So it can be like a needle in a haystack on large pages. Even if the page we are reading contains only the information we need, there is extra information at the beginning that is sent to the arduino. You never see it, but a web server actually sends extra information know as a “header” that tells the browser various information about that page (this is different than HTML’s tag).

Because of this, we need a way to tell the arduino what is junk, and what is the good stuff. To do this we are going to surround the information in < >. When the Arduino starts to read the page we tell it to ignore everything until it sees “<". From this point on we tell the arduino to record each following character until it sees our ending character ">“. At this point the arduino has everything it needs, and disconnects from the server then reports back with the data it found.

The Ethernet Shield library does not come with DNS support out of the box meaning we can not simply access the website we need by its simple URL (like http://arduino.cc). No, sadly, we need to have access the site through an IP address. For instance, bildr’s IP address is 174.123.231.247 and you can actually access bildr like so http://174.123.231.247/~bildr/ – Not every web-server allows you to do this, but when you can it is typically IPADDRESS/~ACCOUNT_USERNAME – So you can see the PHP file I created for you here http://174.123.231.247/~bildr/examples/ethernet/

Code

Without an extra library, the ethernet code does not support DHCP and therefore requires that we hardcode the IP Address, gateway address, and subnet mask for your network. This isn’t really that hard, it is more of a pain when especially if you want to plug it into a different network as the same settings may not work.

If you are familiar with your network and how to do this, awesome, just make sure you change the setting at the top of the code to fit your network. If you are not familiar with it, we can help you in the forum, and have posted some general help in there as well.

As of Arduino 1.0, DHCP is supported, so this should be able to just plug into most networks and start working.

NOW 1.0 COMPATIBLE

//ARDUINO 1.0+ ONLY
//ARDUINO 1.0+ ONLY
#include <Ethernet.h>
#include <SPI.h>

////////////////////////////////////////////////////////////////////////
//CONFIGURE
////////////////////////////////////////////////////////////////////////
byte server[] = { 174,123,231,247 }; //ip Address of the server you will connect to

//The location to go to on the server
//make sure to keep HTTP/1.0 at the end, this is telling it what type of file it is
String location = "/~bildr/examples/ethernet/ HTTP/1.0";


// if need to change the MAC address (Very Rare)
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
////////////////////////////////////////////////////////////////////////

EthernetClient client;

char inString[32]; // string for incoming serial data
int stringPos = 0; // string index counter
boolean startRead = false; // is reading?

void setup(){
  Ethernet.begin(mac);
  Serial.begin(9600);
}

void loop(){
  String pageValue = connectAndRead(); //connect to the server and read the output

  Serial.println(pageValue); //print out the findings.

  delay(5000); //wait 5 seconds before connecting again
}

String connectAndRead(){
  //connect to the server

  Serial.println("connecting...");

  //port 80 is typical of a www page
  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.print("GET ");
    client.println(location);
    client.println();

    //Connected - Read the page
    return readPage(); //go and read the output

  }else{
    return "connection failed";
  }

}

String readPage(){
  //read the page, and capture & return everything between '<' and '>'

  stringPos = 0;
  memset( &inString, 0, 32 ); //clear inString memory

  while(true){

    if (client.available()) {
      char c = client.read();

      if (c == '<' ) { //'<' is our begining character
        startRead = true; //Ready to start reading the part 
      }else if(startRead){

        if(c != '>'){ //'>' is our ending character
          inString[stringPos] = c;
          stringPos ++;
        }else{
          //got what we need here! We can disconnect now
          startRead = false;
          client.stop();
          client.flush();
          Serial.println("disconnecting.");
          return inString;

        }

      }
    }

  }

}
Unless otherwise stated, this code is released under the MIT License – Please use, change and share it.

The example php file. This just creates a random alpha-numberic string like <1Hc2f>

<?php
//the arduino will store anything between '<' and '>'
//So if the output was <1kjhghk5> - the  arduino would read 1kjhghk5


//Just generates a random alphanumeric string
$what_the_arduino_reads = '1'.base_convert(rand(10000,9999999), 10, 36);

echo '<'.$what_the_arduino_reads.'>';
?>
Unless otherwise stated, this code is released under the MIT License – Please use, change and share it.

Pin Control Over the Internet – Arduino + Ethernet

Thursday, June 9 th , 2011

In a previous article we showed you how to control digital pins over over serial, and showed how such a simple thing can be so powerful. One major downfall with that is you need to be nearby to send commands… So today we are going to look at doing the same thing, but this time we will be doing it over the internet using the Arduino Ethernet Shield.

As per our usual style, I am going to make this as simple as possible so it is easier to extend. So if you you are looking for something with an HTML interface, this article will not be covering that.

The Arduino Ethernet Shield is capable of being both a client (like a web browser), and a server, and with the onboard SD card-slot can be quite powerful by hosting up full websites, but for this article we are just looking at using the Arduino as a server and you will control it simply by going to a specific URL.

Note that the Ethernet Shield uses digital pins, 10, 11, 12, and 13 for itself, so it is best to leave these alone and not try to use them for anything else.

Getting It On The Net

Getting the ethernet shield on the internet is going to differ depending on your network, but no matter what, you need to plug it into an ethernet port connected to the internet – So make sure you do that.

Without an extra library, the ethernet code does not support DHCP and therefore requires that we hardcode the IP Address,gateway address, and subnet mask for your network. This isn’t really that hard, it is more of a pain when especially if you want to plug it into a different network as the same settings may not work.

As of Arduino 1.0, the library does support DHCP So you should be able to just plug it into your network and have it work. It will report the ipAddress to the serial monitor

We also left the code in there to do a manual setup if you prefer doing it that way. (I know I do) If you are familiar with your network and how to do this, awesome, just make sure you change the setting at the top of the code to fit your network, and un comment out the parts related to it. If you are not familiar with it, we can help you in the forum, and have posted some general help in there as well.

Hooking It Up

Aside from plugging the Ethernet Shield into you Arduino, there isn’t really anything you HAVE to have. I have LEDs on pins 2-9 on this example so we can see it being controlled. If you do not intend to blink LEDs or test the code with LEDs, you can by all means leave these off.

Code

Assuming the the ip address of the arduino is 192.168.1.167 (check the serial monitor to see what it is), this code will allow you to send a sequence to the arduino via your web browser like so: http://192.168.1.167/?23456789

The way the code is currently setup is that the pins will go high one at a time, in sequence, for 25ms then move on to the next in line. So this example would blink pin 2,3,4,5,6,7,8 then 9.

As it stands, the user will not see the webpage return until the arduino is finished processing the request. You could change the code to return right away and then process the request afterwards if needed, but I liked seeing the return after it was completed – like a receipt, instead of an order.

NOW 1.0 COMPATIBLE

//ARDUINO 1.0+ ONLY
//ARDUINO 1.0+ ONLY


#include <Ethernet.h>
#include <SPI.h>
boolean reading = false;

////////////////////////////////////////////////////////////////////////
//CONFIGURE
////////////////////////////////////////////////////////////////////////
  //byte ip[] = { 192, 168, 0, 199 };   //Manual setup only
  //byte gateway[] = { 192, 168, 0, 1 }; //Manual setup only
  //byte subnet[] = { 255, 255, 255, 0 }; //Manual setup only

  // if need to change the MAC address (Very Rare)
  byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

  EthernetServer server = EthernetServer(80); //port 80
////////////////////////////////////////////////////////////////////////

void setup(){
  Serial.begin(9600);

  //Pins 10,11,12 & 13 are used by the ethernet shield

  pinMode(2, OUTPUT);
  pinMode(3, OUTPUT);
  pinMode(4, OUTPUT);
  pinMode(5, OUTPUT);
  pinMode(6, OUTPUT);
  pinMode(7, OUTPUT);
  pinMode(8, OUTPUT);
  pinMode(9, OUTPUT);

  Ethernet.begin(mac);
  //Ethernet.begin(mac, ip, gateway, subnet); //for manual setup

  server.begin();
  Serial.println(Ethernet.localIP());

}

void loop(){

  // listen for incoming clients, and process qequest.
  checkForClient();

}

void checkForClient(){

  EthernetClient client = server.available();

  if (client) {

    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    boolean sentHeader = false;

    while (client.connected()) {
      if (client.available()) {

        if(!sentHeader){
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          sentHeader = true;
        }

        char c = client.read();

        if(reading && c == ' ') reading = false;
        if(c == '?') reading = true; //found the ?, begin reading the info

        if(reading){
          Serial.print(c);

           switch (c) {
            case '2':
              //add code here to trigger on 2
              triggerPin(2, client);
              break;
            case '3':
            //add code here to trigger on 3
              triggerPin(3, client);
              break;
            case '4':
            //add code here to trigger on 4
              triggerPin(4, client);
              break;
            case '5':
            //add code here to trigger on 5
              triggerPin(5, client);
              break;
            case '6':
            //add code here to trigger on 6
              triggerPin(6, client);
              break;
            case '7':
            //add code here to trigger on 7
              triggerPin(7, client);
              break;
            case '8':
            //add code here to trigger on 8
              triggerPin(8, client);
              break;
            case '9':
            //add code here to trigger on 9
              triggerPin(9, client);
              break;
          }

        }

        if (c == '\n' && currentLineIsBlank)  break;

        if (c == '\n') {
          currentLineIsBlank = true;
        }else if (c != '\r') {
          currentLineIsBlank = false;
        }

      }
    }

    delay(1); // give the web browser time to receive the data
    client.stop(); // close the connection:

  } 

}

void triggerPin(int pin, EthernetClient client){
//blink a pin - Client needed just for HTML output purposes.  
  client.print("Turning on pin ");
  client.println(pin);
  client.print("<br>");

  digitalWrite(pin, HIGH);
  delay(25);
  digitalWrite(pin, LOW);
  delay(25);
}
Unless otherwise stated, this code is released under the MIT License – Please use, change and share it.

Video

Extending This

By adding to, or changing, the switch statement in the code, you could easily add support for letters as well allowing you to call even more functions over the web. You can also change the code that is run if needed. So instead of blinking pins, maybe you control motor directions or timing. This could be easily be used for automation or art installations.

If I can take the non ethernet version of this article and make web controllable bells, just imagine what you can do with this.