Any way to get data from a webpage?

I want the imp to pull sunrise/sunset times from some webpage. The only interfaces I’ve seen is to post data to it, and to have data posted from it. Any way to have it just download content from a webpage?


Ok, I figured that I need a combinatin of an output and HttpIn. I set them up with the goal of getting the text back from a JSON page containing some astronomy data from weather underground. The data looks like this :

{ "response": { "version": "0.1", "termsofService": "", "features": { "astronomy": 1 } }, "moon_phase": { "percentIlluminated": "81", "ageOfMoon": "10", "current_time": { "hour": "9", "minute": "56" }, "sunrise": { "hour": "7", "minute": "01" }, "sunset": { "hour": "16", "minute": "56" } } }

I try to look at this with this
class Astronomy extends InputPort { function set(payload) { server.log("Astronomy Data Received"); server.log(payload); // server.log(payload.sunrise.hour); } }

but I just see this, and I’m not sure how to access the data within. I have it configured to retrieve the data as a string. It shows up as a table, but is there some way to dump the contents of the table?

Sunday, November 18, 2012 21:31:56: (table : 0x0x2001c0a4)

You can iterate over a table for foreach as I remember, eg:

foreach(key,value in table) server.log(key + " = "+ value);

…though I guess if it’s hierarchical you may need to have something more like:

function dumptable(n, t) {
server.log("dump table "+n);
foreach(key, value in t) {
if (typeof value == “table”) dumptable(key, value);
else server.log(key + " = " + value);

dumptable(“root”, table);

…not that I’ve tried this, but…

Note that, depending how much accuracy you need, you can calculate sunrise/sunset times on the imp itself, given the date, latitude, and longitude: (the link “Low Accuracy Equations”).

Better (more accurate) ephemerides are also available: using “View Source” on this page: shows a more involved calculation, easily translatable from Javascript into Squirrel.


Thanks, dumptable worked. I want to pull in other weather related data so using their api is simpler. thanks.


@thereza - any chance I could ask you a couple of questions about how you went about this? Trying to do the same thing.

Here’s how to do it:

function pollAstronomy() {

class Astronomy extends InputPort {
function set(payload) {
server.log(“Astronomy Data Received”);
local hour = payload.moon_phase.sunset.hour.tointeger();
local minute = payload.moon_phase.sunset.minute.tointeger();

    local d = date(time(), 'l');
    if (d.hour >= 8) {
        d.hour = d.hour - 8;
    } else {
        d.hour = 24 - d.hour - 8;
    local currentTime = d.hour * 3600 + d.min * 60;
    //schedule next reading for following day
    imp.wakeup( (26*3600) - currentTime, pollAstronomy);//query at 2am


imp.configure(“Weather Underground Example”, [ Astronomy(“astronomy”,“string”) ], [astronom] );

Then setup an HTTP request in the planner with the weather underground api, use a get mode, and set it to json. Connect the imp to the http request and the http request to the planner (so you have an two arrows, one going each way). When you call pollAstronomy, it will issue a request. When you get a reply, it will call Astronomy.set() and then you can parse the json fairly easily (see example). I then schedule another poll for 2am the following day. Unfortunately, the imp doesn’t handle timezones so I just subtract 8 hours from the time to get my local time but it will not compensate for daylight savings which is a pain.

Hope this helps.

Wow - thank you so much reza. That is perfect! Had been barking up the wrong tree with php.

Really appreciate the help.

I was wondering how @thereza did this you don’t need the looping through to convert JSON to table

//Squirrel 3.0 is possible to declare a table using JSON syntax.
The following JSON snippet:

local x = {
“id”: 1,
“name”: “Foo”,
“price”: 123,
“tags”: [“Bar”,“Eek”]

Is the same as is equivalent to the following squirrel code:

local x = {
id = 1,
name = “Foo”,
price = 123,
tags = [“Bar”,“Eek”]

cloudcontrol - did you have a question?

Sorry no, just quoting in you.

I have a new question related to this earlier thread.

I’m trying to bring back JSON data (small array) from a .php file I have. I encode/echo the array upon linking to the file with an ‘echo json_encode($myArray)’ statement. This produces what appears to be a well formed JSON string in response. The imp HTTP Request is configured to GET the appropriate file URL, and the Content Type is set to ‘application/json’ (as is the file header, if this might also be necessary). Given this, I’m not having any success accessing the data with the imp.

However, modifying the .php file with the following code to first store the results in a .json file extension …

$fp  =  fopen('myResults.json', 'w');
fwrite($fp, json_encode($myArray));

… now, I am able to successfully retrieve the data if I repoint the HTTP Request URL at this myResults.json file. (Note: linking to both the generating .php, or stored .json files appears to produce identical JSON strings).

Question: Why am I not able to process the string generated directly by the .php file, yet able to process the same string stored in a file with the .json extension? (I prefer to just read the data when I call the .php file rather than going through the extra step of storing it in a .json file. Is there a way to do this? It seems like I might be continuing to overlook something simple).


Ok, let me add a bit more here after some more looking … I overlooked a very obvious difference.

Though the .json and .php files appear to produce an identical JSON string output, they indeed do not. The .php string is of course wrapped in all the (non-displayed) page HTML, whereas the .json file output is simply the string. So, I’m thinking this is very likely the cause of my problem.

That said, might there be a way to handle the JSON part of the output that is generated by the .php file … or am I just trying to push a bad position here?


@thereza What do you mean by : ‘… and the http request to the planner’