Jan 14 2010

Using OAuth to Access Twitter from KRL

Kynetx Logo

The latest build (Build 391) of the Kynetx Rule Language (KRL) includes support for accessing Twitter data intrinsically within the language. Integrating interesting data with KRL is an important part of what makes the language so useful for building cross-site applications that mash-up data and user interactions. But what's really interesting about this release is that we're using OAuth to access the Twitter API and have built primitives into the language for dealing with the Twitter OAuth interaction to save developers from doing it.

Not only are we making it easy for developers to write apps that use Twitter, but this is also the first time that end users will see significantly different behavior from an app than their friend might. An app that uses the Twitter library will use my Twitter data for me and your Twitter data for you. Using OAuth, Kynetx apps can now be personalized.

Using Twitter data inside a KRL app generally involves two KRL patterns: authorize then use and initialize then populate

In the authorize then use pattern, a rule is put in place to check if the app is authorized to take a certain action and, if not, do what is necessary to complete the authorization. What makes this work is using the rule postlude to ensure that the rest of the rules (which presumably rely on the authorization) don't run. Here's an example from an app I wrote to demonstrate using the the new Twitter library:

rule auth is active {
  select using ".*" setting ()
  if(not twitter:authorized()) then
      twitter:authorize()
        with opacity=1.0 and
             sticky = true

  fired {
     last
  }
}

Notice that this rule only fires if the predicate twitter:authorized() is false. The action, twitter:authorize(), is what initiates the OAuth ceremony. The action will pop up a notification in the user's browser that looks like this:

The postlude of the rule (fired {...}) runs the last statement if the rule fires to ensure that nothing else happens. Of course, if the app is authorized, the rule doesn't fire, the OAuth ceremony is not initiated, the last is never executed, and the remaining rules in the ruleset are evaluated.

The initialize then populate pattern is important any time you're working with complex data. With complex data, you will frequently need to do something for each component of an array. That's what the foreach statement does as part of the rule selector: executes a rule once for each member of an array.

The problem is that if we use a foreach to loop over the tweets we get back and use notify to place them on the page, we'll end up with one notification box for each tweet. Not very pretty.

A better solution is to use a rule to place the notification box (the initializer) and another rule to loop over the tweets and place them in the notification box (the populater).

Here's the initialization rule:

rule init_tweetdom is active {
  select using ".*" setting ()

  pre {
    init_div = <<
<div id="tweet_list">
</div>
>>
  }

  notify("Friends Tweets", init_div)
    with sticky=true and
         opacity = 1.0

This is a pretty simple rule that places an empty notification box on the page.

The real work is done by the populating rule:

rule populate_tweetdom is active {
  select using ".*" setting ()
    foreach tweets setting (tweet)
      pre {
        text = tweet.pick("$..text");
        div = "<div style='background-color:#666;margin:2px'>" +
              text +
              "</div>";
      }
      append("#tweet_list", div)
}

This rule loops over the tweets, grabs data out of them using pick, and appends the result to the div in the notification box.

The tweets variable was set in the global block:

tweets =
  twitter:authorized() => twitter:friends_timeline({"count": 8})
                        | [];

After you've gone through the OAuth ceremony at Twitter, wherever you run this app, you will see a box like this that contains the last eight tweets from your friends timeline on Twitter. Here's the results for me:

Of course, if you run this app, either by card or bookmarket, you'll see the results from your friends timeline.

Craig Burton has created a nice tutorial about how all this works. There's some important data in the tutorial about how to get the keys from Twitter for your application. The library is also well documented and the source code for the ruleset is available.

The ability to personalize apps by appealing to personal data elsewhere on the Web is a huge step forward for KRL. Look for other APIs to be embedded in KRL in the near term and, eventually, more general support for OAuth so that developers can use any OAuth protected data source.

Jan 14 2010

UtahPolitics.org All Over the Web

Since 2003, I've been running a site called UtahPolitics.org. The site started off as a blog on which I and others posted articles. When I started UtahPolitics.org there was some speculation about my motives. But my motives are simple: create a place I can experiment with new media in an arena that interests me.

Last year, in an effort to continue the experiment, I put up a retweeter for UtahPolitics.org that would retweet any tweet from friends of the @utahpolitics account that contained the tag #utpol. When I did that I also morphed the web site--imperfectly--to a site that just aggregated those tweets. The idea was to move the discussion off of UtahPolitics.org and onto Twitter.

The problem with this is that it's sometimes difficult to follow the thread of a conversation, particularly those about particular pages on the Web. To try to solve that problem, I've build a Kynetx app that will show all the #utpol tweets about page on that page when you visit it. Here's a picture to show you what I mean:

The box on the right is showing any tweets that reference the page its on and contain the tag #utpol. Right now it's tweets by anyone. If that gets to be a problem, I'll restrict it to friends of @utahpolitics so that I have a way to block people.

I'm hoping that you'll use this app to follow Utah Politics during the next legislative session and contribute to the Twitter discussion about specific bills or newspaper articles. Using this app and your Twitter account, you can comment on legislative pages, newspaper pages, and so on. Anyone else who has the app will see your tweets and be able to respond on the page.

You can use this app in one of three ways:

  1. Download the AzigoLite Card selector and install the Utah Politics card - this is the most flexible solution since once you have the card selector installed, you can also use other Kynetx apps without any further downloads. Just install the card and go.
  2. Install a browser extension - this is probably the fastest and most convinient way to get going. Just download the right extension for the browser you want to use (or several if you regularly use more than one browser) and you're good to go. The following extensions are available:
  3. Use a bookmarklet - this is the simplest, but has the drawback of working only when you click on it. You may not always remember to do so and will miss out on people's comments. To use the bookmarket, drag this link to your bookmark bar at the top of your browser

    UtahPolitics Bookmarklet

Once you've got something installed, head on over to House Bill 10 on the Utah Legislature's page to see it in action. I've left some test tweets there.

You don't need to do anything special to use it. Just tweet about a page and include the tag #utpol.
All the tweets about a particular page will show up in the app--even from people who don't have it installed. This is an assymetrical communication tool. Note, if there are no tweets about a particular page, you won't see anything--the app doesn't do anything in that case.

If you're interested in how this was made, I've made the source available on the app page. Feel free to ask questions or just copy it and build your own--Kynetx developer accounts are free.

I'm excited to see how this works--I hope it leads to some interesting discussions about Utah politics and gives some insight about how apps like this one--that work on multiple Web sites--can be made more effective.

Jan 06 2010

Syntax Change to Action Labels

We will release a syntax change to action labels to accommodate a reuse of the : as a namespace separator for actions. The old action label syntax looked like this:

  foo: notify("Hello", "A Note to say hello")

The new syntax uses => like so:

  foo => notify("Hello", "A Note to say hello")

We don't believe that action labels have found widespread use based on a review of the log files and consequently will release this syntax change tomorrow (Jan 7).

Dec 30 2009

Build 384: Annotating Search Results with Large Datasets

Kynetx Logo

Recently Azigo and the Better Business Bureau launched the BBB app that helps people locate BBB accredited business:

The BBB's Accredited Business Locator is powered by Azigo's free, downloadable browser plug-in and displays the "BBB Accredited Business" seal whenever consumers search for products or services using popular search engines such as Google, Yahoo! and Bing. This seal indicates that a business is reputable and offers one-click access to the BBB report for that business. This provides maximum consumer protection and minimizes the risk of online fraud.

This app is powered by Kynetx and the techniques necessary to make it work have been factored into the Kynetx Rule Language (KRL) so that they're available to everyone. The BBB app required two things that KRL didn't support well:

  1. The BBB data set of accredited businesses was very large running to millions of records
  2. The BBB was interested in annotating local search results

The problem with large data sets is that previous versions of KRL had an action called annotate_search_results that relied on preloaded data. The Kynetx Network Service (KNS) ensured that the data was available based on information from the KRL program. With millions of records we could hardly just load up the entire data set and there was no way to segment it since you don't know what the user's going to search on (and thus what results are relevant) until they do it. Enter remote

We've added a new option to annotate_search_results called remote (optional parameters to actions are specified using a with clause). The remote option specifies a URL that is the location of a remote data set. When it's used the selector function is ignored (and can be left out) of the annotate_search_results call. Instead the remote data URL is called after the search results are available and passed the relevant data from the results. The remote service uses the data about the search results to query the database and returns a JSONP response that will be evaluated to annotate the results.

Given that this runs after the search results page has loaded, you might think it would be slow, but our experience with BBB has convinced us that it works just fine.

We've also added a annotate_local_search_results action that works just like annotate_search_results, but is specifically tailored to annotate the local results in each search engine.

You can find more information about search annotation in KRL in the documentation. You can also download and try out the BBB app to see how it works for yourself. We've built a sample app so that you can see a sample app that uses these two new methods with remote data.

One other thing: build 384 included the let_it_snow() action because it's winter and snowy Web sites can be fun.

Dec 02 2009

Build 354: Control Statements in Postludes

Kynetx Logo

This afternoon we releases Build 354 of KNS supporting the addition of a last control statement to postludes. In addition, we also now allow guard conditionals on any statement in a postlude. These are relatively minor additions to KRL in anticipation of some larger features that are coming soon.

The use a last statement in a postlude will halt the execution of the ruleset at that rule if it is executed. So, the following statement would halt execution after the current rule if the rule fired:

fired {
  last
}

This can be useful for rules that initiate action that must be completed before any of the remaining rules in the ruleset are meaningful (like authorization...hint, hint). That saves you from having to guard all of the remaining rules with a premise checking that the action has been completed.

As an example of using a guard condition on a postlude statement, consider this example:

notfired {
  ent:page_count += 2 from 1 if(not_available eq "yes")
}

This would only be applicable if the rule didn't fire and in that case only increment the page_count entity variable if the value of the variable not_available was "yes".

Dec 01 2009

Announcing the Kynetx Developer Exchange

Kynetx Logo

Today we're releasing the Kynetx Developer Exchange. This is a forum, based on the StackExchange service that is the same code that runs StackOverflow, ServerFault, and SuperUser sites. The functionality is excellent and we're hoping that it provides a fruitful place for developers to interact with Kynetx programmers and each other.

Mike Grace, who is now working for Kynetx, has done a good job of seeding it with the questions he's had as he's gotten up to speed. I'll be participating and so will others on the Kynetx team. We hope you'll use it to answer questions about programming in KRL and making the KNS system suit your needs.

To get started, just go to the Dev Exchange homepage and log in using any OpenID, including Google, Yahoo, or AOL. The more you participate, the more cred you'll have.

Nov 24 2009

Build 353: Coercion and Random Numbers

Kynetx Logo

We released Build 353 of the Kynetx Rules Engine (KRE) this morning. Actually, I should point out that Wade Billings released it and I watched over his shoulder. This is the first time in the history of Kynetx that someone besides me released KRE code. Yeah!! This is an exciting thing in and of itself. Wade's job is to build a continuous integration environment for Kynetx so that code changes appear in production as quickly as possible. Until then, I've pushed the pain on him to ensure he's properly motivated.

The new release adds two new features to KRL:

  • The as operator provides a way to coerce primitive values. The most interesting ones are coercing strings and regular expressions. You can construct a string and then coerce the result to be a regular expression for use in a replace:
    my_str.replace(("/this/"+ e).as("regexp"),"do you want a")
    
  • You can use math:random to generate random numbers. The function takes an argument and generates a random number between 0 and the argument. For example, you could bind r to a random number between 0 and 9999999 with the following:
    pre {
      r = math:random(9999999)
    }
    notify("Randomness", "A new random number: " + r)
    

    Here's the result showing on my blog:

Both of these features were the result of developers asking for something they needed. We can't promise same day turn around, but for small features like this we try to be as responsive as we can. On Monday we'll announce the new Kynetx Developer Exchange where you can ask questions, answer questions, and make feature requests. More on that later.

Nov 09 2009

KNS Build 351: FLWR Comes to KRL

Kynetx Logo

One of the big features missing from KRL as a rule language is a foreach statement that allows looping. Build 351 of KNS (released today) fixes that problem. The thing that kept holding me back was confusion on my part about the best way to add it and how it should work.

The problem was that I wanted, thought KRL needed, more than just looping. I wanted full-blown FLWOR statements (foreach, let, where, order by, result). I realized one day on a bike ride that the entire rule ought to be a FLWOR statement and that meant that the foreach needed to happen before the rule body executed.

The rule prelude already functions in the capacity of a "let," the rule premise (condition on the action) already functions in the capacity of a "where," and the rule action itself is the "result." Yeah, I left out "order." More on that later.

KRL now allows one or more foreach clauses to be added to the select statement like so:

select using "/archives/" setting ()
  foreach [1, 2, 3] setting (x)

The value of x will be bound to the values 1, 2, and 3 on successive executions of the rule body. KNS optimizes the rule so that declarations in the prelude aren't executed inside the loop unless they depend on the value of the variable (directly or indirectly).

Of course, the array is an expression, so it doesn't have to be an array literal. You could do this:

select using "/archives/" setting ()
  foreach f.pick("$..store") setting (x)

This works fine as long as f has been declared outside the rule in the global and the pick returns an array.

The entire rule body--everything after the select is executed once for every loop. If the premise is true, an action is produced, so a rule with a foreach over a three element array would produce three actions if the premise were true each time.

We can't order the array yet. Adding a sort operator on arrays would do the trick but to do that I need someway to specify the comparison function and KRL doesn't have functions or closures...yet.

You can see more information about foreach in the KRL documentation on rule selection.

Looping has been a long time coming, but I'm excited to see what people do with it. Surprise me!

Sep 09 2009

Conditional Expressions and Explicit Logging

Kynetx Logo

This afternoon I released build 329 of Kynetx Network Services (KNS). This build includes two new features for the Kynetx Rule Language (KRL): explicit logging and conditional expressions.

Explicit logging allows developers to place information in the ruleset log when a ruleset runs. For example, the following example would place a string with the value of a variable named query in the log if the rule fired:

fired {
  log "query:"+query
}

Explicit logging is useful for recording information about the rule environment in the logs for later analysis.

Conditional expressions allow expressions to take on different values depending on the results of a predicate evaluation. You can use conditional expressions as part of any expression anywhere in the language. Here's a simple example:

pre {
  z = (x > y) => y | y + 3;
}

Conditional expressions nest nicely. Here's a longer example:

month = (m eq "01") => "January" |
        (m eq "02") => "February" |
        (m eq "03") => "March" |
        (m eq "04") => "April" |
        (m eq "05") => "May" |
        (m eq "06") => "June" |
        (m eq "07") => "July" |
        (m eq "08") => "August" |
        (m eq "09") => "September" |
        (m eq "10") => "October" |
        (m eq "11") => "November" |
                       "December"

In order to make the parser more efficient (and avoid hoops to avoid left recursion in the BNF) most predicate expressions require parentheses. The only exceptions are standalone intrinsic predicates:

pre {
  weather = cloudy() => "cloudy" | "sunny"
}

For as simple as they are, conditional expressions introduce significat expressive power to KRL. I think they'll come in handy in lots of situations.

Sep 02 2009

Persistent Variables in KRL: Threading Sites Together

Kynetx Logo

Yesterday I released build 325 of Kynetx Network Services (KNS) which includes a significant addition to the feature set of Kynetx Rule Language (KRL): persistent variables. Persistent variables allow KRL rulesets to store and react to data over multiple visits. This data isn't personally identifying information, but rather the kind of information that makes writing intelligent Web applications easier. Here's an example: an information box placed on a Web site can now have a "don't show this to me again" check box and act accordingly.

Persistent variables, or just persistents, will ultimately come in two flavors:

  • Entity variables store persistent data about an entity. Think "user, browser, device, etc."
  • Application variables store persistent data about an application, or ruleset. Think "class variable."

In this build we're only supporting entity variables. Application variables will follow later.

All persistent variables are one of three type: flags, counters, or trails. We may add others in the future.

  • Flags store boolean data and can be set or cleared.
  • Counters store integer data and can be initialized, incremented, and decremented.
  • Trails store places in a 20 place stack. You can think of places as URLs, although trails allow you to store anything. Trails can be marked (push a place on the stack) and places can be forgotten (removing them from the trail).

These mutations to the persistent data can occur as a result of rule firing (or not) in the rule's postlude or as the result of user action on the page in the rule's callback specification.

In addition to the mutations of persistents indicated in the preceding description, entity variables can be referenced in KRL expressions and tested in predicates. For example, the following action will fire when the persistent variable ent:archive_pages is greater than 3:

if ent:archive_pages > 3 then
   notify(...)

There are also some special predicates available for persistents.
Persistent flags and counters can also be tested with an associated timeframe. For example, the following action will fire when persisent variable ent:archive_pages is greater than 3 and the last time it as set was within the last 2 hours:

if ent:archive_pages > 3 within 2 hours then
   notify(...)

The value to test against and the time value are both KRL expressions. The timeframe (hours in the example) can be seconds, minutes, and so on through years. This reflects our belief that its often useful to know that action has been taken within a certain timeframe.

Persistent trails can be tested using the seen predicate. There are two forms:

  • The first form asks whether a regular expression has been seen within an optional timeframe (specified in the same way as timeframes for counters and flags). So you can say:
    if seen "/archive/2006" in ent:my_trail then
       notify(...)
    

    or

    if seen "/archive/2006" in ent:my_trail within 3 days then
       notify(...)
    

    The first argument is a string that is interpreted as a regular expression.

  • The second form tests whether a place in a trail matching the first regular expression comes before or after a place in the same trail matching a second regular expression. This example
    if seen "/archive/2006" before "/archive/2007" in ent:my_trail
    then
      notify(...)
    

    would fire the action when a place matching the regular expression "/archive/2006" was placed on the trail before a place matching the regular expression "/archive/2007".

Complete documentation for persistents is available on the Kynetx developer's site.

Here's how you could use persistents to create a notification that has a "don't show this to me again" checkbox:

rule show_note is active {
  select using "www.windley.com/cto_forum" setting ()

  pre {
    h = <<
Welcome to the CTO Breakfast page!!!<br/>
<input id="KOBJ_clear" type="checkbox" name="option1" value="Clear">Don't show again
 >>
  }

  if not ent:dont_show then
    notify("Welcome", h)
        with sticky = true

  callbacks {
    failure {
      change id="KOBJ_clear" triggers set ent:dont_show
    }
  }
}

This shows a notification as long as the entity persistent ent:dont_show is false (or undefined). It also gives a persistent statement (set ent:dont_show) to be triggered whenever the user changes an element with the ID KOBJ_clear.

We could put another rule in the ruleset to clear the flag:

rule  clear_notice is active {
   select using "www.kynetx.com" setting ()

   notify("Notice", "I'm clearing the notice flag for www.windley.com")

   fired {
      clear ent:dont_show
   }
}

This clears the ent:dont_show flag whenever this rule fires, which it always will as long as the selection statement is matched since the action has no conditional. So in this example, you'd see the notification on the CTO Breakfast page until you checked the box, then you'd never see them again unless you visited the Kynetx web site. If you did, the flag would be cleared and you'd begin getting the notifications again.

This example ruleset is a good example of the abstractive power of KRL and the cross-site capbilities of the KNS system. Clearly you could do something like this with Javascript, PHP, or other systems, but it would take more code and the purpose would not be as clear. Moreover, doing it across multiple Web sites would require the cooperation of those Web sites in some way, whereas with KNS, the client is controlling the cross-site capabilites--and it's all built-in.

In addition to adding persistents, there were a few other changes:

  • The KRL runtime is now based on jQuery 1.3.2. The previous runtime was based on 1.2.6. The complete power of the jQuery system is available to KRL developers who dip below the abstraction layer provided by the language and write their own Javascript.
  • We added a page:url datasource to return information about the calling page's URL such as domain name, path, query string, etc. This could previously be done using the replace operator, but seemed like such a likely use case that we built it in to make it easy.
  • We added a page:param datasource to return information about page parameters (set in the KNS_config object by the endpoint). This data has previously been available via the page:env datasource but that availability will be curtailed in future releases for security reasons.

If any of this looks interesting to you, please signup for a Kynetx developer's account--it's free and come to our developer's conference in November. We'll show you how to easily create context-aware, cross-site Web applications that do things server-side solutions can't even imagine.