Date: 8/10/2010

Start Time: 1330 MST on 8/7/2010

Stop Time: 2200 MST on 8/9/2010

Duration: 02:08:30 (DD:hh:mm)

Severity: Sev2 - Degraded Platform Performance

Incident Summary:

KNS application users began to notice slowness on the platform which got worse over time.

Services Impacted:

Initialization Servers

Evaluation Servers

Log Servers

Root Cause Analysis:

The root cause was determined to be caused by a infrastructure change, This change introduced a new class of KNS platform server to replace an older class of KNS platform server.

Recovery Steps:

Once the root cause was identified, the servers were pulled out of rotation and the original servers were put back into service. Currently the platform is in a "last known working state" and will remain there until the new servers have undergone further analysis to pinpoint the failing component(s). Once they have be certified and tested, they will be put into service in a "rolling migration" fashion where they will be slowly introduced and allowed to soak in.

Date: 8/10/2010

Start Time: 1630 MST on 8/6/2010

Stop Time: 1141 MST on 8/9/2010

Duration: 02:19:15 (DD:hh:mm)

Severity: Sev2 - Internet Explorer users experienced an error which prevented KNS applications from firing inside the browser.

Incident Summary:

During a routine code deployment, a runtime was deployed which contained a syntactical error which prevented Internet Explorer from executing the runtime. This condition prevented KNS applications from firing because the runtime could not be executed.

Services Impacted:

Initialization Servers

Root Cause Analysis:

The root cause was determined to be a syntactical error within the runtime code which was deployed into production late Friday (8/6/2010) afternoon.

Recovery Steps:

Once the root cause was identified, a new runtime was developed, tested and promoted to production as a "hot fix". Furthermore, it has been determined that the current automated and manual testing processes do not provide sufficient code coverage for Internet Explorer. These processes are under review and will be augmented to close the coverage gap so this issue will not repeat.

Date: 7/9/2010

Start Time: 1134 MST on 7/9/2010

Stop Time: 1222 MST on 7/9/2010

Duration: 00:00:48 (DD:hh:mm)

Severity: Sev2 - Degraded performance processing ruleset evaluations on the KNS platform with periods of unavailability (<15 minutes) of the evaluation servers.

Incident Summary:

During a routine code deployment, requests for ruleset parsing were negatively impacted by the inability of the KRL parser to keep up with demand. During this period, requests for ruleset evaluations were either delayed or blocked.

Services Impacted:

Evaluation Servers (cs.kobj.net)
KRL Parser (krl.kobj.net)
AppBuilder (appbuilder.kynetx.com)

Root Cause Analysis:

The root cause was determined to be a single ruleset which took on average > 60 seconds to parse. Due to the volume of parse requests for this particular ruleset, the requests for parsing were never completed and were retried. As the parse requests compounded, requests for other rulesets were delayed or dropped.

Recovery Steps:

Once the root cause was identified, the offending ruleset was deactivated and the platform returned to normal operation within minutes. The author of the ruleset has been notified and the Kynetx Engineering team is reviewing the ruleset's syntax.

As we mentioned earlier, along with the new AppBuilder came an open source command-line Ruby gem that lets you access most of the features of AppBuilder through an API. This allows you to write KRL with your favorite editor or IDE and then push your code to KNS from the command line or a plugin.

Installing the Gem

You can find the complete installation instructions here, but this is the basic process:

1. Install Ruby (version 1.8.7 or later).
2. Install the KRL gem (leave off sudo if you're on Windows):
   sudo gem install krl
3. Use OAuth to connect to the KNS servers:
   krl-connect
   This will start a server on localhost:3009 that will walk you through the OAuth ceremony. Once that's all finished, you can kill the webserver (with Ctrl+C).

Once the gem is installed, you're ready to get started.

Basic commands

First off: krl help is the definitive guide to the gem. The command will list all the possible things you can do, along with their syntax. (Tip: the text is easiest to read if your terminal is at least 90 characters wide.)

Check out your app

Run krl apps to get a list of all your apps. That displays the ruleset ID and name of each app, along with your role on the app (owner or developer).

Note: Until you save your app in the new AppBuilder or with the command-line gem, its name won't show up in the list. You can see that in the screenshot--the third app in the list has only been edited in the old AppBuilder, so the name doesn't show up yet.

Once you know which app you want to work on, a krl checkout command with the ruleset ID, like this:
krl checkout a163x2

This will create a new folder in your working directory with the ruleset ID as the name. (You can rename the directory to something more descriptive if you like.)

Editing your app

You can edit the <rulesetId>.krl file with any editor you like. Near the end of this post are some plugins for common editors to do syntax highlighting and integrate with the command-line gem.

Committing your changes

Once you've made changes and want to commit them, simply run krl commit in the same directory as your .krl file. If your code had any parse errors, these will be displayed to you. If everything passed, the gem will tell you what version was committed:

If you (or another developer) makes some changes later on AppBuilder and you want to pull them down to your local copy, simply run krl update. Note that this will overwrite your local copy of the code; it doesn't do version-control-style conflict resolution for you.

Generating endpoints

The gem will let you create both test and production endpoints. Here are the various commands:

• Test bookmarklet: krl test
• Test Infomation Card: krl test infocard
• Production bookmarklet: krl generate bookmarklet
• Production browser extension: krl generate [firefox|chrome|ie] <name> <author> <descr>
• Production Information Card: krl generate infocard <name>

Viewing older versions

If you ever want to go back and look at an older version of your app, simply run a krl show <version number> command. This will print out the KRL from that version to your console. You can even pipe it to a file to open in your editor (although you'll have to remove the first line, "KRL for version XX," if you want to use the file as a valid ruleset).

Running the krl versions command will list all the versions your app has been through, along with the timestamp, the developer who made the commit, and any notes associated with each version. That list will also tell you which version of the app is currently in production.

A note on the commands

Most of the functions must be performed inside a directory where you've checked out your app. commit and update are good examples of commands in that category. A few functions can be performed anywhere, since they're not specific to any particular app. Those are apps, checkout, create, and info. Refer to krl help for more information about where you can run the various commands.

Integrating with your favorite editor

The command-line gem is platform- and editor-agnostic, so you can create plugins for any editor or IDE the offers a programmable interface. Here are a few of the ones that have been developed so far:

Emacs

This Emacs mode provides syntax highlighting for KRL. Find the .el file here.

TextMate

This TextMate bundle maps Cmd+K to save the app and then commit it. Find it here.

Vim

This Vim plugin enables syntax highlighting and intelligent indentation of KRL files. Get the .vim files here.

Growl and libnotify integration

You can have bash notify you when the krl commit command finishes running. See the solution here for Growl (Mac OS X) or here for libnotify (Linux).

Other editors and IDEs

Have you created a package for any other editors or IDEs? Leave a link in the comments!

The gem is open source

The Ruby code for the KRL command-line gem is open source. You can find it on the Kynetx github account.

If you have any issues with the gem or if you'd like to offer improvements, please do! You can submit bug fixes on the Issues tab on github. Or (even better) you can fork the code, fix the bug you found, and then send us a pull request.

Announced at Kynetx Impact in April, the powerful new Kynetx AppBuilder is now live at http://appbuilder.kynetx.com/. Here's a quick tour to get you up to speed with the awesome new features.

Logging in

AppBuilder now uses OAuth authentication for long-life logins, meaning that you won't have to log in again every time you visit the site.

Application list

When you first log in to AppBuilder, you'll see a list of all your apps with their ruleset IDs. Clicking on the name or ruleset ID of an app will open the app so you can edit it.

You can now easily duplicate an app with the link in the right-hand column.

Another cool new feature is searching--you can filter the list by app name, ruleset ID, or by your role in the app (owner or developer).

Editor

The most salient feature of the editor in the new AppBuilder is the beta Bespin embedded editor. It provides a lot of the functionality you'd expect from a full-fledged text editor. Syntax highlighting makes it easy to see that your code is correct, allowing you to catch errors even before going through the parser.

Even if you've never used Bespin before, it will be pretty familiar to you. It behaves quite like a desktop code editor: you can type as usual; it will handle indentation (including the use of the tab key); you can select and scroll with the mouse. And of course syntax highlighting, a must-have for any serious coder, is built in.

Saving an application is as easy as Ctrl+S (also Cmd+S in Firefox on the Mac). It's all done through AJAX, so you can keep working on your code while it saves.

If your code has errors in it, you'll see en error icon below the Save button, and the error log will pop up at the bottom of the window. It describes where the error occurred in your code. If you want to close that log so you can get back to coding, click on the X in the corner of it or press Ctrl+E. Clicking on the error icon also toggles the error log, so you can show or hide it as you like.

If the save was successful, you'll see a green check mark below the Save icon, as well as the timestamp of the last successful save.

Bespin uses the HTML5 canvas tag, so any browser that supports it will be able to run Bespin without any trouble. If you're using Internet Explorer, you'll be redirected to an old-style (less feature-rich) editor.

Production and development versioning

AppBuilder gives you an intuitive, simple way to manage production and development versions of your app. The Deploy tab shows you a list of all the versions your app has been through. You can "deploy" a specific revision, meaning that it will be the production version that is run on the endpoints your clients use (browser extensions, InfoCards, etc.). Your development branch is the most recent revision of the code that you're using to test.

You can give each revision a label to make it easier to identify. That way if something breaks, you can easily go back to an older stable version that you are sure works.

The Deploy tab also lets you see the KRL for any particular revision. Simply click on the version number in the left-hand column.

Testing and distributing your app

The Test and Distribute tabs let you work with various test and deployment endpoints for your apps, such as browser extensions, bookmarklets, site tags, and InfoCards.

The other cool new feature here is integration with the Kynetx Marketplace. You can list your app in the Marketplace with just a few clicks. Choosing the "Marketplace" option from the Distribute tab will take you directly to the Marketplace and pre-fill all the fields that AppBuilder already knows about.

AppBuilder Ruby Gem

The new AppBuilder is an open source Ruby on Rails application, and it's built on top of an open source Ruby gem we created. What this means for you is that if you'd prefer to write KRL in your favorite desktop editor, you can! The Ruby gem allows you to access the majority of the AppBuilder features right from the command line or your IDE. Keep your eyes open for a blog post all about that, coming soon.

Where do we go from here?

Below you can watch the Kynetx Webinar that walks through AppBuilder for a more in-depth introduction:

You can also watch Mike Farmer's presentation of AppBuilder from Kynetx Impact 2010 (a series of 7 videos beginning here).

Try it out today! Log in now with your Kynetx account and get started right away.

If you have questions about using AppBuilder, feel free to ask them on DevEx. You can also send support requests to support@kynetx.com.

The following is an important announcement regarding KRL and comments from Phil's Technometria blog that will affect everyone.  We wanted to make sure this announcement gets to everyone.

Poor decisions in language design are tough to hide because fixing them is akin to changing the API of your library. You can hide all kinds of sins below the covers, but errors in the syntax and semantics can only be fixed by confessing those sins to the world. Consider this a confession.

One of the niggling little details of Kynetx Rule Language has been comment treatment. The parser we user, Parse::RecDescent, does not include a lexer—tokens are specified as regular expressions in productions. That works fine for everything but comments. Because there’s no lexer, you can’t flag a token as a comment and then throw it out. There are a number of ways around that; the method I’ve used is to strip comments from the source using a regular expression before they go into the parser.

Comments in KRL are like those in Javascript: a double slash starts a comment and it goes to the end of the line. If I were designing KRL over again, I’d definitely rethink that because it makes stripping comments from code that contains URLs problematic. The regular expression for matching comments is not trivial. Nevertheless, Javascript-style comments are what we’re stuck with for now.

The problem of stripping comments from code without stripping code was made even tougher by a poor decision I made regarding extended quotes a few years ago. (Extended quotes in KRL start with << and end with >>.) For some reason, I decided that stripping newlines from the material inside an extended quote was the right thing to do. It’s not. Quoted material ought to be left alone. That’s why the developer quoted it; they want it to stay the way they wrote it.

Lately there have been some problems that have caused me to dive into this and rework it. Consequently, starting with the next code release, we will no longer strip newlines from extended quotes and not remove anything—even things that look like comments—from inside them. When material in an extended quote is used for a Javascript emit, it will be emitted unchanged. When it is turned into a Javascript string we will escape quotes and newlines so that they remain in place when Javascript evaluates them.

This will normalize some things in KRL, but there may be code in applications that relies on the fact that we’ve been stripping newlines. In particular, Javascript won’t let you split strings across lines. Since we’re escaping material that gets turned into string, there shouldn’t be any problem there. But emitted Javascript that hasn’t been careful about this will break.

Another fallout from this change is that any Javascript comments you’ve been putting in your emits will remain whereas before they’ve been stripped. That means you shouldn’t be putting company secrets in your Javascript comments since they might be seen by users. On the positive side, the fact that were now leaving newlines in the Javascript source will make single stepping through your code more fruitful.

As always, we’re happy to work with you to help you through this transition. I anticipate that this code will roll out Tuesday afternoon. We’ll put notifications in the appropriate places beforehand.

Last week at IIW Dion got an opportunity to spend some time with Phil where he showed him how Kynetx breaks the old web silos with contextually relevant experiences and how Google's Chrome Webstore can help to speed up the trend.  Check out his lastest post.

SALT LAKE CITY, Utah – May 11, 2010 – ACULIS announces a partnership with Utah-based Kynetx™. ACULIS will work with Kynetx as a professional services partner to aid in the company’s custom software development and integration needs. The relationship will provide Kynetx increased capacity to deliver on customer implementation and customization demands.

Kynetx provides the first Context Automation Development Platform with applications offering customer relationships and revenue streams never before seen by extending experiences out beyond a single site or device.

“The Kynetx technology is a great match for our software development services division. We are excited about the technology as an integrator and innovator,” says James Stone, CEO of ACULIS. “Augmenting and automating software development tasks is one of our core areas of focus.”

With AppBuilder, Kynetx’s flexible app creation tool and the Kynetx Rules Language (KRL), ACULIS developers can quickly create secure, “opt-in apps” that offer a wide range of functions, for example: messaging through multiple touch points (including desktop, browser extensions and smart phones), information query, replacing content for more relevant information, offering personal targeted search engine results, and much more. For businesses, the result of these closely integrated technologies is a simple way to quickly increase KPI’s like sales dollars per transaction, as well as customer satisfaction and loyalty. For user, they experience the web in a more contextual and relevant way.

“The relationship is truly exciting,” explains Dave McNamee, Kynetx VP of VAR Services, “Kynetx and ACULIS together will be providing users the ability to implement and deploy the Kynetx technology efficiently and affordably.”

——————–

About ACULIS
ACULIS is a global IT services outsourcing company, founded in 2002. Customers rely on ACULIS for an integrated approach to custom software development, quality assurance & software testing, globalization & multilingual translation services, and flexible IT staffing & recruitment . With an emphasis on blended-shore IT solutions the ACULIS goal is to meet and surpass any software development objectives.

About Kynetx
Founded in 2007, Kynetx is a private company that has developed a proprietary rules-based development platform that is the first infrastructure to support applications that understand and respond to user context across any web-enabled device. Developers and businesses can use this platform to quickly mash up data, functions and interface points in order to create the next era of software services and Internet applications.

###

SOURCE: ACULIS Inc.
DATE: 05.11.2010
CONTACT: Craig Cleveland
PHONE: 801-377-5360
WEBSITE: www.aculis.com