Idea for Thin middleware/Distributed Web applications

I have an idea (oh no not another) that draws on a number of different things that seem to be coming together to offer hope of a new architecture for developing and deploying web applications. One that offers a richer experience for the user coupled with more flexibility and resiliance for developers and administrators...

yes it's a magic bullet ;-)

No, more seriously. I have been thinking about how the following connect:

• REST based servers
• XMLHttpRequest based web browser UI's see 42: Auto complete comes of age - SitePoint DHTML & CSS Blog and 42: Joel on Software - Wednesday, December 15, 2004 and 42: Rest Fest.
• Tim Bray's comments on threading for multi-core servers ongoing: Software in the TLP Era.
• Jon Udell's comments on tiny http servers eg The present and future value of Python and  XML.com: Lightweight XML Search Servers.
• The problems the SpreadFirefox website had with bandwidth which required all images to be handled by a sub-domain. (42: Notice: Please change your affiliate image links - Spread Firefox)
• The cost of hosting dynamic sites for charities, when you want to go beyond php or mod_perl.
• The announcement of version to 2 of Sleepycat Software: Products: Berkeley DB XML
• Joel on Software - Friday, December 10, 2004. (Google Suggest raises the bar for web apps)

So my thinking is to split up the application into chunks that are hosted separately.

1. Static html (and images, but put them on a sub-domain from the start).
2. An application UI written using JavaScript (served with the static html) calling REST based http servers
3. A variety of small http servers handling simple REST api's

The fact that your REST servers are scattered over different domains will be invisible to the user, all they see is the html from your static site. The REST servers can be small chunks of functionality bought in from a number of suppliers. From the very simple (attach notes to anything) through payment servers to whole systems eg order processing, contact management. All the connection is hidden from the user as javascript uses XMLHttpRequest to  GET/PUT/POST/DELETE to the REST api on whichever server it needs.
You will need a core REST server to handle security and the various other REST servers will need to use a REST API themselves to check permissions etc.

We have seen a number of ways that web sites include bits of dynamic content from other sites in the side bars etc (ads, weather, clocks, blogrolls ...). Now I am suggesting that we build entire web applications this way. It means taking the server part described in model N of Serving Client-Side Applications (my earlier comments at 42: Serving Client-Side Applications) and splitting it into those bits that can come from static hosting solutions and the REST bits that do not need to come from a single server.

When you run applications like this on your intranet it gives you the ability to make use of all those threads your multi-core processor offers, even when using a programming language like python where each instance does not thread very well. (eg lots of small http servers running small parts of the REST api). I have been reminded recently how much smaller and simpler a REST based application server is than a full html application server. Moving client state and presentation out of the server really simplifies things.
In fact if you look at some of the work being done with Inspirational Technology: Introducing Syncato and Sleepycat Software: Products: Berkeley DB XML and by Jon Udell the http server almost disappears as the rest api maps almost directly onto the XML database.

When you only have static hosting you can still have fully dynamic web sites. You application is in the javascript that comes from the static pages and that calls all the services it needs. Critically it is the javascript that holds state on the client.

Note the flexibility in deployment you gain when the javascript calls one REST server to discover the URLs for all the services it needs. Move a service to another domain, just change the data in one REST server. Want to load balance a particular service then just alternate between the servers when you return the URLs. Fail-over: return multiple URLs for each service.

You should be able to rent significant chunks of functionality yet have total control over the user experience.

Look also at the flexibility. We can develop UI's quickly using an http server and html, better looking using Laszlo and best functioning using javascript.

I suspect the key challenges to be resolved are:

• UI development productivity with JavaScript (ugh)
• Security: common practices needed
• The move away from a single SQL database containing everything. Now your reporting gathers data via REST from a variety of sources. Maybe you gather from a variety of transaction processing databases into a single analytical database.
• Fear of lack of control.

What does anyone else think? Have I been drinking too much coffee or eating too much turkey?

Comments

[erik]

the concept is not too far off...the REST architecture may be a little too restrictive to achieve your goal.

essentially, you can do what you're suggesting with frames and cgi scripts. your "static webpage(s)" would provide a set of tools -- feeding information to webservers... it's ok - but it still is leaving most of the processing to your web-client.

I've been fiddling with something similar, but think the basic model here is too restrictive. I've been trying to move my "productivity tools" to the internet (so they can completely live out in the ether somewhere).

Consider something simple like a blog. Say you want two or three websites to hold your blog-content (for redundancy)

you go to siteA and kick open a form to type in a new entry...hit submit, and the item is added to the message-store at siteA. now you need a way to move it from siteA to siteB and siteC...CGI's aren't really robust enough for that...and it's a hassle to pass the data back to the client and make a user re-submit.

so I've been thinking about changing the formula a little...say then your web-page includes a small applet. The applet joins the client machine into a network of machines (maybe an Instant-messenger chat-room?). then pass requests to all the machines...so you could type the blog-entry locally on the client and "post" it to all the machines listening ... your blog gets pushed to each of the storage areas at the same time.

similarly, queries to the chat-room could find information across each of the servers...again, leaving presentation as an issue for the client.

if your way-out where you can't join the net, basic web/cgi scripts from any of the servers

I guess what I'm getting at - is you still need a mechanism where you can reqest information from a number of servers.. HTML isn't too good at that.

[DaveW]

Applets can only talk to the original server. I would prefer the servers to handle such syncing, as you say keeps the client thin and also the client has less reliable bandwidth.

[Daira's pop]

I come from a background of classic 2 tier (vb and ms sql server) platform and I guess the trouble I am having is seeing how to deal with a stateless app and concurrency. Look at the problem of a customer of ours discovering a problem with an order and two people call. One person calls the sales desk and another person calls the sales manager to yell at him about the problem ever happening. The sales manager adjusts the customer's discount and the sales clerk overwrites the transaction with the order adjustment. In traditional applications you would start a transaction to lock the data or do a "freshness test" on the second transaction to see if the records that it holds is valid. How do you handle this situation in a stateless multi user app?

I have been looking a mod python and tinkering with sqlObject. I can see downloading domain type validators written in JavaScript in the outer frame of the rest app etc. What I don't get is how to handle the concurrency.

Help? Suggestions? Ideas where to look? Fish? 42? How do you explain tea to a food fabricating computer?

[DaveW]

Daira,

H'mm big questions. The toolsets I have used even since one tier applications (Dataflex) have not used pessimistic locking as you describe. There are 2 key elements to the answer. a) stateless b) optimistic vs pessimistic locking. The answer deserves more than a comment, I'll write a post on it at some point as I think the answers are interesting.