Window levelling an image is a way of specifying a linear transform to the pixels in an image, clipping values to the output range. For a 0-255 output range, and a width width of w and center of c, the transformation (to the grayscale component L), is:
Loutput = 255 * (Linput - c + w/2) / w clipped to 0..255

The problem is to figure out how to efficiently apply this to an image within a browser, without some plugin such as a Java applet or an Active X control. While doing this, the server resources still need to be kept in a reasonable bound, as do network resources, and the latency of a change needs to be kept fairly low.

The best performance possible is that the browser fetches the image exactly once and then applies the window level continuously as appropriate. Failing that, fetching the image some fixed number of times like twice or three times, once for initial display once for window level, and a third time once window levelling is done would be ideal. Finally, fetching the image multiple times might be required. Currently, the only known approach to achieve one fetch is to use SVG colour transforms to modify the window level. Any AJAX approach is likely to involve multiple fetches, perhaps many of them per second.

SVG Approach

The SVG approach uses the feColorMatrix to apply the window level as a transform to an image. This works as long as:

1. SVG 1.1 feColorMatrix is implemented
2. JavaScript is working
3. Updates of the feColorMatrix work (this only partially works in Batik SVG - you need to also clear/reset the CSS style element)
4. SVG is recognized in-line, not as a plugin

An example of this approach can be seen by using the WL SVG toolbar - this will be enabled by default in a current build of the client. Just use the regular interface, navigate to an image, and select the wl-svg tool. Copy this URL to Batik Squiqqle (an Apache project), and then use window level in the normal way (left mouse down to start WL, left mouse up to complete it.)

Static HTML approach

It might be possible to get a purely static HTML version of window levelling by using a server side image map, and using absolute positions to choose the window level. This would still require a round trip for each display, but the user could likely get used to selecting points on the screen fairly quickly to modify the window level.

Server Update Image Approach

This is where it becomes far harder to get acceptable performance for window level, because a server round trip is required for every window-level change (of course, the static HTML with a server side image map also has the same problem.) In general, somewhere around 2-5 updates per second is required to get acceptable performance for window levelling. This has implications for server, network and client costs.

Server Costs

The effort on the server to provide a window level comprises:

1. Figure out where the image is located
2. Read the DICOM from disk
3. Decompress the image
4. Apply the Modality LUT
5. Re-size the image
6. Apply the Window Level
7. Compress the image
   Assuming that regular image viewing is acceptable in terms of server costs, the question becomes how much more of a burden does adding window levelling impose. After analyzing the various features, it appears that only CPU usage maybe critical in terms of server load.

Disk IO

Disk IO for reading the image 5 times or more per second with 10 concurrent users would almost certainly be a problem. Thus, we need to minimize this step. The simple expedient of having a server cache that caches just before the window level is applied will be important - then, a window level operation will only result in reading the data 3 times - once for the initial display, once while window levelling, and then once again to window level the full-resolution image.

Memory

For a repeated operation such as window level, some of these can be cached - in fact, everything before applying the window level can be cached on the server. The cost for caching a 512x512 grayscale, 16 bit image is 512 k, uncompressed, assuming essentially perfect storage (which should be very close to what is achievable without too much effort.) However, by the time a 4k x 4k image is used, the cost goes up to 32 megabytes, and the assumption is that we might have 10 people window levelling concurrently - that would require 320 megabytes. This is still probably acceptable, as very few of the users will use that large an image - in fact, most likely none of them will. Thus, it can be concluded that server memory is NOT an issue for window levelling.

CPU Usage

Assuming the system keeps the image in-memory, with a version that only requires window levelling and compression, then the primary costs are for those two operations. The window levelling produces 8 bit/pixel data, using a look up table. This should be O(h*w) where h is height in pixels and w is width. The compression time is under 20 ms on a cached image (ofter 15 ms or less) - that suggests that a 512x512 image will not cause server problems to window level.

Network Usage

From a server point of view, one would assume a 1 gbit/second as a bare minimum. Thus, assuming 512x512 images, compressed lossy at least 10:1 would produce images of around 26k. The network could handle 3846 such images per second - thus, this seems unlikely to be a bottleneck, as 10 concurrent users window levelling at 5 images/second is only 50 images/second - well under the network limit. Even at 100 concurrent users at 10/second, only 1/3 of the network capacity would be used for images (this seems like an unrealistictly high limit based on previous observations of actual Agfa WEB1000 installations.)

h3 Client Costs
The client costs are looking at whether a client has the resources to display window levelling at 5 images/second.

Network Usage

Area of concern
From a client point of view, the networks costs only need to handle 5 images/second - as shown earlier, that is 26 k/image, or 131 kbytes/second. That is 1.3 mbits/second - definitely below any local area network, but within the range of broadband, and above that for modem usage. There are 2 ways to reduce this - decrease the image size and decrease the quality. In general, it is recommended to decrease image quality first and image size second, only as needed. A change to 100:1 lossy JPEG produces an image that has some significant defects, but may still be acceptable for fast window levelling. Such a change would produce network costs of 13.1 kbytes/second - that is still twice modem speeds, but is below all of the broadband performance. Beyond that, it is likely that a decrease in the number of frames/second to around 2-3 frames/second would be required for a modem user. Given the requirements here, it seems very likely that this area will need to be further investigated for image quality and performance in order to have reasonable network use for users.

CPU Usage

At this point, it is expected that any client system can keep up with decoding at least 5 JPEG 512x512 images/second, and thus CPU usage should not be a consideration. Time for the server to respond should be around 25 ms on an unloaded server.

Interaction Delay

Even if all the systems can manage the throughput, it isn't clear yet whether the combined delay of needing to do a server round trip will cause the whole system to have too much lag for the user to accept. There are some possible solutions here as well - but what they are depends on where the lag occurs.

Browser request lag

This should be very fast - the browser should get a request off to the server very quickly. Assume 1 ms here for now as a guestimate.

Network lag

The size of the request and the size of the response, and the performance and load on the network all determine how much network lag there will be. The size of the request is about a 1/2 k, while the size of the response is around 26k. The time to send these, assuming everything else is done before/after and that the network can be use close to capacity would be, for a 1.5 mbit/second network: 2.7 ms request, 139 ms response. Thus, assuming an infinitely fast server, the user wouldn't see an update for at least 143 ms. This is still likely within the acceptable range if there are no other delays - however, most likely there will also be 100 ms or more of other network delay. That would mean a minimum refresh in 250 ms - that is starting to get into the noticeable range, even with a server that responds immediately. Thus, the network lag alone may require small image sizes on slower networks or ones with more delays.

Initial Read Lag

Although it is important to consider overall for initial start of window levelling, the time to read an image into server memory and set it up for window levelling will not currently be considered - this is an area for looking at in the future.

Server compute time lag

The server takes about 25 ms to compute the updated image. This will bring the speed on a fast network to around 175 ms/image at best. This is still within the acceptable range.

Browser display time

It is expected that the browser will take about the same amount of time to display the image as it took to compress the image (probably a bit shorter, but this is a reasonable first order estimate). Thus, the time per image is now up to 200 ms/image - that is the maximum value acceptable for 5 images/second. Thus, the total time is VERY tight.

First questions

Given some of the above observations, it looks like we need to measure window level application time and JPEG compression time on the server, as well as throughputs for those - that will determine what we can do around lag and so on, and whether or not that will be an issue.

Time to respond with a 30k cached response object at default JPEG encoding, starting from an 8 bit grayscale image took 12-20 ms (include network transmission time), while the window levelling takes 3 ms. That would allow server wise, even with just 1 CPU around 50 images/second (a bit less since efficiency decreases near the resource limits, but that isn't really relevant for this discussion). CPU usage was 8% for 1 user, so even a single CPU should be able to support 10 or so concurrent users, CPU wise (assuming nothing else is going on).

Given the measured time to respond, and expected times above, it looks like it should be possible to manage 5 images/second on a medium to fast network. Thus, proceeding with actual attempts at window levelling is definitely the next step.

Answers

Even for a 512x512, 16 bit image at regular lossy levels, 5 images/second appears reasonably smooth and the network and browser can keep up. The network on 100 mbit/s network close to the uses up 1.5% capacity, or 188 kbytes/s - roughly the expected usage amounts. I don't currently have a simulation environment setup to allow other levels to be tested. Client CPU usage on a 3 GHz machine is 24% of 1 CPU. Memory usage is fairly high - around 512 mb for Firefox. IE is similar in CPU usage. IE and Firefox are both flicker-free, so the appearance is quite nice.
Server CPU usage on a reasonable server machine (2.8 GHz p4) is around 8% CPU for a Linux machine running JAI CLib image IO i586 version 1.0_1.

Further questions

How to capture the mouse

How to get notifications of image received/ready for display

Slow display under IE (not update while moving the mouse)

Standard WADO implementation being fast enough, but not too much of a memory hog

It would be very nice to have the WADO implementation that handles the window levelling also work fairly well for regular WADO applications, and not be too much of a memory hog - it should also be able to handle zoom/pan of images as large as 64k x 64k (yes, there are medical images that big, and in fact some on the order of 128k or more on a side - mostly lab results in visible light.)
The standard WADO service should also handle cached information for DICOM for various cache types and image types, but not cache too much data.

Caching

The memory cache should be used whenever there is a high likelihood of asking for some of the same data a subsequent time. It is fairly important to NOT use it for requests for subsequent images in the same series.

The disk cache should be used for things like thumbnails, overlays and other objects likely to be requested multiple times, but often at some length of time apart, especially if it is expensive to recompute the object.

At the moment, the design of this is going to be fairly simple - the disk cache is checked first, and used if present. If not, then the image is loaded, and put into L1 cache, as well as being saved to the Disk cache. A subsequent request at a different W/L, but the same resolution will promote the data to the L2 cache for somewhat longer storage. L2 cached items are NOT saved to the disk cache, as it is expected that they will be re-used multiple times.

Other options will be added as required once we understand what is actually needed (we are avoiding pre-mature optimizations and extra up-front work.)

Possible Options

Here are some ideas that we may need to consider in the future. It is too early to figure out which of these are needed.

1. Identifier cached - in order to figure out what goes in L1 and what goes in L2, if L1 is really large, then it might not be possible to store enough data in it.
2. Size and/or cost to read based caching - large items may not be cached at all, as it maybe similar costs to just re-read them.
3. Partial image loading - for JPEG2000 type images just loading to the required resolution and caching at one or more resolutions.
4. Header caching - if many parts of the image are loaded, or for multi-frames, it might be worth caching the header and/or disk location.
5. Common resolutions caching - it may sometimes be worth using a common resolution, and then re-sizing when sending, or perhaps even sending the wrong sized image (not sure what the implications of that might be.)
6. Resolution doubling/having - for a quick, low quality image, the lower resolution could be re-used to produce the higher resolution, or the higher resolution could be re-sized instead of reading in the original data. This is probably best done client side when it knows that the image is in client side cache.
7. Client requests for specific cache level with start/stop caching - this would allow more efficient use of space if it is KNOWN that the client is going to be doing window levelling for some length of time - perhaps based on mouse up/down type actions.