Transferring a Large Number of Objects From Server to Client

Date: 09-Nov-98


o-< Problem: In today's distributed environments, transferring large amounts of data between a server and a client can cause serious performance problems. What can be done to avoid this?


o-< Anilkumar Ambati posed the problem:

I have a client that needs to display huge amount of data on a single mouse click. [...] For example: If there are 100,000 thousand customers and each customer object has lot of behavior and data associated with it.

[...] It might take few minutes or so to get all the data from database. [...]

I have some thoughts in mind:

Make query to find how many customers are there (100,000). If this number is more than predefined number (100), get some data (100) at a time and populate the table as data is read. [...]

Get predefined number of records and get remaining data on demand. But this can be complicated.


o-< Thomas Moore replied:

I'm reminded of an old joke. A ship goes down, and the captain of a nearby vessel comes over to rescue the three passengers stranded in the water. He explains that before he can rescue them, they each must answer a question. The first one is asked "What is the worst disaster in naval history?" to which he replies "The Titanic." The second one is asked "How many people died that night?", to which he replies "About 1,500." The captain turns to the third one [...] and says "Name them."

I see the initial question as being like the captain's third question. It is going to result in a deluge of information that the user might think he wants, but actually does not. The user is looking for information. For example, if the user says that he wants a list of all customers sorted by state, IMHO you're better off displaying something like a summary that shows (say) a list of states, the number of customers in each, and any other useful summary data (total purchases, average purchases, salespeople, and so on). The user would then be able to drill down to more detailed information about each state.

What happens is that the user requests the information be displayed so that he can parse it himself. Being a user (rather than a programmer), they tend to think of data as if it were a printout (like a spreadsheet), rather than raw material to be processed into information. The user probably (at least subconsciously) realizes the difference, but in his worldview he expects to have to do the summarizing himself. I've actually seen people staring at a screenful of information with a hand calculator going, trying to do what should have been anticipated by the programmer.

I absolutely believe there is a pattern here, one that talks about UI design with regard to information presentation - and it's not about cramming one hundred records on a screen by using a scrollbar. And it's also not about structuring efficient data transfer methods to get 100,000 records across a network by doling out 100 at a time. I think it should be called something like "Summarize, Don't Braindump." :-)