A whitepaper: The LiveDist(Java) (java data-base distributor) product

•	Galiel-3.14 Ltd. has intellectual property rights relating to implementations of the technology described in this white-paper. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.
•	This presentation is provided “AS IS” without warranty of any kind, either express of implied, including, but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation could include technical inaccuracies or typographical errors. Changes are periodically added to the information herein. These changes will be incorporated in new editions of the presentation. Galiel-3.14 Ltd’ May make improvements and/or change in any technology, product, or program described in this presentation at any time.

General description:

LiveDist(Java) is a robust J2EE product, that is used for swiftly and selectively distributing database modifications to thousands of clients, while preserving transactional integrity. The updates made by each committed transaction are distributed in 1-2 seconds to all interested clients (there may be thousands of them).

LiveDist(Java) is written in Java, but when using triggers and JDBC to connect it directly to the database, it can be used by systems which are written in any language, and on any platform (the LiveDist(Java) servers run on dedicated low-cost machines and not on the application's servers). Furthermore, LiveDist(Java) distributes the database transparently to the application code (the business logic), and thus can be used as an extension to existing systems.

The LiveDist(Java) intercepts database modifications in one of two mechanisms:
•	By using database triggers, which accumulate log of inserts, updates, and delete operations, in a shadow-table. The data is periodically fetched from the shadow-table, packed into a single buffer, and passed to LiveDist(Java) (by inserting the buffer into LiveDist(Java)'s log-table). This mechanism is transparent to the application, but has an overhead of executing the triggers and writing into the shadow-table. The overhead does not depend on the number of clients.
•	When the software, which the application uses for accessing the database, can expose to the application, a log of all database changes in the current transaction, that log can be passed to LiveDist(Java) (via Java API), and thus the triggers and the shadow-table are bypassed.

The clients of LiveDist(Java) receive distribution data in one of three mechanisms:
•	The clients of LiveDist(Java) use HTTP to access the distribution-servers. The delta information is XML encoded (the column-names are configurable), and thus the client can be written in any language and on any platform. A browser can also be a client, and even palm computers and cellular phones can be clients.
•	Alternatively the delta information can be encoded in a more efficient binary encoding, and there is a Java API, which the client can use for receiving distribution data and for decoding the binary data.
•	A local database at the client's machine can be modified, upon reception of distribution info. This effectively enables efficient replication of a database to many copies.

These are the main features of the LiveDist(Java) product:

•	Fully scalable, can grow to support an unlimited number of clients, with no reduction in performance:
Most of the LiveDist(Java) software is deployed as java-web-components (java servlets). In each site, the servlets can be deployed upon any number of machines, utilizing ordinary http load balancing. The machines have symmetrical functionality, and thus the system is linearly scaled, when more machines are added. Additionally, no load is added upon the application servers and the database servers, when more users are added. Thus the product is linearly scalable.

•	Runs on low-cost hardware with no expensive third-party software:
The LiveDist(Java) servlets require only CPU power and memory, and they do not use internal databases or large files. Thus they are not required to run on expensive server machines. They can run on low-cost hardware (PC’s or low-cost Unix-machines), while each machine will support 300-1000 users (depending on load). Additionally, no expensive third party software is required (only standard Java is used). The used container (servlets-engine) can be TomCat (which is free), or any other customer chosen servlets-engine.

•	Pure and portable J2EE product:
The product is a pure Java-software. It is composed from a delta-servlet (responsible for distributing database modifications to clients), a cache-servlet (responsible for downloading snapshots of both cached and un-cached database tables to the clients), and some Java classes, that are required on the client’s side (only when not using XML encoding). The servlets can be deployed into any servlets-container (Tomcat is recommended).

•	Supported databases:

The first release of the product supports only the Oracle database engine. Later releases will support Microsoft-SQL-server and IBM-DB2.

•	Short latency in delta-distribution (1-2 seconds):
Database modifications are distributed, to all connected clients, in less than 2 seconds (when the communication bandwidth is not a bottleneck). There is no degradation in latency when some of the clients are loading snapshots (as long as the communication bandwidth is not too small). Also there is no degradation in latency when more clients are added. When the communication to a client fails, other clients don’t suffer degradation in performance. The product automatically recovers from all possible failures.

•	Fast and concurrent cached-snapshots:
When a new client first joins the system, a snapshot of a portion of the database (for the client’s declared topics of interest) is loaded into the client’s machine. After the loading of the snapshot is completed, normal delta-distribution begins (starting from the point in time when the snapshot has begun). Clients with local database can bypass the snapshot. When a client changes its topic-subscription, a partial-snapshot, for the newly acquired-topics of interest, is downloaded to the client. The partial snapshot is synchronized with normal delta-distribution, to preserve integrity. The cache-servlets contain updated copies of, customer chosen, database tables. When a client needs to download a snapshot, it is already prepared in the memory of the cache-servlets (for cached tables). Thus no load is imposed on the application servers or on the database, to prepare the snapshot. Additionally, when accessing the database to prepare the snapshot, it takes a lot of time. Thus the entire operation is much faster, when the snapshot is already prepared in the memory of the cache-servlets. The cache-servlets can concurrently service multiple snapshot requests from multiple clients. The cache-servlets are also used for downloading un-cached database tables.

•	Small overhead on the application-servers and the database-servers:
When using triggers, the overhead on the database-servers is that of writing into the shadow-table. That overhead depends on the updates-rate. Each modification fires a trigger, which executes an insert into the shadow-table (a small-table without any indexes). Periodically, and in a separate transaction, all rows from the shadow-table are read, packed into a single-buffer, inserted into LiveDist(Java)'s log-table, and removed from the log-table.
When not using triggers, then there is almost no overhead on the database-servers. Just before the end of each transaction, a buffer containing all the modifications done in the current transaction is appended (in a single INSERT operation) into LiveDist(Java)'s log-table.
In both cases, there is almost no overhead on the application-servers. Once every x seconds, all newly appended rows are passed to the delta-servlet. And that is the entire overhead.
Everything else is done inside the delta-servlet and the cache-servlet (they are running on other machines).

•	No locking:
No database rows are ever locked by LiveDist(Java).

•	No 2-phase commit is required:
The passing of delta-information to LiveDist(Java) doesn’t require 2-phase commit transactions (in either mechanism).
Note: using 2-phase commit causes transactions to take longer.
·	
Selective distribution using topic-subscription:
Each row can contain a field, which holds a comma-separated list of topics (alternatively the triggers can call application-supplied routines for calculating the list of topics, which is based on the value of the row's columns). The row is distributed (published) to all the listed topics. The application fills-in the value of this field. The application can update the topic-list (adding new topics to the list, and removing existing topics from the list). Each client, at anytime, is subscribed to a list of topics-of-interest. All rows, which are distributed to at least one of the topics-of-interest, are distributed to the client. It is possible to distribute all rows of a table to the same topic-list, and in that case no column or function is required.

•	New clients can join any time:
The LiveDist(Java) product does not assume a stable network. On the contrary, it assumes that new clients can join the network at any time, and that some known clients can leave the network or become disconnected any time. The product guarantees stable performance to existing clients, even though the topology is changing all the time.

•	Clients are changing their topic-subscription regularly:
It is quite normal for clients to change their topics-of-interest from time to time. The cache-servlet enables these changes to be fast and not traumatic.

•	Supports multi-sites distribution (an hierarchy of sites model): 
LiveDist(Java) supports an hierarchy of sites. Information is passed only once from a mother-site to every child-site. In each site we can run the servlets on any number of machines, and thus each site is separately scalable. The support of this topology enables better utilization of the physical-network.

•	Highly available, guaranty delivery, and automatic recovery from all possible failures:
When deploying the delta-servlets and the cache-servlets on multiple machines, the machines backup each other. When any machine fails, the others continue to function, and the clients continue to be serviced unnoticeably. When a new (or repaired) machine is added to the cluster, it automatically participates in the load balancing. When all LiveDist(Java)'s servers fail, the recovery is automatically done from the log-table in the database.

•	Plug-able security framework:
The LiveDist(Java) product integrates with the application’s security-subsystem, to authenticate clients, encrypt messages between clients and servlets, and to determine which topics are allowed to which clients. LiveDist(Java) doesn’t supply any security solution, instead it interacts with the security solution, that is used by the application.

•	No code generation, fully-configured by XML file:
The product doesn’t need to change, when the data-dictionary is changed. The data-dictionary defines which columns from which tables to distribute. The data-dictionary is defined using an XML configuration-file. No code generation is required when the data-dictionary (schema) is changed.

•	Supports on-the-fly data-dictionary modifications:
When the data-dictionary is changed, the XML configuration-file is updated. The LiveDist(Java) product senses that change (at a single location), and automatically adapts to the new data-dictionary. The system need not be rebooted, or even suspended.

•	Zero administration:
No administrator or operator is required, when using LiveDist(Java). The software components are deployed once, on all participating machines. After deployment, the distribution-servers can be considered as black-boxes, and they require no administration.

Technical requirements:

Database:
•	Oracle 8i or above

On Web-server machines (one for each 300-1000 users):
•	Any Java Servlets container (for example Tomcat).
•	At least 2 Giga-Bytes memory.

On applicaton-server machines (when used):
•	Any Java Application-server (J2EE 1.3 or above).

On client machines:
	Nothing, when client is receiving XML buffers (over HTTP connection).
	Java runtime environment (JRE) 1.4 or above (when client is receiving Java objects).
	No special memory or CPU requirements.