AfflictionNetworks/depot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Moved wiki docs into main project.

Browse Source 
Why have them separate? Github allows reading docs right in the main tree.
This commit is contained in:
Michael Bayne
2014-06-25 16:15:18 -07:00
parent 3db7b4445b
commit 634651fa5a
8 changed files with 695 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Caching.md
+47
View File
@@ -0,0 +1,47 @@
_Overview of Caching._
Records are cached by primary key. Lookups by primary key first check the cache and on a miss, load
the record from the database and insert it into the cache.
Queries for records that have primary keys are automatically split into phases:
* A query is made to the database for the primary keys of all rows that match the query.
* Any records in the cache are obtained from the cache.
* All remaining records are loaded by primary key in a single additional query and placed into
the cache.
For deletions using a `Where` clause, first the primary keys that match the deletion clause are
loaded, then those records are deleted from the database and the cache using their primary key.
Decomposition for updates using a `Where` clause is not yet implemented, but a fallback mechanism
to invalidate the cache manually is provided for those cases.
If one already has the primary keys for the records they desire, it is possible to avoid the first
phase of the decomposed query using `loadAll()` instead of `findAll()`:
```java
@Entity
public class MemberNameRecord extends PersistentRecord
{
/** This member's unique id. */
@Id public int memberId;
/** The name by which this member is known. */
public String name;
}
/**
* Looks up members' names by id.
*/
public Map<Integer, MemberName> loadMemberNames (final Set<Integer> memberIds)
{
final Map<Integer, MemberName> names = Maps.newHashMap();
for (MemberNameRecord name : loadAll(MemberNameRecord.class, memberIds)) {
names.put(name.memberId, name.name);
}
return names;
}
```
This will efficiently fetch all records that it can from the cache and then load and cache any
remaining records.
docs/ComputedRecords.md
+66
View File
@@ -0,0 +1,66 @@
_Overview of Computed Records._
Note: computed records are largely deprecated in favor of ad-hoc queries or the use of
`selectInto`. See [the example queries page](ExampleQueries) for examples of such use.
## Computed Records
You can easily define record with computed fields or records that represent a join across multiple
tables (see below). It is very easy to select a subset of a record's fields:
```java
@Entity @Computed(shadowOf=PersonRecord.class)
public PersonNameRecord extends PersistentRecord
{
public int personId;
public String name;
}
List<PersonNameRecord> allNames = findAll(PersonNameRecord.class);
List<PersonNameRecord> youngNames = findAll(
PersonNameRecord.class, new Where(PersistentRecord.AGE.lessEq(18)));
```
You can also define computed records that calculate information:
```java
@Entity @Computed
public CountRecord extends PersistentRecord
{
@Computed(fieldDefinition="count(*)")
public int count;
}
int personCount = load(CountRecord.class, new FromOverride(PersonRecord.class)).count;
// or if you want to be less general purpose
@Entity @Computed(shadowOf=PersonRecord.class)
public PersonCountRecord extends PersistentRecord
{
@Computed(fieldDefinition="count(*)")
public int count;
}
int personCount = load(PersonCountRecord.class).count;
// or something more specific
@Entity @Computed(shadowOf=PersonRecord.class)
public PersonAvgAgeRecord extends PersistentRecord
{
@Computed(fieldDefinition="avg(*)")
public float averageAge;
}
float averageAge = load(PersonAvgAgeRecord.class).averageAge;
```
The reader may be alarmed to notice some hard-coded SQL in those classes. One is advised to stick
to very standard SQL in cases like this to avoid introducing portability problems, but we didn't
think it was worth the trouble to try to model these simple and mostly standard operations in a
more complex way. YMMV.
## Joins with Computed Records
TBD
docs/Configuration.md
+207
View File
@@ -0,0 +1,207 @@
_Configuration and Dependencies._
## JVM Version Requirement
Depot currently requires JDK 1.6 or greater.
### Integrate with Ivy or Maven
Depot is published to the Maven Central repository and can be added as a dependency using the
following configuration: `com.samskivert:depot:1.6.4`
This will automatically include the Google Guava and samskivert dependencies. You can add Ehcache
as well via `net.sf.ehcache:ehcache:1.6.0` (or a newer version, if available).
## Manually Adding Dependencies
Depot depends on a small number of external libraries:
* Google Guava - http://code.google.com/p/guava-libraries/
* samskivert - http://code.google.com/p/samskivert/
* Ehcache (optional) - http://ehcache.sourceforge.net/
Depot also requires a JDBC driver for the database with which you plan to operate. Depot currently
supports three database backends:
* Postgresql - http://jdbc.postgresql.org/
* MySQL - http://www.mysql.com/products/connector/j/
* HSQLDB - http://hsqldb.org/ (useful for unit testing)
## Configuration
The two main components that require configuration are the JDBC connection provider and the cache
implementation.
### StaticConnectionProvider
For testing and other simple systems that don't require connection pooling, the
`StaticConnectionProvider` is a simple way to provide JDBC connections to Depot. It is used as
follows (this example uses MySQL):
```java
Properties props = new Properties();
// you'd probably load these properties from a file, but for the purposes
// of this example, we'll set them directly in the code
props.setProperty("default.driver", "com.mysql.jdbc.Driver");
props.setProperty("default.url", "jdbc:mysql://localhost:3306/dbname");
props.setProperty("default.username", "username");
props.setProperty("default.password", "password");
PersistenceContext perCtx = new PersistenceContext(
"default", new StaticConnectionProvider(props), null);
```
### DataSourceConnectionProvider
Production systems are more likely to use a JDBC `DataSource` to obtain their connections as those
provide connection pooling and integrate with JNDI and such. The only non-obvious aspect of
configuring Depot with a `DataSource` is that you can provide two datasources: one for read-only
connections and one for read-write connections. Depot will obtain connections from the appropriate
source depending on whether or not it is doing a query that is safe to be performed against a
read-only mirror of your data or if it's doing a query that must talk to a database master.
What follows is a simple example of manually creating and configuring a Postgresql pooling
`DataSource`:
```java
PoolingDataSource readSource = new PoolingDataSource();
readSource.setDataSourceName("MyReadSource");
readSource.setServerName("myReadOnlyServerHost");
readSource.setDatabaseName("myDatabaseName");
readSource.setPortNumber(5432);
readSource.setUser("myUsername");
readSource.setPassword("myPassword");
readSource.setMaxConnections(4); // tune to your applications needs
PoolingDataSource writeSource = new PoolingDataSource();
writeSource.setDataSourceName("MyWriteSource");
writeSource.setServerName("myReadWriteServerHost");
writeSource.setDatabaseName("myDatabaseName");
writeSource.setPortNumber(5432);
writeSource.setUser("myUsername");
writeSource.setPassword("myPassword");
writeSource.setMaxConnections(1); // tune to your applications needs
PersistenceContext perCtx = new PersistenceContext(
"notused", new DataSourceConnectionProvider("jdbc:postgresql", readSource, writeSource), null);
```
See the note below on lifecycle management.
### EHCacheAdapter
You may have noticed the second argument to the `PersistenceContext` constructor in the above
examples was always null. That is where the `CacheAdapter` is provided. By passing null, Depot will
not use caching. Depot comes with integration for Ehcache and implementing additional cache
integrations is as simple as implementing the `CacheAdapter` interface and supplying an instance to
the `PersistenceContext` constructor.
The following example assumes that you have an `ehcache.xml` configuration file in your classpath.
There are other ways to configure Ehcache but we'll leave that explanation to their documentation.
```java
CacheManager cacheMgr = CacheManager.getInstance();
ConnectionProvider conProv = // ...
PersistenceContext perCtx = new PersistenceContext("ident", conProv, new EHCacheAdapter(cacheMgr));
```
See the note below on lifecycle management.
### PersistenceContext Lifecycle
When your application is shutting down it should shutdown its `PersistenceContext`. However, to
avoid integration headaches, Depot does not take responsibility for shutting down certain of its
dependencies as those may be used by other parts of your application and you may wish to shut Depot
down independently of these other components.
#### ConnectionProvider
Depot will shutdown its connection provider when the `PersistenceContext` is shutdown, however the
two `ConnectionProvider` implementations have different shutdown behavior as explained below.
* `StaticConnectionProvider` will close all JDBC `Connection` instances it has created when it is
shutdown. If you are using Depot with `StaticConnectionProvider` you can simply shutdown your
`PersistenceContext` and you're done.
* `DataSourceConnectionProvider` will not shutdown its underlying `DataSource` implementations
(indeed there is no API for doing so). As long as no queries are executing at the time that
`PersistenceContext` is shutdown, then all JDBC `Connection` instances will have been closed
and returned to the `DataSource` connection pool, so the application can shutdown its data
sources in whatever way is appropriate.
#### CacheAdapter
Depot will shutdown its `CacheAdapter` when the `PersistenceContext` is shutdown, however the
`CacheAdapter` implementation is free to do nothing in its `shutdown` call.
* `EHCacheAdapter` does not shutdown its underlying `CacheManager` when it is shutdown to avoid
conflict with other aspects of the application that may use Ehcache. Thus the application is
responsible for shutting down the `CacheManager` itself when it is known to no longer be
needed.
## Injection
We use Guice around these parts for dependency injection. Using injection allows you to inject the
`PersistenceContext` into your repository implementations:
```java
@Singleton
public class FooRepository extends DepotRepository {
@Inject public FooRepository (PersistenceContext perCtx) {
super(perCtx);
}
}
```
and then inject your repositories wherever you need them.
We also find the following pattern to be very effective:
```java
public class FooModule extends AbstractModule {
@Override protected void configure () {
super.configure();
// depot dependencies (we will initialize this persistence context later when the
// server is ready to do database operations; not initializing it now ensures that no
// one sneaks any database manipulations into the dependency resolution phase)
bind(PersistenceContext.class).toInstance(new PersistenceContext());
}
}
public class WhateverHandlesAppServerLifecycle {
public void init () {
// initialize our persistence context
ConnectionProvider conProv = // ...
_perCtx.init("ident", conProv, new EHCacheAdapter(_cacheMgr));
// initialize our depot repositories; this runs all of our schema and data migrations
_perCtx.initializeRepositories(true);
}
public void shutdown () {
_perCtx.shutdown();
_cacheMgr.shutdown();
}
@Inject protected PersistenceContext _perCtx;
protected CacheManager _cacheMgr = CacheManager.getInstance();
}
```
One major benefit to the approach of delaying the initialization of your persistence context until
the dependency resolution phase is complete is to ensure that no code accidentally (or
intentionally) starts talking to the database during that phase. You almost certainly want to
resolve all of your injection dependencies and then before you turn your application server loose,
call `initializeRepositories` to cause all of your schema and data migrations to be run (or to fail
and abort the initialization of your application).
If you don't call `initializeRepositories` then Depot will lazily initialize each
`PersistentRecord` class when it is first accessed and run any schema migrations for that record.
Data migrations will be disabled if you choose this lazily initialized approach.
Another note on `initializeRepositories` is that this will initialize all repositories that have
been constructed with the supplied `PersistenceContext` up to that point. Any repositories
constructed after `initializeRepositories` has been called will be initialized at that time
(running schema and data migrations for their records) and a warning will be generated to alert you
to this undesirable behavior. Again, experience has shown that you generally want to get all of
your schema and data migrations out of the way immediately and before the application server starts
normal operation.
docs/ExampleQueries.md
+152
View File
@@ -0,0 +1,152 @@
_Describes various example queries._
Depot queries are constructed using a builder-pattern.
## Whole record queries
A basic query to select all rows from a table looks like so:
```java
from(PersonRecord.class).select();
```
Note that the above query and the others in these examples will use the `PersonRecord` from the
SimpleCodeExample page.
Various query clauses may be added to the basic query above to do filtering, ordering and the like.
Here's a query with a simple where clause:
```java
// selects all records with age <= 25
from(PersonRecord.class).where(PersonRecord.AGE.lessEq(25)).select();
```
One can also order and limit the results:
```java
// selects the first ten records in ascending alphabetic order on name
from(PersonRecord.class).ascending(PersonRecord.NAME).limit(10).select();
```
More complex orderings are also possible:
```java
OrderBy order = OrderBy.ascending(PersonRecord.NAME).thenDescending(PersonRecord.AGE);
from(PersonRecord.class).orderBy(order).select();
```
## Ad-hoc queries
Instead of selecting whole rows from the database, one can select individual columns, or the
results of aggregate and other functions. Here's a simple projection:
```java
List<Tuple2<Integer,String>> results =
from(PersonRecord.class).select(PersonRecord.ID, PersonRecord.NAME);
```
Depot annotates the `ColumnExp` constants generated in your `PersistentRecord` classes with their
type so that queries like the above can be done in a type-safe manner. `Tuple` classes are provided
up to `Tuple5` for such ad-hoc queries.
As an alternative to a `Tuple` class, you can use a type-safe builder to receive the results of
your query like so:
```java
public class IdName {
public static Builder2<IdName, Integer, String> IDNAME_BUILDER =
new Builder2<IdName, Integer, String>() {
public IdName build (Integer a, String b) {
return new IdName(a, b);
}
};
public int id;
public String name;
public IdName (int id, String name) {
this.id = id;
this.name = name;
}
}
List<IdName> results =
from(PersonRecord.class).select(IDNAME_BUILDER, PersonRecord.ID, PersonRecord.NAME);
```
Such queries will result in compile time error if the types of the columns do not match the types
expected by the builder. The `BuilderN` interfaces are also only available up to arity-5.
For situations where type-safety is not a major concern, and for cases where you wish to select
more than five columns, you can use `selectInto` which uses reflection to construct results:
```java
public class NameCount {
public String name;
public int count;
public NameCount (String name, int count) {
this.name = name;
this.count = count;
}
}
List<NameCount> results =
from(PersonRecord.class).groupBy(PersonRecord.NAME).selectInto(
NameCount.class, PersonRecord.NAME, Funcs.countStar());
```
Note that the class supplied to the `selectInto` method must have exactly one public constructor,
and the arguments to that constructor must match __in order__, the columns specified in the
`selectInto` call. The types of the selected columns (or expressions) must be convertible to the
type needed by the constructor (which means they will be widened or unboxed, but not converted from
`float` to `int` or other non-automatic conversions). These requirements are unfortunately not
checkable at compile time, and instead result in a runtime error when violated. Fortunately,
testing tends to catch any such errors before they make it into the wild.
## Count queries
The `selectCount` method exists for when you wish to simply select the count of rows that match
your query. For example:
```java
int youngins = from(PersonRecord.class).where(PersonRecord.AGE.lessEq(12)).selectCount();
```
You may also wish to group by certain columns and select the counts of rows that match each group.
This is done with an ad-hoc query:
```java
List<Tuple2<String,Integer>> results =
from(PersonRecord.class).groupBy(PersonRecord.NAME).
select(PersonRecord.NAME, Funcs.countStar());
```
## Other functions
A variety of other functions are defined in
[Funcs](http://depot.googlecode.com/svn/apidocs/com/samskivert/depot/Funcs.html),
[StringFuncs](http://depot.googlecode.com/svn/apidocs/com/samskivert/depot/StringFuncs.html),
[DateFuncs](http://depot.googlecode.com/svn/apidocs/com/samskivert/depot/DateFuncs.html), and
[MathFuncs](http://depot.googlecode.com/svn/apidocs/com/samskivert/depot/MathFuncs.html). These can
be used in queries, like so:
```java
List<PersonRecord> eldest =
from(PersonRecord.class).where(PersonRecord.AGE.eq(Funcs.max(PersonRecord.AGE))).select();
```
And you can select the value of a function in an ad-hoc query:
```java
// note that load() can be used for selections that will only ever return one row
Number maxAge = from(PersonRecord.class).load(Funcs.max(PersonRecord.AGE));
// alternatively
List<Number> maxAge = from(PersonRecord.class).load(Funcs.max(PersonRecord.AGE));
assert(maxAge.size() == 1);
// here's a more complex (if somewhat nonsensical) query that groups people by the first
// letter of their name and selects the sum of all ages of the people in those groups
SQLExpression<String> firstLetter = StringFuncs.substring(PersonRecord.NAME, 0, 1);
List<String,Integer> results = from(PersonRecord.class).groupBy(firstLetter).
select(firstLetter, Funcs.sum(PersonRecord.AGE));
```
docs/README.md
+15
View File
@@ -0,0 +1,15 @@
_Overview of the Depot documentation._
## Docs
- [Simple code example](wiki/SimpleCodeExample)
- [Configuring and integrating Depot](wiki/Configuration)
- [Example queries](wiki/ExampleQueries)
- [Caching](wiki/Caching)
- [Computed records](wiki/ComputedRecords)
- [Schema migration](wiki/SchemaMigration)
## Google Group
If you have further questions or suggestions, please post to the
[OOO Libs Google Group](http://groups.google.com/group/ooo-libs).
docs/SchemaMigration.md
+50
View File
@@ -0,0 +1,50 @@
_Overview of Schema and Data Migration._
## Schema Migration
Depot makes simple schema migrations extremely simple and complex schema migrations pretty easy.
* Automatic schema migration: adding a new column to a persistent record is as simple as adding
the new field to the POJO and incrementing the `SCHEMA_VERSION_NUMBER` constant.
* Assisted schema migration: dropping, renaming and retyping columns is very easy, and more
sophisticated custom migrations can also be easily incorporated into Depot's schema migration
system.
* Data migration: migrations that do not change record schemas but manipulate their data can also
be registered and Depot will ensure that they run successfully and only once.
* Distributed migration coordination: Depot is designed so that you can bring up a dozen
application servers on a dozen machines and during their initialization they will coordinate
(through the database) which server will handle each migration and the other servers will block
any database access until those migrations have successfully completed.
The addition of columns is automatic. Dropping, renaming and retyping are very simple:
```java
public DecorRepository (PersistenceContext ctx)
{
super(ctx);
ctx.registerMigration(DecorRecord.class,
new SchemaMigration.Rename(17004, "scale", DecorRecord.ACTOR_SCALE));
ctx.registerMigration(DecorRecord.class, new SchemaMigration.Drop(17004, "offsetX"));
ctx.registerMigration(DecorRecord.class, new SchemaMigration.Drop(17004, "offsetY"));
ctx.registerMigration(DecorRecord.class,
new SchemaMigration.Retype(17004, DecorRecord.FURNI_SCALE));
}
```
More complex migrations are also possible, one has to take care if they wish to preserve database
agnosticism:
```java
ctx.registerMigration(FooRecord.class, new SchemaMigration(42) {
@Override
public Integer invoke (Connection conn, DatabaseLiaiason liaison) throws SQLException {
// go crazy with your raw JDBC connection or use the DatabaseLiaison to
// help you do things in a database agnostic way
}
});
```
## Data Migration
TBD
docs/SimpleCodeExample.md
+153
View File
@@ -0,0 +1,153 @@
_A simple Depot code example._
Here's a simple example to give you a quick overview of what code using Depot looks like.
Start by defining a persistent record, this maps to a database table:
```java
@Entity
public class PersonRecord extends PersistentRecord
{
/** Increment this value if you change this record's schema. */
public static final int SCHEMA_VERSION = 1;
/** A unique identifier for this record. Automatically filled in at row insertion time. */
@Id @GeneratedValue(strategy=GenerationType.IDENTITY)
public int personId;
/** This person's name. Note: one difference between EJB3 and Depot is that columns are
* non-nullable by default. */
@Column(length=100)
public String name;
/** This person's age. */
public int age;
}
```
Then you run a simple Ant task or Maven plugin that adds some unfortunately non-POJO boilerplate
code to your record class, but this code allows you to talk about your record in queries in a way
that the compiler can check which is a huge win.
If and when Java adds field literals, Depot will absolutely take advantage of them and eliminate
this undesirable boilerplate.
```java
@Entity
public class PersonRecord extends PersistentRecord
{
// AUTO-GENERATED: FIELDS START
public static final Class<PersonRecord> _R = PersonRecord.class;
public static final ColumnExp<Integer> PERSON_ID = colexp(_R, "personId");
public static final ColumnExp<String> NAME = colexp(_R, "name");
public static final ColumnExp<Integer> AGE = colexp(_R, "age");
// AUTO-GENERATED: FIELDS END
/** Increment this value if you change this record's schema. */
public static final int SCHEMA_VERSION = 1;
/** A unique identifier for this record. Automatically filled in at row insertion time. */
@Id @GeneratedValue(strategy=GenerationType.IDENTITY)
public int personId;
/** This person's name. Note: one difference between EJB3 and Depot is that columns are
* non-nullable by default. */
@Column(length=100)
public String name;
/** This person's age. */
public int age;
// AUTO-GENERATED: METHODS START
/**
* Create and return a primary {@link Key} to identify a {@link PersonRecord}
* with the supplied key values.
*/
public static Key<PersonRecord> getKey (int personId)
{
return newKey(_R, personId);
}
// AUTO-GENERATED: METHODS END
}
```
Next you define a repository class which will provide an application-specific persistence API. We
highly recommend preserving this boundary and having all Depot code inside repository classes and
only pass persistent record classes outside to your application.
```java
public class PersonRepository extends DepotRepository
{
/**
* Creates this repository and provides it with a context via which it will obtain JDBC
* connections.
*/
public PersonRepository (PersistenceContext ctx)
{
super(ctx);
}
/**
* Loads and returns the person with the specified id, or null if no person exists with that
* id.
*/
public PersonRecord loadPerson (int personId)
{
return load(PersonRecord.getKey(personId));
}
/**
* Loads records for all people with an age less than or equal to the specified maximum.
*/
public List<PersonRecord> loadYoungPeople (int maxAge)
{
return from(PersonRecord.class).where(PersonRecord.AGE.lessEq(maxAge)).select();
}
/**
* Loads the names of all people in the repository.
*/
public Set<String> loadNames ()
{
Set<String> names = new HashSet<String>();
names.addAll(from(PersonRecord.class).select(PersonRecord.NAME));
return names;
}
/**
* Inserts a newly created person record into the database. If record.personId is non-zero (or
* non-null in the case of a non-primitive integer field) an exception will be thrown.
*/
public void insertPerson (PersonRecord record)
{
insert(record);
}
/**
* Updates a person record. If record.personId is zero (or null in the case of a non-primitive
* integer field) an exception will be thrown.
*/
public void updatePerson (PersonRecord record)
{
update(record);
}
/**
* Updates a person record. If record.personId is zero (or null in the case of a non-primitive
* integer field) a new row will be created for this record, if not the matching row will be
* updated.
*/
public void storePerson (PersonRecord record)
{
store(record);
}
@Override // from DepotRepository
protected void getManagedRecords (Set<Class<? extends PersistentRecord>> classes)
{
classes.add(PersonRecord.class);
}
}
```
See the [example queries](ExampleQueries) page for examples of other kinds of queries.