Roedy Green wrote:
> On Mon, 23 Jun 2008 01:54:08 GMT, Roedy Green
> <see_website@[EMAIL PROTECTED]
> wrote, quoted or indirectly quoted
> someone who said :
>
>> Today, when we have RAM coming out our ears we have reverted back to
>> SINGLE buffering. We alternate between processing and waiting for I/O
>> on the same buffer.
>
> I will offer this hypothesis. Double buffering requires multitasking.
> This is perceived as complicated, and hence was left out of early
> personal OSes such as CPM where you might have had only 4 to 64K to
> play with.
>
> In the 1940s to circa1970 , computers primarily did batch processing
> and worked on one problem at a time. Hence the elapsed time for batch
> processing was very im****tant. Double buffering helps this.
>
> Later with mainframes, you had several batch jobs running at once.
> Here the computer could work on some other job while a batch job was
> blocked on i/o. Double buffering became less im****tant.
>
> Today with desktops there is little sequential batch processing.
> Double buffering is for sequential batch processing. Unlike COBOL, C
> and Java don't have the notion built in. Even if the OS offered it, it
> would take a while for it to become sup****ted again.
>
> In my world, I tend to read/write entire files in one i/o operation..
> Double buffering could not beat that.
>
> The reason may be that sequential batch processing is now considered a
> niche application.
>
Just wrote this on my commute today. Only partially tested, and not
tested for speed.
If you're interested, I'd like to see a benchmark between a standard
InputStream, and this DoubleBufferedInputStream in your CSV application.
--- Start DoubleBufferedInputStream.java ---
/*
This code is being released into the public domain. My only request
is that you leave the @[EMAIL PROTECTED]
attribution intact. Adding new
attributions is fine.
It is distributed for use as-is. If there is a problem with it, feel
free to correct.
If you need help correcting it, feel free to seek that from someone
who is willing to help.
Also, feel free to contribute any improvement to the public domain.
Donations also welcome. Check at virtualinfinity.net for contact
info.
*/
package net.virtualinfinity.io;
im****t java.io.InputStream;
im****t java.io.IOException;
im****t java.util.concurrent.locks.*;
/**
* A <code>DoubleBufferedInputStream</code> will buffer a wrapped input
* stream in a separate thread, so that reading can be done concurrently
* with processing. The supplied InputStream must be safe to be
* accessed from a single thread other than the one that created the
* DoubleBufferedInputStream object. Note that this implementation does
* NOT sup****t mark/reset. If you need that functionality, wrap the
* stream with a java.io.BufferedInputStream.
*
* There has been no attempt to make this class thread-safe, other than
* what is necessary to handle the read-ahead behaviour.
*
* @[EMAIL PROTECTED]
Daniel Pitts
* @[EMAIL PROTECTED]
1.0, 06/23/2008
*/
public class DoubleBufferedInputStream extends InputStream {
private final static int defaultBufferSize = 4096;
/**
* Represents one buffer. In practice, two buffers per
* DoubleBufferedInputStream instance can exist at one time, the
* "previously read" buffer, and the "in the process of reading".
*/
private static class Buffer extends InputStream {
private final byte[] data;
private int count;
private int position;
private boolean endOfStream;
private IOException exception;
public Buffer(int bufferSize) {
data = new byte[bufferSize];
}
/**
* Attempts to fill this buffer with data from in. If EOS is
* reached or an exception encountered. These facts are recorded
* for later.
* @[EMAIL PROTECTED]
in the stream to fill from.
*/
public void fill(InputStream in) {
assert !doneFilling() : "Fill called when done filling!";
try {
int read = in.read(data, count, data.length - count);
if (read == -1) {
endOfStream = true;
} else {
count += read;
}
} catch (IOException e) {
exception = e;
}
}
/**
* Test if fill should be called again.
* @[EMAIL PROTECTED]
false if fill can safely and usefully be called
* again.
*/
public boolean doneFilling() {
return count == data.length ||
endOfStream ||
exception != null;
}
/**
* Test if the meaningful data in this buffer has been depleted.
* @[EMAIL PROTECTED]
true if a new buffer should be retrieved.
*/
public boolean depleted() {
return !lastBuffer() && position >= count;
}
@[EMAIL PROTECTED]
public int read() throws IOException {
if (checkEndOfBuffer()) return -1;
return data[position++];
}
@[EMAIL PROTECTED]
public int read(byte[] b, int off, int len) throws IOException {
if (checkEndOfBuffer()) return -1;
final int toCopy = Math.min(available(), len);
System.arraycopy(data, position, b, off, toCopy);
position += toCopy;
return toCopy;
}
/**
* Check for end-of-buffer conditions and handle appropriately.
* @[EMAIL PROTECTED]
true if were at the end of the stream.
* @[EMAIL PROTECTED]
IOException at end of buffer when an exception
* occurs during fill()
* @[EMAIL PROTECTED]
IllegalStateException when end of buffer is reached
* and no other state is indicated.
*/
private boolean checkEndOfBuffer() throws IOException {
if (position >= count) {
if (exception != null) {
throw exception;
}
if (endOfStream) {
return true;
}
throw new IllegalStateException("Buffer depleted.");
}
return false;
}
/**
* @[EMAIL PROTECTED]
false if there could be more buffers to follow.
*/
private boolean lastBuffer() {
return endOfStream || exception != null;
}
@[EMAIL PROTECTED]
public int available() {
return count - position;
}
}
// The current buffer, or null if not yet filled. Access guarded by
// bufferLock.
private Buffer buffer;
// Flag to short-circuit the filling routine and return with any
// available bytes. Write access guarded by bufferLock.
private volatile boolean waitingForBuffer;
// Flag to indicate stream is closed.
private volatile boolean closed;
// Lock to guard buffer.
private final Lock bufferLock = new ReentrantLock();
// Condition waiting for the predicate that buffer is not null.
private final Condition bufferSet = bufferLock.newCondition();
// Condition waiting for the predicate that buffer is null.
private final Condition bufferEmpty = bufferLock.newCondition();
// The maximum buffer size.
private final int bufferSize;
// The wrapped input stream.
private InputStream in;
// The thread that will read a buffer.
private final Thread bufferReader = new Thread() {
public void run() {
Buffer buffer;
do {
buffer = new Buffer(bufferSize);
do {
buffer.fill(in);
} while (!buffer.doneFilling() && !waitingForBuffer);
putBuffer(buffer);
} while (!buffer.lastBuffer() && !closed);
}
};
/**
* Waits for the current buffer to be empty, and the new buffer.
* @[EMAIL PROTECTED]
nextBuffer the new buffer.
*/
private void putBuffer(Buffer nextBuffer) {
bufferLock.lock();
try {
while (buffer != null) {
bufferEmpty.awaitUninterruptibly();
}
buffer = nextBuffer;
waitingForBuffer = false;
bufferSet.signalAll();
} finally{
bufferLock.unlock();
}
}
/**
* Creates a <code>DoubleBufferedInputStream</code>
* and saves its argument, the input stream
* <code>in</code>, for later use.
*
* @[EMAIL PROTECTED]
in the underlying input stream.
*/
public DoubleBufferedInputStream(InputStream in) {
this (in, defaultBufferSize);
}
/**
* <code>DoubleBufferedInputStream</code> and saves its argument,
* the input stream <code>in</code>, for later use. Sets the
* maximum size of each buffer. Note that there can be two buffers,
so
* the amount of memory used is at least 2*bufferSize.
* @[EMAIL PROTECTED]
in the underlying input stream.
* @[EMAIL PROTECTED]
bufferSize the size for each buffer.
*/
public DoubleBufferedInputStream(InputStream in, int bufferSize) {
this.bufferSize = bufferSize;
this.in = in;
}
/**
* Check to make sure stream is not close, and then get the next
* available buffer.
* @[EMAIL PROTECTED]
the available buffer.
* @[EMAIL PROTECTED]
IOException if the stream is closed, or the current
* thread is interrupted.
*/
private Buffer getBufferIfOpen() throws IOException {
if (closed) {
throw new IOException("Stream closed");
}
bufferLock.lock();
try {
startReaderIfNecessary();
emptyDepletedBuffer();
makeBufferAvailable();
return buffer;
} finally {
bufferLock.unlock();
}
}
/**
* Make sure that the current buffer is available, asking the
* reader thread to short-curcuit its work if necessary.
*
* A call to this method must be guarded by bufferLock.
* @[EMAIL PROTECTED]
IOException if the current thread is interrupted while
* waiting.
*/
private void makeBufferAvailable() throws IOException {
while (buffer == null) {
waitingForBuffer = true;
try {
bufferSet.await();
} catch (InterruptedException e) {
throw
new IOException("Interrupted while waiting for buffer.",
e);
}
}
}
/**
* Ensure that the bufferReader thread has actually been started.
* If it hasn't been started, this method pre-fills one buffer
* before starting the reader.
*
* A call to this method must be guarded by bufferLock.
*/
private void startReaderIfNecessary() {
if (bufferReader.getState() == Thread.State.NEW) {
buffer = new Buffer(bufferSize);
buffer.fill(in);
bufferReader.start();
}
}
/**
* Empty a buffer if it has been depleted.
*
* A call to this method must be guarded by bufferLock.
*/
private void emptyDepletedBuffer() {
if (buffer != null && buffer.depleted()) {
buffer = null;
bufferEmpty.signalAll();
}
}
@[EMAIL PROTECTED]
public int read() throws IOException {
int result = getBufferIfOpen().read();
cleanBuffer();
return result;
}
/**
* Convenience method that cleans up a depleted buffer.
*/
private void cleanBuffer() {
bufferLock.lock();
try {
emptyDepletedBuffer();
} finally{
bufferLock.unlock();
}
}
@[EMAIL PROTECTED]
public int read(byte b[], int off, int len) throws IOException {
int result = getBufferIfOpen().read(b, off, len);
cleanBuffer();
return result;
}
@[EMAIL PROTECTED]
public int available() throws IOException {
if (bufferLock.tryLock()) {
try {
if (buffer != null) {
return buffer.available();
}
} finally{
bufferLock.unlock();
}
}
return 0;
}
@[EMAIL PROTECTED]
public void close() throws IOException {
closed = true;
in.close();
}
}
--- End DoubleBufferedInputStream.java ---
--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
|
