After reading two good papers on this subject (previously mentioned OpenGL Insights chapter and Order-Independent Transparency in DirectX11), I am here to summarize what I learned.
The core idea behind order independent transparency is that we need to store all the transparent fragments somewhere. ALL the transparent fragments. This means turning off the depth test to allow every fragment to be processed. With all this data, the first approach is to keep a per-pixel linked list. The second approach is to pack the data sequentially and store offset pointers for each pixel.
I decided that I would implement the linked list approach first, as it seems like a nicer algorithm. Essentially each pixel cell needs to store a pointer to the head of its linked list. This indexes into a global memory array which stores all the fragments that have been processed. Besides storing color and depth, each member of the global memory array also stores an index to the next node in the linked list.
By the end of processing, each pixel on the screen has a corresponding linked list. I drew this diagram to explain how the algorithm works. It requires an atomic counter for node_count (to avoid a slew of memory conflict issues), a 2D uint texture for heads, a 1D vec4 buffer texture for color data, a 1D uint buffer texture for next. The missing value is depth, which may need its own similar texture. The image load, store, and exchange functions of OpenGL 4 are suitable for writing/reading/exchanging data from these textures.
When this procedure is over, each fragment loops over its linked list, sorts the fragments in back to front order, and blends them with the current pixel color (which may contain colors from the opaque objects in the scene).
As for sorting, I plan on using a bubble sort. This is a very easy algorithm to write for a fragment shader and is suitable for the smallish input size (maybe maximum of 30 overlapping transparent fragments?).
There are tons of optimization details that I haven't mentioned, and probably won't implement until I get the basic thing working.
I'm hoping that I'll have something more to show by tomorrow.