Anatomy of File Write:
1. The client creates the file by calling create() on DistributedFileSystem.
2. DistributedFileSystem makes an RPC call to the namenode to create a new file in the filesystem’s namespace, with no blocks associated with it.
**The namenode performs various checks to make sure the file doesn’t already exist, and that the client has the right permissions to create the file. If these checks pass, the namenode makes a record of the new file; otherwise, file creation fails and the client is thrown an IOException.
**The DistributedFileSystem returns an FSDataOutputStream for the client to start writing data to.
3. As the client writes data, DFSOutputStream splits it into packets, which it writes to an internal queue, called the data queue.
**The data queue is consumed by the Data Streamer, whose responsibility it is to ask the namenode to allocate new blocks by picking a list of suitable datanodes to store the replicas.
4. The list of datanodes forms a pipeline—we’ll assume the replication level is three, so there are three nodes in the pipeline. -> pipelined write
5. DFSOutputStream also maintains an internal queue of packets that are waiting to be acknowledged by datanodes, called the ack queue. A packet is removed from the ack queue only when it has been acknowledged by all the datanodes in the pipeline (Ack's from the 2nd and 3rd datanodes will be given to Datanode1 and Datanode1 will submit the Ack to DFSOutputStream).
6. When the client has finished writing data, it calls close() on the stream.This action flushes all the remaining packets to the datanode pipeline and waits for acknowledgments before contacting the namenode to signal that the file is complete.
7. The namenode already knows which blocks the file is made up of (via Data
Streamer asking for block allocations), so it only has to wait for blocks to be minimally replicated before returning successfully.
If a datanode fails while data is being written to it, then the following actions are taken,which are transparent to the client writing the data.
1) First the pipeline is closed, and any packets in the ack queue are added to the front of the data queue so that datanodes that are downstream from the failed node will not miss any packets.
2) The current block on the good datanodes is given a new identity, which is communicated to the namenode,so that the partial block on the failed datanode will be deleted if the failed datanode recovers later on.
3) The failed datanode is removed from the pipeline and the remainder of the block’s data is written to the two good datanodes in the pipeline. The namenode notices that the block is under-replicated, and it arranges for a further replica
to be created on another node. Subsequent blocks are then treated as normal.
1. The client creates the file by calling create() on DistributedFileSystem.
2. DistributedFileSystem makes an RPC call to the namenode to create a new file in the filesystem’s namespace, with no blocks associated with it.
**The namenode performs various checks to make sure the file doesn’t already exist, and that the client has the right permissions to create the file. If these checks pass, the namenode makes a record of the new file; otherwise, file creation fails and the client is thrown an IOException.
**The DistributedFileSystem returns an FSDataOutputStream for the client to start writing data to.
3. As the client writes data, DFSOutputStream splits it into packets, which it writes to an internal queue, called the data queue.
**The data queue is consumed by the Data Streamer, whose responsibility it is to ask the namenode to allocate new blocks by picking a list of suitable datanodes to store the replicas.
4. The list of datanodes forms a pipeline—we’ll assume the replication level is three, so there are three nodes in the pipeline. -> pipelined write
5. DFSOutputStream also maintains an internal queue of packets that are waiting to be acknowledged by datanodes, called the ack queue. A packet is removed from the ack queue only when it has been acknowledged by all the datanodes in the pipeline (Ack's from the 2nd and 3rd datanodes will be given to Datanode1 and Datanode1 will submit the Ack to DFSOutputStream).
6. When the client has finished writing data, it calls close() on the stream.This action flushes all the remaining packets to the datanode pipeline and waits for acknowledgments before contacting the namenode to signal that the file is complete.
7. The namenode already knows which blocks the file is made up of (via Data
Streamer asking for block allocations), so it only has to wait for blocks to be minimally replicated before returning successfully.
If a datanode fails while data is being written to it, then the following actions are taken,which are transparent to the client writing the data.
1) First the pipeline is closed, and any packets in the ack queue are added to the front of the data queue so that datanodes that are downstream from the failed node will not miss any packets.
2) The current block on the good datanodes is given a new identity, which is communicated to the namenode,so that the partial block on the failed datanode will be deleted if the failed datanode recovers later on.
3) The failed datanode is removed from the pipeline and the remainder of the block’s data is written to the two good datanodes in the pipeline. The namenode notices that the block is under-replicated, and it arranges for a further replica
to be created on another node. Subsequent blocks are then treated as normal.
No comments:
Post a Comment