Improve dataflow api and configuration

[tuan08]

1. Currently when a task try to read the data from a stream, if it cannot read more the data from the stream, it consider the task is finished. This strategy is
   correct for stream like hdfs , S3. But it is not correct for stream like kafka , since the data can be avaiable after a period of time. How should we handle this situation.

@smorin @KingOfPoptart

1. In order to get the best performance , we should have an executor dedicate to a stream or task. This should avoid the task switch. But we need to have enough machine and thread to dedicate an executor to a thread or we switch the task when it complete only. Should we update the api, configuration to support different way of switching the task.

[smorin]

Let's discuss on our next call

On Jun 3, 2015, at 01:46, Tuan Nguyen notifications@github.com wrote:

Currently when a task try to read the data from a stream, if it cannot read more the data from the stream, it consider the task is finished. This strategy is correct for stream like hdfs , S3. But it is not correct for stream like kafka , since the data can be avaiable after a period of time. How should we handle this situation.
@smorin @KingOfPoptart

In order to get the best performance , we should have an executor dedicate to a stream or task. This should avoid the task switch. But we need to have enough machine and thread to dedicate an executor to a thread or we switch the task when it complete only. Should we update the api, configuration to support different way of switching the task.
—
Reply to this email directly or view it on GitHub.