Skip to content

Flusher step 2: CPU and NIC options #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
e-ago wants to merge 7 commits into
base: devel
Choose a base branch
from
Open

Flusher step 2: CPU and NIC options #32

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
71ec1fe
Initial commit flusher
e-ago Jan 16, 2018
82b11e1
flusher inlcude, nvtx removed
e-ago Jan 16, 2018
31229e2
CU_STREAM_WAIT_VALUE_FLUSH no longer supported in CUDA 9.x. Closes #31
e-ago Jan 17, 2018
56a4e7a
Commented dump wait params useful for debug
e-ago Jan 17, 2018
b00b350
CU_STREAM_WAIT_VALUE_FLUSH code restored commented as a reminder
e-ago Jan 17, 2018
f3c4cf1
flusher env vars merged in a single one. flusher types enum instead o…
e-ago Feb 5, 2018
d058537
enum gds_flusher_type renamed
e-ago Feb 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ libgdsyncincludedir = $(includedir)/gdsync
libgdsyncinclude_HEADERS = include/gdsync/core.h include/gdsync/device.cuh include/gdsync/mlx5.h include/gdsync/tools.h

src_libgdsync_la_CFLAGS = $(AM_CFLAGS)
src_libgdsync_la_SOURCES = src/gdsync.cpp src/memmgr.cpp src/mem.cpp src/objs.cpp src/apis.cpp src/mlx5.cpp include/gdsync.h
src_libgdsync_la_SOURCES = src/gdsync.cpp src/memmgr.cpp src/mem.cpp src/objs.cpp src/apis.cpp src/mlx5.cpp src/flusher.cpp include/gdsync.h
src_libgdsync_la_LDFLAGS = -version-info 2:0:1

noinst_HEADERS = src/mem.hpp src/memmgr.hpp src/objs.hpp src/rangeset.hpp src/utils.hpp src/archutils.h src/mlnxutils.h
Expand Down
13 changes: 8 additions & 5 deletions include/gdsync/core.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,14 @@ typedef enum gds_param {
int gds_query_param(gds_param_t param, int *value);

enum gds_create_qp_flags {
GDS_CREATE_QP_DEFAULT = 0,
GDS_CREATE_QP_WQ_ON_GPU = 1<<0,
GDS_CREATE_QP_TX_CQ_ON_GPU = 1<<1,
GDS_CREATE_QP_RX_CQ_ON_GPU = 1<<2,
GDS_CREATE_QP_WQ_DBREC_ON_GPU = 1<<5,
GDS_CREATE_QP_DEFAULT = 0,
GDS_CREATE_QP_WQ_ON_GPU = 1<<0,
GDS_CREATE_QP_TX_CQ_ON_GPU = 1<<1,
GDS_CREATE_QP_RX_CQ_ON_GPU = 1<<2,
GDS_CREATE_QP_GPU_INVALIDATE_TX_CQ = 1 << 3,
GDS_CREATE_QP_GPU_INVALIDATE_RX_CQ = 1 << 4,
GDS_CREATE_QP_WQ_DBREC_ON_GPU = 1<<5,
GDS_CREATE_QP_FLUSHER = 1<<6,
};

typedef struct ibv_exp_qp_init_attr gds_qp_init_attr_t;
Expand Down
6 changes: 4 additions & 2 deletions src/apis.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
#include "utils.hpp"
#include "memmgr.hpp"
//#include "mem.hpp"

#include "flusher.hpp"

//-----------------------------------------------------------------------------

Expand Down Expand Up @@ -622,7 +622,9 @@ int gds_stream_post_descriptors(CUstream stream, size_t n_descs, gds_descriptor_
// move flush to last wait in the whole batch
if (n_waits && no_network_descs_after_entry(n_descs, descs, last_wait)) {
gds_dbg("optimizing FLUSH to last wait i=%zu\n", last_wait);
move_flush = true;

@drossetti drossetti Jan 27, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who is setting move_flush=true in the 'GPU support native flusher' case?

//If GPU doesn't support native flusher
n_mem_ops += gds_flusher_count_op();
gds_dbg("optimizing FLUSH to last wait i=%zu, n_mem_ops: %d, flusher ops: %d\n", last_wait, n_mem_ops, gds_flusher_count_op());
}
// alternatively, remove flush for wait is next op is a wait too

Expand Down
Loading