Skip to content

feat(cudf): Add GPU array access support for element_at, subscript, and get#17243

Open
ReemaAlzaid wants to merge 40 commits intofacebookincubator:mainfrom
ReemaAlzaid:add-gpu-element-at-array
Open

feat(cudf): Add GPU array access support for element_at, subscript, and get#17243
ReemaAlzaid wants to merge 40 commits intofacebookincubator:mainfrom
ReemaAlzaid:add-gpu-element-at-array

Conversation

@ReemaAlzaid
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid ReemaAlzaid commented Apr 19, 2026

Summary

This PR adds GPU support for array access functions (element_at, subscript, and Spark get) in the cuDF expression evaluator. Until now, these functions were not handled on the GPU, so queries using them would fall back. To match Velox semantics, the change adds shared array access helpers that validate and normalize indices before calling cuDF list extraction.

What changed

  • Added cuDF GPU support for Presto element_at and subscript.
  • Added cuDF GPU support for Spark get.
  • Introduced reusable array access helpers (ArrayAccessFunction and ArrayAccessPolicy) to centralize index validation and normalization before cuDF extraction.
  • Kept Presto and Spark function registration in their engine-specific files, with shared array access machinery reused through common helpers.
  • Added support for constant array literals by materializing the literal array as a repeated cuDF lists column matching the index column cardinality.
  • Added tests covering selection and filter/project cases, including constant, column-based, negative, zero, null, and out-of-bound indices.

Implementation notes

  • element_at uses 1-based indexing, supports negative indices from the end of the array, and returns null for out-of-bound indices.
  • subscript uses 1-based indexing and raises on zero, negative, or out-of-bound indices.
  • Spark get uses 0-based indexing and returns null for negative or out-of-bound indices.
  • Constant literal indices use cuDF scalar extraction when possible.
  • Constant array literals are currently converted to repeated cuDF lists columns because Velox does not pass literal arguments through inputColumns, while cuDF list extraction expects a lists column input.

Limitation

Compared with native C++ Velox, this cuDF implementation validates invalid indices by materializing and scanning masks across all rows before extraction. Native Velox can perform checks during per-row array access. This may introduce extra performance cost; fusing validation with extraction in a custom GPU kernel can be considered as future work.

Constant array literal handling is also a functional bridge rather than the ideal long-term representation. Materializing the constant array as a repeated lists column is less efficient than representing the constant complex value directly in cuDF. A better follow-up would be native support for constant complex/list values, similar in spirit to scalar constants such as cudf::numeric_scalar, if cuDF adds or exposes an appropriate representation.

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 19, 2026

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 548c67a
🔍 Latest deploy log https://app.netlify.com/projects/meta-velox/deploys/69fcad9e7273e40008f67cc6

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 19, 2026
@devavret devavret changed the title Add GPU array element_at support feat(cudf): Add GPU array element_at support Apr 20, 2026
@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid this looks good. It certainly checks the boxes as far as a "reusable base class". Some of the helper functions seem a little long-winded, and maybe the array-specific stuff can be moved to a separate CU as this one is already too big. Can you also add a function alias for subscript as that's what TPC-DS expects. Is element_at a function name expected by other implementations? Then get the TPC-DS tests and benchmarks from the velox-testing repo working and check that that this implementation fills the gap in TPC-DS Q70 (I think) so that there is no CPU fallback.

@ReemaAlzaid ReemaAlzaid marked this pull request as ready for review April 22, 2026 15:12
@ReemaAlzaid ReemaAlzaid changed the title feat(cudf): Add GPU array element_at support feat(cudf): Add GPU array access support for element_at, subscript, and get Apr 22, 2026
Copy link
Copy Markdown
Collaborator

@simoneves simoneves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for not really giving this a proper review the first time. It looks good, with a few minor nits, but I have three main concerns, and one question:

  • Support for INTEGER index columns without casting up to BIGINT
  • Use scalar-index variant of extract_list_element instead of materializing column
  • Is it possible to do what validateIndices does on GPU instead of CPU

Please also explain the policy variations.

#include <cudf/datetime.hpp>
#include <cudf/fixed_point/fixed_point.hpp>
#include <cudf/hashing.hpp>
#include <cudf/lists/extract.hpp>
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these other two includes actually needed?

Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
Comment thread velox/experimental/cudf/tests/ExpressionEvaluatorSelectionTest.cpp Outdated
Comment thread velox/experimental/cudf/expression/ExpressionEvaluator.cpp Outdated
testUnaryFunction("abs(c0)", -5.5, 5.5);
}

TEST_F(CudfSimpleFilterProjectTest, elementAtArrayConstantIndex) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also feels like these tests can be merged somewhat, although I see that the differing policy behavior means that the tests aren't identical. Fine either way.

}

const ArrayAccessPolicy policy_;
bool hasConstantIndex_{false};
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag seems superfluous given that constantIndex_ itself is a std::optional. The logic in the constructor could then perhaps be simplified a little.


auto rawIndexColumn =
prepareRawIndexColumn(inputColumns, listsView.size(), stream, mr);
auto rawIndex64 = ensureInt64Indices(rawIndexColumn, stream, mr);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this step (and corresponding function) needed? Surely prepareRawIndexColumn will always return an INT64 column, the same type as the scalar it's generated from?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming that extract_list_element with a live column can accept either an int64_t/BIGINT or int32_t/INTEGER column? Do you have a test for that?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I have added the test coverages in FilterProjectTest for both converges int64_t and int32_t

Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid not sure why this PR isn't running full CI. Wondering if you need more permissions, or if it just behaves differently for IBM folks? @majetideepak @kgpai ?

@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid not sure why this PR isn't running full CI. Wondering if you need more permissions, or if it just behaves differently for IBM folks? @majetideepak @kgpai ?

It says "5 workflows awaiting approval". Perhaps that's why.

@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid not sure why this PR isn't running full CI. Wondering if you need more permissions, or if it just behaves differently for IBM folks? @majetideepak @kgpai ?

It says "5 workflows awaiting approval". Perhaps that's why.

Thanks @czentgr for fixing the permissions. CI now running.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 22, 2026

Build Impact Analysis

Full build recommended. Files outside the dependency graph changed:

  • velox/experimental/cudf/expression/ArrayAccessFunctions.cpp
  • velox/experimental/cudf/expression/ArrayAccessFunctions.h
  • velox/experimental/cudf/expression/CMakeLists.txt
  • velox/experimental/cudf/expression/CommonFunctions.cpp
  • velox/experimental/cudf/expression/CommonFunctions.h
  • velox/experimental/cudf/expression/PrestoFunctions.cpp
  • velox/experimental/cudf/expression/SparkFunctions.cpp
  • velox/experimental/cudf/tests/CudfFunctionBaseTest.h
  • velox/experimental/cudf/tests/ExpressionEvaluatorSelectionTest.cpp
  • velox/experimental/cudf/tests/FilterProjectTest.cpp
  • ... and 1 more

These directories are not fully covered by the dependency graph. A full build is the safest option.

cmake --build _build/release

Slow path • Graph generated from PR branch

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 22, 2026

CI Failure Analysis

Auto-generated by the CI Failure Analysis workflow. This comment is updated in place each time CI fails on a new commit, so it always reflects the latest run — re-pushing or re-running CI will refresh the analysis below. Last updated 2026-05-05 20:42:29 UTC from workflow run 25398575706.

❌ Build with GCC / cudf-tests — TEST Failure View logs

Failed test binary: velox_cudf_filter_project_test (4 gtest cases failed)

Failed tests:

  1. CudfSimpleFilterProjectTest.arrayAccessSupportedScalarElementTypes — Decimal precision mismatch
  2. CudfSimpleFilterProjectTest.arrayAccessSupportedComplexElementTypes — cuDF list_view type not supported
  3. CudfSimpleFilterProjectTest.arrayAccessConstantArraySupportedTypes — Arrow bridge encoding error
  4. CudfSimpleFilterProjectTest.subscriptConstantArrayVariableIndex — Arrow bridge encoding error

Failure details:

Test 1: arrayAccessSupportedScalarElementTypes
  File: velox/vector/tests/utils/VectorTestBase.cpp:149
  Trace: velox/experimental/cudf/tests/FilterProjectTest.cpp:1525 / :1521

  For short decimal arrays (element_at, subscript — column, cast(int), cast(bigint), null index):
    Expected: DECIMAL(10, 2)
    Actual:   DECIMAL(18, 2)

  For long decimal arrays (same operations):
    Expected: DECIMAL(20, 4)
    Actual:   DECIMAL(38, 4)

  The array access functions always return DECIMAL with max precision for
  the backing type (18 for short, 38 for long) instead of preserving the
  input array's element precision.

Test 2: arrayAccessSupportedComplexElementTypes
  C++ exception: "CUDF failure at:
    _build/release/_deps/cudf-src/cpp/src/scalar/scalar_factories.cpp:146:
    list_view type not supported"

  Nested list (ARRAY(ARRAY(BIGINT))) is not supported by cuDF scalar factories.

Test 3: arrayAccessConstantArraySupportedTypes
Test 4: subscriptConstantArrayVariableIndex
  File: velox/vector/arrow/Bridge.cpp:1162
  Function: exportFlattenedVector
  Expression: vec.valueVector() == nullptr || vec.wrappedVector()->isFlatEncoding()

  "An unsupported nested encoding was found."
  Thrown when converting constant array expressions to cuDF via Arrow bridge.

Correlation with PR changes:

These failures are directly caused by the PR. PR #17243 introduces new ArrayAccessFunctions (element_at, subscript) for the cuDF expression evaluator along with the test file velox/experimental/cudf/tests/FilterProjectTest.cpp that contains all 4 failing tests. The issues are:

  1. Decimal precision loss: The ArrayAccessFunction::eval() in ArrayAccessFunctions.cpp extracts list elements via cuDF's lists::extract_list_element, which returns decimals at max precision for the backing integer type (18 digits for int64, 38 for int128) rather than preserving the original precision/scale from the input array's element type.

  2. Nested list unsupported: cuDF's scalar factories don't support list_view type, so ARRAY(ARRAY(BIGINT)) cannot be processed. This is a known cuDF limitation (see issue Differences Between Velox and cuDF (Data Types & Operators) #17300 — "Differences Between Velox and cuDF").

  3. Constant array Arrow bridge: Constant array vectors with non-flat inner encoding fail when exported through the Velox-Arrow bridge (Bridge.cpp:1162), which expects flat encoding for nested vectors.

Known issues:

  • Issue #17300 documents differences between Velox and cuDF data types, including decimal precision semantics — the decimal precision mismatch is a manifestation of this known gap.
  • The cudf-tests job is skipped on main (these tests only exist in this PR), so this is not a pre-existing/flaky failure.

Reproduce locally:

# Build cuDF tests (requires NVIDIA GPU + cuDF)
./_build/release/velox/experimental/cudf/tests/velox_cudf_filter_project_test \
  --gtest_filter="CudfSimpleFilterProjectTest.arrayAccessSupportedScalarElementTypes:\
CudfSimpleFilterProjectTest.arrayAccessSupportedComplexElementTypes:\
CudfSimpleFilterProjectTest.arrayAccessConstantArraySupportedTypes:\
CudfSimpleFilterProjectTest.subscriptConstantArrayVariableIndex"

Recommended fixes:

  1. Decimal precision: After extracting a list element via cuDF, cast the result column back to the original decimal type (preserving precision/scale from the input array's element type) before converting to a Velox vector. Alternatively, construct the output Velox vector with the correct decimal type metadata.

  2. Nested list (complex element types): Either skip the ARRAY(ARRAY(BIGINT)) subcase in the test (marking it as a known limitation with a TODO referencing Differences Between Velox and cuDF (Data Types & Operators) #17300), or implement a fallback to CPU evaluation for unsupported nested types.

  3. Constant array Arrow bridge: Flatten the constant array vector before exporting through the Arrow bridge, or implement a different code path for constant array inputs that avoids the Arrow bridge entirely (e.g., construct the cuDF column directly from the constant value).

@ReemaAlzaid ReemaAlzaid requested a review from simoneves April 23, 2026 20:27
Copy link
Copy Markdown
Collaborator

@simoneves simoneves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of change c/w the previous version, and I'm not sure I follow some of it, but it appears you have resolved my three main concerns, although the new ConstantIndex implementation seems bogus.

Plus some new stuff out of nowhere. Please advise.


void SetUp() override {
previousMemoryResource_ = CudfConfig::getInstance().memoryResource;
CudfConfig::getInstance().memoryResource = "cuda";
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason for adding this?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and in FilterProjectTest.cpp...?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that as a local test workaround for the default async memory resource path so I’ve removed it

auto expected = makeNullableFlatVector<int32_t>({3, 6, std::nullopt, 7});

facebook::velox::test::assertEqualVectors(
expected, evaluate("element_at(c0, -1)", makeRowVector({arrays})));
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe overkill, but maybe add index-from-column versions of these tests too (both types)...?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added them


class CudfFunctionBaseTest : public velox::functions::test::FunctionBaseTest {
protected:
using velox::functions::test::FunctionBaseTest::evaluate;
Copy link
Copy Markdown
Collaborator

@simoneves simoneves Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed? Surely it was resolving that function type just fine before, as it's an existing test base class and surely worked? Or did you uncover something more insidious? If so, please explain.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, needed for these tests. Adding our own evaluate(...) here hides the parent class versions in C++. The array access tests now use one of those parent versions directly, so using just makes it visible again

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove it, @simoneves will add a new function in this class to verify the CPU and GPU results, this PR has under review and will merge soon

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address this comment

#include <cudf/transform.hpp>
#include <cudf/types.hpp>
#include <cudf/unary.hpp>
#include <cudf/utilities/error.hpp>
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You removed one of the two. Isn't this one superfluous also? Apologies if not...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, forgot to remove after moving the array access logic out

Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
return;
}

constantIndex_.emplace();
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do, without any arguments? I mean, the nested std::optional is surely wrong anyway? (see added-earlier comment below) This will create the outer one, but not the inner one. I guess the inner one gets assigned below, but it still seems wrong unless I'm missing something...?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My reasoning there was to represent 3 states index before, That said, I agree this is awkward and easy to misread. I’ve simplified it by removing ConstantIndex and changing this back to std::optional<int64_t> constantIndex_

if constexpr (std::is_same_v<IndexType, int32_t>) {
return normalized;
} else {
return cudf::cast(
Copy link
Copy Markdown
Collaborator

@simoneves simoneves Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the actual final indices passed to extract_list_element ONLY be int32_t (and we just provide the BIGINT version of the function as a convenience) or can they also actually be int64_t with values above 2^32? (seems unlikely in practice, but even so)?

Copy link
Copy Markdown
Collaborator

@simoneves simoneves Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADDENDUM: OK, so it seems that array indices are indeed limited to cudf::size_type which is int32_t (surprisingly) so it makes sense that your code now generates final index values/columns as int32_t.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADDENDUM TO ADDENDUM: I am frankly very surprised that cudf::size_type is int32_t as that applies to row counts of columns too? (hence a 2B-row limit). I'm not surprised that array indices can't be full-64-bit, but presumably SQL allows for it, if TPC-DS requires that variant of subscript?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this matches what I found. Since cudf::size_type is int32_t, the final indices passed to extract_list_element should be int32_t

We can still expose a BIGINT variant for SQL compatibility, but it needs to cast/validate before calling cuDF values outside int32_t can’t be represented anyway (also consistent with rapidsai/cudf#3958 (comment)).

I also checked TPC-DS (incl. Q70), and there’s no array/subscript usage, so it doesn’t require a BIGINT subscript variant

mr);

if constexpr (std::is_same_v<IndexType, int64_t>) {
auto minIndex = cudf::numeric_scalar<int64_t>(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this block intended to do? Seems like it's checking that int64_t indices are within a range, and I thought perhaps that would be the int32_t range (such that the cast-back-to-int32_t later would be valid) but the minIndex and maxIndex are int64_t...?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADDENDUM: Sorry! I was reading it wrong. I see now that the actual VALUES are min/max of cudf::size_type which is int32_t, just then CAST to int64_t for the purpose of the comparison passes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it’s just guarding the later cast to cudf::size_type. I renamed the locals to make that clearer

Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
};

struct ConstantIndex {
std::optional<int64_t> value;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(see later)

@simoneves
Copy link
Copy Markdown
Collaborator

The CI is blocked again. Not sure why. @czentgr can you work your magic again, please? Why didn't your first fix stick?

@simoneves
Copy link
Copy Markdown
Collaborator

simoneves commented Apr 24, 2026

Looking into the Velox CPU subscript implementation, it appears that integer and bigint SQL index types are supported, as well as MAP types (as you had in one of your tests, that I naively suggested you remove) although it too limits the actual index values to facebook::velox::vector_size_t which is also int32_t.

Your index manipulation and validation code looks a lot like what's in velox/functions/lib/SubscriptUtil.h so I guess that's where you got it from. Ditto the four policy flags and their justification (see comments on earlier thread that I added before I officially started this re-review).

Comment thread velox/experimental/cudf/exec/ToCudf.cpp Outdated
Comment thread velox/experimental/cudf/expression/ArrayAccessFunctions.cpp Outdated
const ArrayAccessPolicy& policy,
rmm::cuda_stream_view stream,
rmm::device_async_resource_ref mr) {
auto sizes = castSizes<IndexType>(sizesView, stream, mr);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the IndexType same with rawIndexView type?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The caller guarantees this: INT8/INT16 inputs are first cast to INT32 before calling normalizeAndValidateIndicesTyped<int32_t>(); INT32 and INT64 inputs are dispatched directly to int32_t and int64_t

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then do you need the template class IndexType? You can use rawIndexView.type()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've done a patch for that and used rawIndexView.type()

VELOX_USER_FAIL("Array subscript index cannot be negative");
}

std::unique_ptr<cudf::column> positiveNormalized;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update to cudf::column_view to avoid copy in else branch, please notice you need to reserve the original column to avoid memory issue
auto positiveNormalized = rawIndexView;
if (policy.indexStartsAtOne)

#include "velox/expression/FunctionSignature.h"

#include <initializer_list>
#include <string>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the header

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify what issue you see in the header? I checked CommonFunctions.h, and it already includes the headers for the types it uses directly: <initializer_list>, <string>, and <vector>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These headers are already included in "velox/expression/FunctionSignature.h"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove them then thanks


class CudfFunctionBaseTest : public velox::functions::test::FunctionBaseTest {
protected:
using velox::functions::test::FunctionBaseTest::evaluate;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove it, @simoneves will add a new function in this class to verify the CPU and GPU results, this PR has under review and will merge soon

@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid the test refactor that @jinchengchenghh mentioned above is 0ec817a from #16913 which will indeed hopefully land very soon.

@jinchengchenghh
Copy link
Copy Markdown
Contributor

Please resolve this comment, #17243 (comment), add the test for all the data type of array vector and index data type, idealy, the tests should cover all the paths, that's also what CPU does, for GPU project, this is a young project, so we make sure to cover the functionality path

@simoneves
Copy link
Copy Markdown
Collaborator

Where are we on this one?

Build errors in both versions of FilterProjectTest and some files need a clang-format (see pre-commit CI job).

@jinchengchenghh did you have any other concerns?

@devavret @karthikeyann or anyone else?

@simoneves
Copy link
Copy Markdown
Collaborator

@ReemaAlzaid please refactor (at least the cudf version of) FilterProjectTest to use the assertExpressionMatchesCpu helper function that landed in #16913 thx (and consider a similar refactor to the Spark one if you think it's sensible)

@ReemaAlzaid
Copy link
Copy Markdown
Collaborator Author

ReemaAlzaid commented May 5, 2026

@ReemaAlzaid please refactor (at least the cudf version of) FilterProjectTest to use the assertExpressionMatchesCpu helper function that landed in #16913 thx (and consider a similar refactor to the Spark one if you think it's sensible)

Done the refactor in both files

Copy link
Copy Markdown
Contributor

@jinchengchenghh jinchengchenghh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please solve the red CI

Comment thread velox/experimental/cudf/tests/sparksql/FilterProjectTest.cpp Outdated
const RowVectorPtr& input,
const std::string& message) {
auto exprSet = compileExpression(expr, input->rowType());
VELOX_ASSERT_THROW(evaluate(*exprSet, input), message);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VELOX_ASSERT_THROW(evaluate(expr, input), message); there is API that accepts string expression

@ReemaAlzaid
Copy link
Copy Markdown
Collaborator Author

Please solve the red CI

the tests are passing now

protected:
using velox::functions::test::FunctionBaseTest::evaluate;

void assertExpressionMatchesCpu(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


class CudfFunctionBaseTest : public velox::functions::test::FunctionBaseTest {
protected:
using velox::functions::test::FunctionBaseTest::evaluate;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address this comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants