Skip to content

Fix compose-tests stability: quickstart startup race and missing Hadoop conf path #3565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SethSmucker wants to merge 4 commits into
base: integration
Choose a base branch
from
Open

Fix compose-tests stability: quickstart startup race and missing Hadoop conf path #3565

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
54cb9d9
Add /etc/hadoop/conf symlink in quickstart Docker image
SethSmucker May 20, 2026
785f751
Add quickstart dependency for services that connect to ZooKeeper or H…
SethSmucker May 20, 2026
e20b985
Merge branch 'integration' into task/compose-tests-stability
SethSmucker May 20, 2026
64e8cbc
Bump spring-boot-starter-datawave-query consumers to 1.0.11-SNAPSHOT
SethSmucker May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions contrib/datawave-quickstart/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,11 @@ RUN /bin/bash -c "/usr/bin/nohup /usr/sbin/sshd -D &" && \
rm -rf contrib/datawave-quickstart/accumulo/logs/* && \
rm -rf contrib/datawave-quickstart/wildfly/standalone/log/*

# Accumulo TabletServer-side datawave iterators load Hadoop config from /etc/hadoop/conf;
# without this symlink, event-table scans error with FileNotFoundException server-side.
RUN mkdir -p /etc/hadoop && \
ln -sfn /opt/datawave/contrib/datawave-quickstart/hadoop/etc/hadoop /etc/hadoop/conf

# Lastly, establish volumes for data, logs & other directories, wire up
# the entrypoint & bootstrap scripts, expose ports, and set default CMD...

Expand Down
18 changes: 18 additions & 0 deletions docker/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,8 @@ services:
depends_on:
configuration:
condition: service_healthy
quickstart:
condition: service_healthy

query-cache:
profiles:
Expand Down Expand Up @@ -361,6 +363,8 @@ services:
depends_on:
authorization:
condition: service_healthy
quickstart:
condition: service_healthy

audit:
image: nationalsecurityagency/datawave/audit-service:${VERSION:-latest}
Expand Down Expand Up @@ -394,6 +398,8 @@ services:
depends_on:
authorization:
condition: service_healthy
quickstart:
condition: service_healthy

metrics:
entrypoint: ["java","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5007","-jar","app.jar"]
Expand Down Expand Up @@ -463,6 +469,8 @@ services:
depends_on:
authorization:
condition: service_healthy
quickstart:
condition: service_healthy

file-provider:
profiles:
Expand Down Expand Up @@ -545,6 +553,8 @@ services:
condition: service_healthy
executor-pool1:
condition: service_started
quickstart:
condition: service_healthy

mapreduce-query:
profiles:
Expand Down Expand Up @@ -583,6 +593,8 @@ services:
condition: service_healthy
executor-pool1:
condition: service_started
quickstart:
condition: service_healthy

executor-pool1:
entrypoint: ["java","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5006","-jar","app.jar"]
Expand Down Expand Up @@ -630,6 +642,8 @@ services:
condition: service_healthy
metrics:
condition: service_healthy
quickstart:
condition: service_healthy

executor-pool2:
profiles:
Expand Down Expand Up @@ -678,6 +692,8 @@ services:
condition: service_healthy
metrics:
condition: service_healthy
quickstart:
condition: service_healthy

modification:
entrypoint: ["java","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5010","-jar","app.jar"]
Expand Down Expand Up @@ -712,6 +728,8 @@ services:
depends_on:
authorization:
condition: service_healthy
quickstart:
condition: service_healthy

# If you use the management center, you can connect to the hazelcast cache as follows:
# In your browser connect to https://localhost:9243/
Expand Down
2 changes: 1 addition & 1 deletion microservices/services/mapreduce-query/jobs/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<properties>
<version.datawave>7.33.1</version.datawave>
<version.datawave.mapreduce-layout-factory>1.0.0</version.datawave.mapreduce-layout-factory>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
Copy link
Copy Markdown
Collaborator

@alerman alerman May 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should release starter-query and only merge this after release so we arent merging in dependencies on a snapshot

Copy link
Copy Markdown
Collaborator

@ivakegg ivakegg May 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there changes in starter-query that are required for this PR ?

Copy link
Copy Markdown
Collaborator Author

@SethSmucker SethSmucker May 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivakegg - changes from 3560 removed eventPerDayThreshold stuff, but the jar wasn't updated so it keeps looking for that property but can never resolve it. 1.0.11-SNAPSHOT is just to include those changes that were made but not reflected in the current jar. So it's more so that the changes were fine, we just need to bump the jar.

@alerman Agreed, I have the snapshot in here just to make sure that all of the tests actually pass and I diagnosed the issues accurately. So we can remove the snapshot from this PR, but the other changes should still probably be made since that was another chunk of issues that we get when running the CI tests.

<version.zookeeper>3.9.2</version.zookeeper>
</properties>
<dependencyManagement>
Expand Down
2 changes: 1 addition & 1 deletion microservices/services/mapreduce-query/service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
<version.datawave.authorization-api>4.0.2</version.datawave.authorization-api>
<version.datawave.mapreduce-query-core-job>1.0.5-SNAPSHOT</version.datawave.mapreduce-query-core-job>
<version.datawave.starter>4.0.7</version.datawave.starter>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
<version.hadoop>3.3.4</version.hadoop>
<!-- dependency declared in datawave-microservice-parent, but the version can be overridden here -->
<version.log4j>2.25.3</version.log4j>
Expand Down
2 changes: 1 addition & 1 deletion microservices/services/query-executor/api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<datawave.webservice.namespace>http://webservice.datawave/v1</datawave.webservice.namespace>
<version.commons-lang3>3.20.0</version.commons-lang3>
<version.datawave>7.33.1</version.datawave>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
<version.jakarta>2.3.3</version.jakarta>
</properties>
<dependencyManagement>
Expand Down
2 changes: 1 addition & 1 deletion microservices/services/query-executor/service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<version.datawave.query-executor-api>1.0.9</version.datawave.query-executor-api>
<version.datawave.query-metric-api>4.1.3</version.datawave.query-metric-api>
<version.datawave.starter>4.0.7</version.datawave.starter>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
<version.datawave.starter-query-metric>3.0.5</version.datawave.starter-query-metric>
<version.guava>31.1-jre</version.guava>
<version.hadoop>3.3.4</version.hadoop>
Expand Down
2 changes: 1 addition & 1 deletion microservices/services/query/service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
<version.datawave.starter>4.0.7</version.datawave.starter>
<version.datawave.starter-audit>4.0.5</version.datawave.starter-audit>
<version.datawave.starter-cached-results>1.0.12</version.datawave.starter-cached-results>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
<version.datawave.starter-query-metric>3.0.5</version.datawave.starter-query-metric>
<version.guava>31.1-jre</version.guava>
<version.hadoop>3.3.4</version.hadoop>
Expand Down
2 changes: 1 addition & 1 deletion microservices/starters/cached-results/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
<version.datawave.query-api>1.0.1</version.datawave.query-api>
<version.datawave.starter>4.0.7</version.datawave.starter>
<version.datawave.starter-audit>4.0.5</version.datawave.starter-audit>
<version.datawave.starter-query>1.0.10</version.datawave.starter-query>
<version.datawave.starter-query>1.0.11-SNAPSHOT</version.datawave.starter-query>
<version.mysql-connector>9.3.0</version.mysql-connector>
</properties>
<dependencyManagement>
Expand Down
Loading