From 722702cc49183a1a7a8895d70855f2fb807e4b7e Mon Sep 17 00:00:00 2001 From: George Apaaboah <35894485+GeorgeAp@users.noreply.github.com> Date: Wed, 11 May 2022 15:25:29 +0200 Subject: [PATCH 01/10] [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes #65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou Closes #4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7ad7 Avoid C++ ABI issues with DictionaryMemo eaede5be2 Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f784e ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon Co-authored-by: Stéphane Campinas Co-authored-by: Antoine Pitrou Co-authored-by: Martin Anseaume Co-authored-by: Johnny Hujol Co-authored-by: ggdupont * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah Co-authored-by: Laurent Goujon Co-authored-by: Stéphane Campinas Co-authored-by: Antoine Pitrou Co-authored-by: Martin Anseaume Co-authored-by: Johnny Hujol Co-authored-by: ggdupont --- README.md | 67 +++++++++++++++++++ java/adapter/orc/pom.xml | 2 +- java/format/pom.xml | 2 +- java/gandiva/pom.xml | 2 +- java/memory/memory-core/pom.xml | 2 +- .../apache/arrow/memory/BoundsChecking.java | 13 ++-- .../apache/arrow/memory/util/MemoryUtil.java | 6 +- .../arrow/memory/TestBoundaryChecking.java | 4 +- java/memory/memory-netty/pom.xml | 51 +++++++++++++- java/memory/memory-unsafe/pom.xml | 2 +- java/memory/pom.xml | 2 +- java/pom.xml | 36 +++++++--- java/vector/pom.xml | 2 +- .../src/main/codegen/includes/vv_imports.ftl | 2 + .../arrow/vector/BaseFixedWidthVector.java | 2 +- .../org/apache/arrow/vector/BitVector.java | 5 ++ .../apache/arrow/vector/BitVectorHelper.java | 8 +-- .../apache/arrow/vector/Decimal256Vector.java | 2 +- .../apache/arrow/vector/DecimalVector.java | 2 +- .../arrow/vector/util/DecimalUtility.java | 2 +- .../arrow/vector/util/VectorAppender.java | 2 +- .../arrow/vector/TestBitVectorHelper.java | 2 +- .../apache/arrow/vector/TestValueVector.java | 2 +- 23 files changed, 179 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 7d10b81c6e4f..f6a9073415f9 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,73 @@ under the License. --> +# Siren fork of Arrow + +- The properties `drill.enable_unsafe_memory_access` and + `arrow.enable_unsafe_memory_access` are prefixed with `siren` and their + default value is set to `true`. The first property is deprecated. + +- In order to avoid conflict with a version of `netty` used in Elasticsearch, we + relocate the netty custom package and dependency in `memory` into a package + named `siren`. The relocation is achieved thanks to the maven shade plugin. + +- The Siren's fork of `netty` is used in `vector`. This means that `netty` + imports in that module need to be prefixed with `siren`. + +## Check that Siren version of Netty is used +- In order to check that Siren version of Netty is being used, + run the unit test `CheckAccessibleTest` in + `https://github.com/sirensolutions/siren-platform/blob/master/core/src/test/java/io/siren/federate/core/common/CheckAccessibleTest.java`. +- Note: the unit test `CheckAccessibleTest` is currently ignored, please set it again to ignore after running the test. + The unit test is ignored because the settings in `CheckAccessibleTest` is not taken into account when the whole unit test suite is run, therefore it fails. + This could be because when the class is loaded, the default settings is used (which is a static block) and the new settings in the `CheckAccessibleTest` is + then not applied when the test suit is run. + +## Build + +To build the `memory`, `format` and `vector` modules: + +```sh +$ cd java +$ mvn clean package +``` + +Because of the default value change of `unsafe_memory_access` property, some +tests in `vector` fail. + +```sh +mvn -pl memory,memory/memory-core,memory/memory-netty,memory/memory-unsafe,format,vector install -Dsiren.arrow.enable_unsafe_memory_access=false -Dsiren.drill.enable_unsafe_memory_access=false +``` + +## Make a new release of Siren's Apache Arrow + +- Tests should pass. + +- Make a new version: + +```sh +mvn versions:set -DnewVersion=siren-0.14.1-2 +``` + +- tag the commit for the release + +```sh +git tag --sign siren-0.14.1-2 +```` + +- Deploy to Siren's artifactory + +```sh +$ mvn deploy -DskipTests=true -P artifactory -Dartifactory_username= -Dartifactory_password= +``` + +## Update to a new version of Siren's Apache Arrow + +- add `git@github.com:apache/arrow.git` as the `upstream` remote. +- execute `git fetch --all --tags` +- create a temporary branch from `siren-changes` +- rebase against the new tag. + # Apache Arrow [![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/arrow.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:arrow) diff --git a/java/adapter/orc/pom.xml b/java/adapter/orc/pom.xml index b4421b7ae6e9..a08bd6e2c6de 100644 --- a/java/adapter/orc/pom.xml +++ b/java/adapter/orc/pom.xml @@ -104,7 +104,7 @@ org.apache.arrow arrow-java-root - 9.0.0 + siren-9.0.0-1-SNAPSHOT ../../pom.xml diff --git a/java/format/pom.xml b/java/format/pom.xml index 1bb562ee1601..fe5480aef0d0 100644 --- a/java/format/pom.xml +++ b/java/format/pom.xml @@ -15,7 +15,7 @@ arrow-java-root org.apache.arrow - 9.0.0 + siren-9.0.0-1-SNAPSHOT arrow-format diff --git a/java/gandiva/pom.xml b/java/gandiva/pom.xml index bc94346ec830..1c32d5b2c529 100644 --- a/java/gandiva/pom.xml +++ b/java/gandiva/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - 9.0.0 + siren-9.0.0-1-SNAPSHOT org.apache.arrow.gandiva diff --git a/java/memory/memory-core/pom.xml b/java/memory/memory-core/pom.xml index 1cd52ed998d1..577796763242 100644 --- a/java/memory/memory-core/pom.xml +++ b/java/memory/memory-core/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - 9.0.0 + siren-9.0.0-1-SNAPSHOT 4.0.0 diff --git a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java index bbf7ff34dc96..a6d798bd093e 100644 --- a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java +++ b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java @@ -34,14 +34,15 @@ public class BoundsChecking { static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BoundsChecking.class); static { - String envProperty = System.getenv("ARROW_ENABLE_UNSAFE_MEMORY_ACCESS"); - String oldProperty = System.getProperty("drill.enable_unsafe_memory_access"); + String envProperty = System.getenv().getOrDefault("SIREN_ARROW_ENABLE_UNSAFE_MEMORY_ACCESS", "true"); + String oldProperty = System.getProperty("siren.drill.enable_unsafe_memory_access", "true"); if (oldProperty != null) { - logger.warn("\"drill.enable_unsafe_memory_access\" has been renamed to \"arrow.enable_unsafe_memory_access\""); - logger.warn("\"arrow.enable_unsafe_memory_access\" can be set to: " + - " true (to not check) or false (to check, default)"); + logger.warn("\"siren.drill.enable_unsafe_memory_access\" has been renamed to " + + "\"siren.arrow.enable_unsafe_memory_access\""); + logger.warn("\"siren.arrow.enable_unsafe_memory_access\" can be set to: " + + " true (to not check, default) or false (to check)"); } - String newProperty = System.getProperty("arrow.enable_unsafe_memory_access"); + String newProperty = System.getProperty("siren.arrow.enable_unsafe_memory_access", "true"); // The priority of determining the unsafe flag: // 1. The system properties take precedence over the environmental variable. diff --git a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java index 16ef39702ca3..ce7a4bd5e210 100644 --- a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java +++ b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java @@ -81,7 +81,6 @@ public Object run() { // get the offset of the address field in a java.nio.Buffer object Field addressField = java.nio.Buffer.class.getDeclaredField("address"); - addressField.setAccessible(true); BYTE_BUFFER_ADDRESS_OFFSET = UNSAFE.objectFieldOffset(addressField); Constructor directBufferConstructor; @@ -99,10 +98,7 @@ public Object run() { constructor.setAccessible(true); logger.debug("Constructor for direct buffer found and made accessible"); return constructor; - } catch (NoSuchMethodException e) { - logger.debug("Cannot get constructor for direct buffer allocation", e); - return e; - } catch (SecurityException e) { + } catch (Exception e) { logger.debug("Cannot get constructor for direct buffer allocation", e); return e; } diff --git a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java index 5b86bed406b3..2e423119a54d 100644 --- a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java +++ b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java @@ -55,7 +55,7 @@ private boolean getFlagValue(ClassLoader classLoader) throws Exception { } /** - * Ensure the flag for bounds checking is enabled by default. + * Siren: Ensure the flag for bounds checking is disabled by default. * This will protect users from JVM crashes. */ @Test @@ -63,7 +63,7 @@ public void testDefaultValue() throws Exception { ClassLoader classLoader = copyClassLoader(); if (classLoader != null) { boolean boundsCheckingEnabled = getFlagValue(classLoader); - Assert.assertTrue(boundsCheckingEnabled); + Assert.assertFalse(boundsCheckingEnabled); } } diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml index 9fcf713c81e0..882039f5c5fa 100644 --- a/java/memory/memory-netty/pom.xml +++ b/java/memory/memory-netty/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - 9.0.0 + siren-9.0.0-1-SNAPSHOT 4.0.0 @@ -35,9 +35,11 @@ io.netty netty-common + org.slf4j slf4j-api + provided org.immutables @@ -45,6 +47,53 @@ + + + + org.apache.maven.plugins + maven-shade-plugin + 3.1.0 + + + package + + shade + + + false + ${project.build.directory}/dependency-reduced-pom.xml + + + io.netty + siren.io.netty + + + + + org.slf4j + com.google.code.findbugs + com.google.guava + org.apache.arrow:arrow-memory-core:*:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + + + + + + + + + + + @@ -887,6 +887,24 @@ + + + artifactory + + + + artifactory-releases + artifactory-releases + ${artifactory.url}/libs-release-local + + + artifactory-snapshots + artifactory-snapshots + ${artifactory.url}/libs-snapshot-local + + + + diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 4421d559dace..71dafbb7b62f 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - 9.0.0 + siren-9.0.0-1-SNAPSHOT arrow-vector Arrow Vectors diff --git a/java/vector/src/main/codegen/includes/vv_imports.ftl b/java/vector/src/main/codegen/includes/vv_imports.ftl index c9a8820b258b..cd0e2d7e2a4c 100644 --- a/java/vector/src/main/codegen/includes/vv_imports.ftl +++ b/java/vector/src/main/codegen/includes/vv_imports.ftl @@ -20,6 +20,8 @@ import static org.apache.arrow.util.Preconditions.checkState; import com.google.flatbuffers.FlatBufferBuilder; +import siren.io.netty.buffer.*; + import org.apache.arrow.memory.*; import org.apache.arrow.util.Preconditions; import org.apache.arrow.vector.types.Types; diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java index ded58b22b00c..70b7c0d7a432 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java @@ -37,7 +37,7 @@ import org.apache.arrow.vector.util.OversizedAllocationException; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * BaseFixedWidthVector provides an abstract interface for diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java index 3bcfd983e641..1935b45f7bd9 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java @@ -119,6 +119,11 @@ public void setInitialCapacity(int valueCount) { lastValueCapacity = valueCount; } + /** + * Get the current value capacity for the vector. + * + * @return number of elements that vector can hold. + */ @Override protected int getValueBufferValueCapacity() { return capAtMaxInt(valueBuffer.capacity() * 8); diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java index c2c88f2ba378..53f84a884277 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java @@ -17,10 +17,10 @@ package org.apache.arrow.vector; -import static io.netty.util.internal.PlatformDependent.getByte; -import static io.netty.util.internal.PlatformDependent.getInt; -import static io.netty.util.internal.PlatformDependent.getLong; import static org.apache.arrow.memory.util.LargeMemoryUtil.checkedCastToInt; +import static siren.io.netty.util.internal.PlatformDependent.getByte; +import static siren.io.netty.util.internal.PlatformDependent.getInt; +import static siren.io.netty.util.internal.PlatformDependent.getLong; import org.apache.arrow.memory.ArrowBuf; import org.apache.arrow.memory.BoundsChecking; @@ -28,7 +28,7 @@ import org.apache.arrow.vector.ipc.message.ArrowFieldNode; import org.apache.arrow.vector.util.DataSizeRoundingUtil; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Helper class for performing generic operations on a bit vector buffer. diff --git a/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java b/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java index 4341e4175a59..6c2dc76f4ebe 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java @@ -35,7 +35,7 @@ import org.apache.arrow.vector.util.DecimalUtility; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Decimal256Vector implements a fixed width vector (32 bytes) of diff --git a/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java b/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java index 3768c65249b6..1fde4526e38a 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java @@ -35,7 +35,7 @@ import org.apache.arrow.vector.util.DecimalUtility; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * DecimalVector implements a fixed width vector (16 bytes) of diff --git a/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java b/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java index f778bcb209fa..692714487fbf 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java @@ -24,7 +24,7 @@ import org.apache.arrow.memory.ArrowBuf; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Utility methods for configurable precision Decimal values (e.g. {@link BigDecimal}). diff --git a/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java b/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java index ea78917c3ddb..003a8c97b611 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java @@ -39,7 +39,7 @@ import org.apache.arrow.vector.complex.NonNullableStructVector; import org.apache.arrow.vector.complex.UnionVector; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Utility to append two vectors together. diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java b/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java index 4563577202bb..57f64bbde58e 100644 --- a/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java +++ b/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java @@ -27,7 +27,7 @@ import org.apache.arrow.vector.ipc.message.ArrowFieldNode; import org.junit.Test; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; public class TestBitVectorHelper { @Test diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java b/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java index 516daa236228..f32d4d5a8913 100644 --- a/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java +++ b/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java @@ -1846,7 +1846,7 @@ public void testSetLastSetUsage() { assertEquals(40, vector.offsetBuffer.getInt(17 * BaseVariableWidthVector.OFFSET_WIDTH)); assertEquals(40, vector.offsetBuffer.getInt(18 * BaseVariableWidthVector.OFFSET_WIDTH)); assertEquals(40, vector.offsetBuffer.getInt(19 * BaseVariableWidthVector.OFFSET_WIDTH)); - + vector.set(19, STR6); assertArrayEquals(STR6, vector.get(19)); assertEquals(40, vector.offsetBuffer.getInt(19 * BaseVariableWidthVector.OFFSET_WIDTH)); From ca20b7c3715676c38e8412d61f22378b7304a889 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Wed, 11 May 2022 15:38:53 +0200 Subject: [PATCH 02/10] Release siren-7.0.0-1 Signed-off-by: George Apaaboah --- java/pom.xml | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/java/pom.xml b/java/pom.xml index 678bf520527a..6eda37713a0b 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -526,6 +526,36 @@ guava ${dep.guava.version} + + io.netty + netty-handler + ${dep.netty.version} + + + io.netty + netty-buffer + ${dep.netty.version} + + + io.netty + netty-common + ${dep.netty.version} + + + com.fasterxml.jackson.core + jackson-core + ${dep.jackson.version} + + + com.fasterxml.jackson.core + jackson-annotations + ${dep.jackson.version} + + + com.fasterxml.jackson.core + jackson-databind + ${dep.jackson.version} + com.google.code.findbugs jsr305 From ee6bd3d18f641c7ce03f19d9b073a8dca9ee9bf2 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Wed, 31 Aug 2022 09:08:23 +0200 Subject: [PATCH 03/10] Fix rebase --- java/pom.xml | 32 +------------------------------- java/vector/pom.xml | 4 ++-- 2 files changed, 3 insertions(+), 33 deletions(-) diff --git a/java/pom.xml b/java/pom.xml index 6eda37713a0b..50e61bd54cda 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -33,7 +33,7 @@ 5.4.0 1.7.25 30.1.1-jre - 4.1.78.Final + siren-4.1.78-1-SNAPSHOT 1.47.0 3.21.2 2.13.2.20220328 @@ -526,36 +526,6 @@ guava ${dep.guava.version} - - io.netty - netty-handler - ${dep.netty.version} - - - io.netty - netty-buffer - ${dep.netty.version} - - - io.netty - netty-common - ${dep.netty.version} - - - com.fasterxml.jackson.core - jackson-core - ${dep.jackson.version} - - - com.fasterxml.jackson.core - jackson-annotations - ${dep.jackson.version} - - - com.fasterxml.jackson.core - jackson-databind - ${dep.jackson.version} - com.google.code.findbugs jsr305 diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 71dafbb7b62f..50e44676f238 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -125,12 +125,12 @@ test - + From 415831ddbaf8a38a237a3ad867f66d0578186bbd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Campinas?= Date: Fri, 16 Sep 2022 15:38:59 +0200 Subject: [PATCH 04/10] Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac --- java/memory/memory-netty/pom.xml | 6 ++++-- java/vector/pom.xml | 10 ++++++---- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml index 882039f5c5fa..03b694877a77 100644 --- a/java/memory/memory-netty/pom.xml +++ b/java/memory/memory-netty/pom.xml @@ -48,7 +48,7 @@ - + org.apache.maven.plugins maven-shade-plugin @@ -62,6 +62,9 @@ false ${project.build.directory}/dependency-reduced-pom.xml + + + io.netty @@ -73,7 +76,6 @@ org.slf4j com.google.code.findbugs com.google.guava - org.apache.arrow:arrow-memory-core:*:* diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 50e44676f238..7863244dda3a 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -177,7 +177,7 @@ org.apache.maven.plugins maven-shade-plugin - 3.1.1 + 3.3.0 package @@ -187,14 +187,16 @@ + org.apache.arrow:arrow-memory-core + org.apache.arrow:arrow-memory-netty org.apache.arrow:arrow-format com.google.flatbuffers:* true - shade-format-flatbuffers - true - true + shade + true + true com.google.flatbuffers From 2722ef8392bc3e7cc54f0889c11de2b80b96b8ff Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Mon, 26 Sep 2022 11:26:25 +0200 Subject: [PATCH 05/10] Add developer tip link to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f6a9073415f9..091892327713 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,7 @@ $ mvn deploy -DskipTests=true -P artifactory -Dartifactory_username= - ``` ## Update to a new version of Siren's Apache Arrow +Developer tips on updating to a new version of Netty can be found here: https://sirensolutions.atlassian.net/wiki/spaces/EN/pages/3108864001/Upgrading+Federate+Apache+Arrow+Version . - add `git@github.com:apache/arrow.git` as the `upstream` remote. - execute `git fetch --all --tags` From 8e992121c7e716a09b2fa57eca5c0ef49e1fb7da Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Tue, 27 Sep 2022 09:41:25 +0200 Subject: [PATCH 06/10] Remove test scope for arrow-memory-netty --- java/vector/pom.xml | 1 - 1 file changed, 1 deletion(-) diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 7863244dda3a..5dbc0b53258d 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -57,7 +57,6 @@ org.apache.arrow arrow-memory-netty ${project.version} - test org.apache.arrow From 1e462663778c3edef3f764c19380e0c5d48334a3 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Tue, 27 Sep 2022 11:52:49 +0200 Subject: [PATCH 07/10] Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes --- java/adapter/orc/pom.xml | 2 +- java/format/pom.xml | 2 +- java/gandiva/pom.xml | 2 +- java/memory/memory-core/pom.xml | 2 +- java/memory/memory-netty/pom.xml | 2 +- java/memory/memory-unsafe/pom.xml | 2 +- java/memory/pom.xml | 2 +- java/pom.xml | 2 +- java/vector/pom.xml | 2 +- 9 files changed, 9 insertions(+), 9 deletions(-) diff --git a/java/adapter/orc/pom.xml b/java/adapter/orc/pom.xml index a08bd6e2c6de..402c098ee872 100644 --- a/java/adapter/orc/pom.xml +++ b/java/adapter/orc/pom.xml @@ -104,7 +104,7 @@ org.apache.arrow arrow-java-root - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT ../../pom.xml diff --git a/java/format/pom.xml b/java/format/pom.xml index fe5480aef0d0..6da1fcb97683 100644 --- a/java/format/pom.xml +++ b/java/format/pom.xml @@ -15,7 +15,7 @@ arrow-java-root org.apache.arrow - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT arrow-format diff --git a/java/gandiva/pom.xml b/java/gandiva/pom.xml index 1c32d5b2c529..6f925d67854d 100644 --- a/java/gandiva/pom.xml +++ b/java/gandiva/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT org.apache.arrow.gandiva diff --git a/java/memory/memory-core/pom.xml b/java/memory/memory-core/pom.xml index 577796763242..ed024bb1d4aa 100644 --- a/java/memory/memory-core/pom.xml +++ b/java/memory/memory-core/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT 4.0.0 diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml index 03b694877a77..94071adce3bd 100644 --- a/java/memory/memory-netty/pom.xml +++ b/java/memory/memory-netty/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT 4.0.0 diff --git a/java/memory/memory-unsafe/pom.xml b/java/memory/memory-unsafe/pom.xml index 84240ef6c741..3285166c58fb 100644 --- a/java/memory/memory-unsafe/pom.xml +++ b/java/memory/memory-unsafe/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT 4.0.0 diff --git a/java/memory/pom.xml b/java/memory/pom.xml index 997f5c9b2fa7..56e98d140695 100644 --- a/java/memory/pom.xml +++ b/java/memory/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT arrow-memory Arrow Memory diff --git a/java/pom.xml b/java/pom.xml index 50e61bd54cda..f79a74feb0ef 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -20,7 +20,7 @@ org.apache.arrow arrow-java-root - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT pom Apache Arrow Java Root POM diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 5dbc0b53258d..5cc638017795 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - siren-9.0.0-1-SNAPSHOT + siren-9.0.0-2-SNAPSHOT arrow-vector Arrow Vectors From 9e6326b845da1bf88e808d9d6ef49e3737605341 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Thu, 29 Sep 2022 20:11:31 +0200 Subject: [PATCH 08/10] Update as per review --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 091892327713..e46a87d78638 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,8 @@ run the unit test `CheckAccessibleTest` in `https://github.com/sirensolutions/siren-platform/blob/master/core/src/test/java/io/siren/federate/core/common/CheckAccessibleTest.java`. - Note: the unit test `CheckAccessibleTest` is currently ignored, please set it again to ignore after running the test. - The unit test is ignored because the settings in `CheckAccessibleTest` is not taken into account when the whole unit test suite is run, therefore it fails. - This could be because when the class is loaded, the default settings is used (which is a static block) and the new settings in the `CheckAccessibleTest` is + The unit test is ignored because the setting in `CheckAccessibleTest` is not taken into account when the whole unit test suite is run, therefore it fails. + This could be because when the class is loaded, the default settings are used (which is a static block) and the new settings in the `CheckAccessibleTest` are then not applied when the test suit is run. ## Build From cfd965270c0af9dda02e963f5e5711988d4633d9 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Mon, 3 Oct 2022 08:45:23 +0200 Subject: [PATCH 09/10] Update comment as per review --- .../java/org/apache/arrow/memory/TestBoundaryChecking.java | 2 +- java/vector/pom.xml | 6 ------ 2 files changed, 1 insertion(+), 7 deletions(-) diff --git a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java index 2e423119a54d..5809add14e7e 100644 --- a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java +++ b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java @@ -56,7 +56,7 @@ private boolean getFlagValue(ClassLoader classLoader) throws Exception { /** * Siren: Ensure the flag for bounds checking is disabled by default. - * This will protect users from JVM crashes. + * Enabling it will protect users from JVM crashes. */ @Test public void testDefaultValue() throws Exception { diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 5cc638017795..db51e380e21b 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -124,12 +124,6 @@ test - From ff136ed47f2a29c2f6296e9a7d9fc227b090edf8 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Wed, 12 Oct 2022 12:47:42 +0200 Subject: [PATCH 10/10] Use stable netty version siren-4.1.78-1 --- java/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/java/pom.xml b/java/pom.xml index f79a74feb0ef..bbce96394b32 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -33,7 +33,7 @@ 5.4.0 1.7.25 30.1.1-jre - siren-4.1.78-1-SNAPSHOT + siren-4.1.78-1 1.47.0 3.21.2 2.13.2.20220328