Athena で読める ORC ファイルをクエリを使わずに生成したいです
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
というフォーマットでおそらく
https://github.com/apache/orc
これを使えばいいと思い
git clone https://github.com/apache/orc.git
cmake -S orc -B orc/build -DBUILD_JAVA=OFF
cd orc/build
make package
make test-out
を実行したところ
~/homebrew/lib
~/homebrew/include
にいろいろ入りました
https://orc.apache.org/docs/core-cpp.html#writing-orc-files
次にこのサンプルを動かそうと以下の様な宣言部分だけかいてコンパイルしようとしました
main.cpp
#include <orc/OrcFile.hh>
int main() {
ORC_UNIQUE_PTR<orc::OutputStream> outStream = orc::writeLocalFile("my-file.orc");
ORC_UNIQUE_PTR<orc::Type> schema(
orc::Type::buildTypeFromString("struct<x:int,y:int>"));
orc::WriterOptions options;
ORC_UNIQUE_PTR<orc::Writer> writer =
createWriter(*schema, outStream.get(), options);
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
project(orc-test)
set (CMAKE_CXX_STANDARD 11)
add_executable(orc-test main.cpp)
とかいて cmake . は通ったんですが make すると
fatal error: 'orc/OrcFile.hh' file not found
といわれます
cmake を初めて使ったのでインクルードやリンクを自動的にやってくれるものかと思ってたんですが、そういうものではないんでしょうか
cmake_minimum_required(VERSION 2.8)
project(orc-test CXX)
set (CMAKE_CXX_STANDARD 11)
include_directories(~/homebrew/include)
add_executable(orc-test main.cpp)
というのを追加してみたところ main.cpp のコンパイルは通ったみたいでリンカーまでいったんですが
[ 50%] Building CXX object CMakeFiles/orc-test.dir/main.cpp.o
[100%] Linking CXX executable orc-test
Undefined symbols for architecture x86_64:
"orc::createWriter(orc::Type const&, orc::OutputStream*, orc::WriterOptions const&)", referenced from:
_main in main.cpp.o
"orc::WriterOptions::WriterOptions()", referenced from:
_main in main.cpp.o
"orc::WriterOptions::~WriterOptions()", referenced from:
_main in main.cpp.o
"orc::writeLocalFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)", referenced from:
_main in main.cpp.o
"orc::Type::buildTypeFromString(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)", referenced from:
_main in main.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [orc-test] Error 1
make[1]: *** [CMakeFiles/orc-test.dir/all] Error 2
make: *** [all] Error 2
というよくわからないエラーになりました
orc のインストールの時
share/doc/orc/LICENSE
share/doc/orc/NOTICE
と lib/ に *.a, include/ に *.hh がインストールされただけで、
cmake のコンフィグっぽいファイルはインストールされてるようにみえないんですが、
この cmake でビルドした orc を使うには CMakeLists.txt にどうかけばいいんでしょうか?
昔うろ覚えなんですが aws-sdk-c++ を cmake でいれたときは
~/homebrew/lib/cmake/
に .cmake というファイルができてて
プロジェクト内の cmake ファイルもたいしたことかかなくても自動でインクルードやリンクをしてくれた記憶があるんですが…
追記です
CMakeList を以下の様に書いてみたところ
cmake_minimum_required(VERSION 2.8)
project(orc-test CXX)
set (CMAKE_CXX_STANDARD 11)
include_directories(~/homebrew/include)
link_directories(~/homebrew/lib)
target_link_libraries(~/homebrew/lib/liborc.a)
add_executable(orc-test main.cpp)
以下の様な警告が表示されます
その後 make の結果は変わらず同じエラーになります
CMake Warning (dev) at CMakeLists.txt:6 (target_link_libraries):
Cannot specify link libraries for target "~/homebrew/lib/liborc.a" which is
not built by this project.
CMake does not support this but it used to work accidentally and is being
allowed for compatibility.
Policy CMP0016 is not set: target_link_libraries() reports error if its
only argument is not a target. Run "cmake --help-policy CMP0016" for
policy details. Use the cmake_policy command to set the policy and
suppress this warning.
This warning is for project developers. Use -Wno-dev to suppress it.
さらに追記です
cmake_minimum_required(VERSION 2.8)
project(orc-test CXX)
set (CMAKE_CXX_STANDARD 11)
include_directories(~/homebrew/include)
link_directories(~/homebrew/lib)
add_executable(orc-test main.cpp)
target_link_libraries(orc-test liborc)
CMake を上の様に書いたところエラーが以下の様になりました
[ 50%] Linking CXX executable orc-test
ld: library not found for -lliborc
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [orc-test] Error 1
make[1]: *** [CMakeFiles/orc-test.dir/all] Error 2
make: *** [all] Error 2
あと気になるのはインストールされた
~/home/brew/lib/liborc.a
に実行権限がついてなかったです
chmod で実行権限をつけてもエラーはかわりませんでした
ls -l ~/homebrew/lib/liborc.a
-rw-r--r-- 1 xxxxx xxxxx 21913912 2 18 23:07 /Users/xxxxx/homebrew/lib/liborc.a
でファイルは間違いなく存在してます
追記3
cmake_minimum_required(VERSION 2.8)
project(orc-test CXX)
set (CMAKE_CXX_STANDARD 11)
find_package(Threads REQUIRED)
include_directories(~/homebrew/include)
link_directories(~/homebrew/lib)
add_executable(orc-test main.cpp)
target_link_libraries(orc-test
orc
snappy
protobuf
protoc
lz4
zstd
z
Threads::Threads
)
こうかえてみたんですがやはりエラーになります
Undefined symbols for architecture x86_64:
"hdfs::FileSystem::New(hdfs::IoService*&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, hdfs::Options const&)", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::ConfigParser::LoadDefaultResources()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::ConfigParser::ConfigParser()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::ConfigParser::~ConfigParser()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::parse_from_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::URI()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::Options::Options()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::StatInfo::StatInfo()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::IoService::New()", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::ConfigParser::get_options(hdfs::Options&) const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::ConfigParser::ValidateResources() const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::str(bool) const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::get_host(bool) const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::get_path(bool) const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::get_port() const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::URI::has_port() const", referenced from:
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
"hdfs::Status::ToString() const", referenced from:
orc::HdfsFileInputStream::read(void*, unsigned long long, unsigned long long) in liborc.a(OrcHdfsFile.cc.o)
orc::HdfsFileInputStream::HdfsFileInputStream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) in liborc.a(OrcHdfsFile.cc.o)
ld: symbol(s) not found for architecture x86_64
エラー内容は同じなんですがエラーを起こす場所
orc::createWriter ではなく hdfs::FileSystem にかわりました
そもそもこんなライブラリが使われてることはどうすればわかるんでしょうか…
うまくいったときのコンパイルコマンドを教えていただけないでしょうか
/Library/Developer/CommandLineTools/usr/bin/c++ -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L$HOME/homebrew/opt/bzip2/lib -L$HOME/homebrew/opt/zlib/lib -L$HOME/homebrew/opt/mysql/lib -L$HOME/homebrew/opt/libxml2/lib -L$HOME/homebrew/opt/sqlite/lib -L$HOME/homebrew/opt/readline/lib -L$HOME/homebrew/opt/openssl/lib -L$HOME/homebrew/opt/libffi/lib -L$HOME/homebrew/opt/gettext/lib CMakeFiles/orc-test.dir/main.cpp.o -o orc-test -L$HOME/homebrew/lib -Wl,-rpath,$HOME/homebrew/lib -lorc -lsnappy -lprotobuf -lprotoc -llz4 -lzstd -lz
自分の環境ではこういうコマンドでコンパイルしようとしています
.zshrc に余計なことをかいてしまってるせいなんでしょうか