GDB 命令调试 observer 的 coredump 时无法获取堆栈的符号表信息

适用版本

• OceanBase 数据库所有版本。

• 低于 V9 版本的 GDB。

问题现象

GDB 调试 observer 的 coredump 时遇到 unable to initialize decompress status for section .debug_info，无法打印堆栈的符号表信息。

使用 GDB 命令去调试 observer 的 coredump 时无法获取堆栈的符号表信息。

For help, type "help".Type "apropos word" to search for commands related to "word"...Reading symbols from /home/admin/oceanbase/bin/observer...BFD: /usr/lib/debug/home/admin/oceanbase/bin/observer.debug: unable to initialize decompress status for section .debug_infoBFD: /usr/lib/debug/home/admin/oceanbase/bin/observer.debug: unable to initialize decompress status for section .debug_info​warning: File "/usr/lib/debug/home/admin/oceanbase/bin/observer.debug" has no build-id, file skippedBFD: /usr/lib/debug/home/admin/oceanbase/bin/observer.debug: unable to initialize decompress status for section .debug_infoBFD: /usr/lib/debug/home/admin/oceanbase/bin/observer.debug: unable to initialize decompress status for section .debug_info`/usr/lib/debug/home/admin/oceanbase/bin/observer.debug': can't read symbols: file format not recognized.[New LWP 1660196][New LWP 1662810]​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments[Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segments​warning: Loadable section ".note.gnu.property" outside of ELF segmentsCore was generated by `./bin/observer'.Program terminated with signal SIGSEGV, Segmentation fault.#0  0x00007f6cd03f5c6f in raise () from /lib64/libpthread.so.0[Current thread is 1 (Thread 0x7f6cd12dec40 (LWP 1660196))]Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-3.uelc20.x86_64 cyrus-sasl-lib-2.1.27-5.uelc20.x86_64 expat-2.2.6-2.uelc20.4.x86_64 glibc-2.28-102.uelc20.08.x86_64 keyutils-libs-1.5.10-8.uelc20.1.x86_64 krb5-libs-1.18.2-8.uelc20.1.x86_64 libaio-0.3.112-3.uelc20.1.x86_64 libcom_err-1.45.4-5.uelc20.1.x86_64 libcurl-7.61.1-18.uelc20.01.x86_64 libidn2-2.2.0-3.uelc20.1.x86_64 libnghttp2-1.41.0-5.uelc20.1.x86_64 libpsl-0.20.2-7.uelc20.1.x86_64 libselinux-2.9-3.uelc20.1.x86_64 libssh-0.9.4-2.uelc20.02.x86_64 openldap-2.4.46-13.uelc20.2.x86_64 openssl-libs-1.1.1c-18.uelc20.1.x86_64 pcre2-10.32-2.uelc20.x86_64 sssd-client-2.2.3-22.uelc20.01.x86_64 zlib-1.2.11-16.uelc20.1.x86_64(gdb) bt#0  0x00007f6cd03f5c6f in raise () from /lib64/libpthread.so.0#1  0x000000000de74a16 in ?? ()#2  0x0005f4a0007a1764 in ?? ()#3  0x0000000000000000 in ?? ()(gdb) bt#0  0x00007f6cd03f5c6f in raise () from /lib64/libpthread.so.0#1  0x000000000de74a16 in ?? ()#2  0x0005f4a0007a1764 in ?? ()#3  0x0000000000000000 in ?? ()(gdb) quit

问题原因

OceanBase 数据库 V2.2.76 之后版本，使用 V7.11 版本的 GDB 读取 debuginfo 可能会出现 unable to initialize decompress status for section，需要升级 GDB 到 V9.1 版本。

升级 GDB 的方法

手工将 GDB 升级到最新的 GDB 版本，比如 GDB V9.1。

1. 下载 GDB V9.1 的安装包。

   [root@localhost ~]# wget https://ftp.gnu.org/gnu/gdb/gdb-9.1.tar.gz--no-check-certificate
   

2. 解压下载的安装包。

   [root@localhost ~]# tar zxvf gdb-9.1.tar.gz
   

3. 切换到 gdb-9.1 文件夹。

   [root@localhost ~]# cd gdb-9.1/
   

4. 查看 gdb-9.1 文件夹内容。

   [root@localhost gdb-9.1]# ls
   

5. 新建 build 文件夹。

   [root@localhost gdb-9.1]# mkdir build
   

6. 切换到 build 文件夹。

   [root@localhost gdb-9.1]# cd build/
   

7. 运行 configure 命令。

   [root@localhost build]# ../configure
   

8. 运行 make 命令。

   [root@localhost build]# make
   

9. 安装 GDB。

   [root@localhost build]# make install
   

10. 查看 GDB 安装目录。

    [root@localhost build]# which gdb
    /usr/bin/gdb
    

11. 查看 GDB 是否更新成功。

    [root@localhost build]# gdb --version
    GNU gdb (GDB) 9.1
    Copyright (C) 2020 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    

FAQ

• 如果 make 命令遇到下面的错误，需要先安装 texinfo 包：# yum -y install texinfo。

  /root/gdb-9.1/missing: line 81: makeinfo: command not found
  WARNING: 'makeinfo' is missing on your system.
  You should only need it if you modified a '.texi' file, or
  any other file indirectly affecting the aspect of the manual.
  You might want to install the Texinfo package:
  <http://www.gnu.org/software/texinfo/>
  The spurious makeinfo call might also be the consequence of
  using a buggy 'make' (AIX, DU, IRIX), in which case you might
  want to install GNU make:
  <http://www.gnu.org/software/make/>
  make[4]: *** [Makefile:490: gdb.info] Error 127
  make[4]: Leaving directory '/root/gdb-9.1/build/gdb/doc'
  make[3]: *** [Makefile:2010: subdir_do] Error 1
  make[3]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[2]: *** [Makefile:1655: all] Error 2
  make[2]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[1]: *** [Makefile:9564: all-gdb] Error 2
  make[1]: Leaving directory '/root/gdb-9.1/build'
  make: *** [Makefile:852: all] Error 2
  Ref: http://blog.chinaunix.net/uid-13627944-id-4028431.html
  

• 如果 make 命令遇到如下的报错，需要先安装 Locale::Messages module。

  Can't locate Locale/Messages.pm in @INC (you may need to install the Locale::Messages module) (@INC contains: /usr/lib64/texinfo /usr/share/texinfo /usr/share/texinfo /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /usr/bin/makeinfo line 109.
  BEGIN failed--compilation aborted at /usr/bin/makeinfo line 109.
  make[4]: *** [Makefile:490: gdb.info] Error 2
  make[4]: Leaving directory '/root/gdb-9.1/build/gdb/doc'
  make[3]: *** [Makefile:2010: subdir_do] Error 1
  make[3]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[2]: *** [Makefile:1655: all] Error 2
  make[2]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[1]: *** [Makefile:9564: all-gdb] Error 2
  make[1]: Leaving directory '/root/gdb-9.1/build'
  make: *** [Makefile:852: all] Error 2
  

  解决方法：

  [root@localhost ~]# wget https://cpan.metacpan.org/authors/id/G/GU/GUIDO/libintl-perl-1.33.tar.gz
  [root@localhost ~]# tar zxvf libintl-perl-1.33.tar.gz
  [root@localhost ~]# cd libintl-perl-1.33/
  [root@localhost libintl-perl-1.33]# perl Makefile.PL
  [root@localhost libintl-perl-1.33]# make
  [root@localhost libintl-perl-1.33]# make install
  

• 如果 make 命令遇到如下的报错，需要先安装 Text::Unidecode module。

  Can't locate Text/Unidecode.pm in @INC (you may need to install the Text::Unidecode module) (@INC contains: /usr/lib64/texinfo /usr/share/texinfo /usr/share/texinfo /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /usr/share/texinfo/Texinfo/Convert/NodeNameNormalization.pm line 29.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/NodeNameNormalization.pm line 29.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Parser.pm line 59.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Parser.pm line 59.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Report.pm line 51.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Report.pm line 51.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Convert/Converter.pm line 32.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/Converter.pm line 32.
  Compilation failed in require at /usr/bin/makeinfo line 111.
  BEGIN failed--compilation aborted at /usr/bin/makeinfo line 111.
  make[4]: *** [Makefile:490: gdb.info] Error 2
  make[4]: Leaving directory '/root/gdb-9.1/build/gdb/doc'
  make[3]: *** [Makefile:2010: subdir_do] Error 1
  make[3]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[2]: *** [Makefile:1655: all] Error 2
  make[2]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[1]: *** [Makefile:9564: all-gdb] Error 2
  make[1]: Leaving directory '/root/gdb-9.1/build'
  make: *** [Makefile:852: all] Error 2
  

  解决方法：

  [root@localhost ~]# wget https://cpan.metacpan.org/authors/id/S/SB/SBURKE/Text-Unidecode-1.30.tar.gz
  [root@localhost ~]# tar zxvf Text-Unidecode-1.30.tar.gz
  [root@localhost ~]# cd Text-Unidecode-1.30/
  [root@localhost Text-Unidecode-1.30]# perl Makefile.PL
  [root@localhost Text-Unidecode-1.30]# make
  [root@localhost Text-Unidecode-1.30]# make install
  

• 如果 make 命令遇到如下的报错，需要先安装 Unicode::EastAsianWidth。

  Can't locate Unicode/EastAsianWidth.pm in @INC (you may need to install the Unicode::EastAsianWidth module) (@INC contains: /usr/lib64/texinfo /usr/share/texinfo /usr/share/texinfo /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /usr/share/texinfo/Texinfo/Convert/Unicode.pm line 31.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/Unicode.pm line 31.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Convert/Text.pm line 28.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/Text.pm line 28.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Convert/NodeNameNormalization.pm line 33.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/NodeNameNormalization.pm line 33.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Parser.pm line 59.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Parser.pm line 59.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Report.pm line 51.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Report.pm line 51.
  Compilation failed in require at /usr/share/texinfo/Texinfo/Convert/Converter.pm line 32.
  BEGIN failed--compilation aborted at /usr/share/texinfo/Texinfo/Convert/Converter.pm line 32.
  Compilation failed in require at /usr/bin/makeinfo line 111.
  BEGIN failed--compilation aborted at /usr/bin/makeinfo line 111.
  make[4]: *** [Makefile:490: gdb.info] Error 2
  make[4]: Leaving directory '/root/gdb-9.1/build/gdb/doc'
  make[3]: *** [Makefile:2010: subdir_do] Error 1
  make[3]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[2]: *** [Makefile:1655: all] Error 2
  make[2]: Leaving directory '/root/gdb-9.1/build/gdb'
  make[1]: *** [Makefile:9564: all-gdb] Error 2
  make[1]: Leaving directory '/root/gdb-9.1/build'
  make: *** [Makefile:852: all] Error 2
  

  解决方法：

  [root@localhost ~]# wget https://cpan.metacpan.org/authors/id/A/AU/AUDREYT/Unicode-EastAsianWidth-12.0.tar.gz
  [root@localhost ~]# tar zxvf Unicode-EastAsianWidth-12.0.tar.gz
  [root@localhost ~]# cd Unicode-EastAsianWidth-12.0/
  [root@localhost Unicode-EastAsianWidth-12.0]# perl Makefile.PL
  [root@localhost Unicode-EastAsianWidth-12.0]# make
  [root@localhost Unicode-EastAsianWidth-12.0]# make install
  ​```