Linux監控工具介紹系列——OSWatcher Black Box，oswatcherbox

Linux監控工具介紹系列——OSWatcher Black Box，oswatcherbox

 

OSWatcher Balck Box簡介

 

OSWatcher Black Box (oswbb)是Oracle開發、提供的一個小巧，但是實用、強大的系統工具，它可以用來抓取操作系統的性能指標，用於輔助監控系統的資源使用。其安裝部署、卸載都非常簡單；資源消耗也比較小，原理也十分簡單，它通過調用OS的的一些命令（例如vmstat、iostat等）來采集、存儲CPU/Memory/Swap/Disk IO/Nentwork相關數據。安裝和運行oswbba可以幫助在性能診斷時提供豐富多樣的各類性能數據、圖文報表支持。

 

OSWatcher 在4.0的版本時被命名為OSWatcher Black Box,簡稱為oswbb，同時增加了數據分析功能，即OSWatcher Black Box Analyzer (OSWbba)這個繪圖和分析工具，其捆綁在 OS Watcher Black Box當中。替代了之前的OSWg。 也就是說在OSWatcher 4.0 之前是：OSWatcher 和 OSWg的關系，OSWatcher 4.0 後變成了： OSWbb 與 OSWbba 的關系。

 

OSWatcher Black Box（oswbb）支持多個操作系統，也分Linux與Window版本，當然這兩個版本有所差別，本篇只講述Linux版本。另外，OSWatcher Black Box（oswbb）由兩個部分組成：

 

1. oswbb: 一個Unix的 shell script腳本集合，其用來收集和歸檔數據，從而幫助定位問題。

 

2. oswbba: 一個Java工具來自動分析數據，提供建議，並且生成一個包含圖形的 html 文檔。

 

OSWatcher Black Box（oswbb）的官方下載地址以及相關資料如下（Oracle Metalink上的資料需要賬號）

 

How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service(文檔 ID 580513.1)

OSWatcher Analyzer User Guide (文檔 ID 461053.1)

 

 

    官方介紹文檔對oswbb，oswbba的介紹如下：

 

OSWatcher (oswbb) is a collection of UNIX shell scripts intended to collect and archive operating system and network metrics to aid support in diagnosing performance issues. OSWatcher operates as a set of background processes on the server and gathers OS data on a regular basis, invoking such Unix utilities as vmstat, netstat and iostat. OSWatcher can be downloaded from this note. OSWatcher is also included in the RAC-DDT script file, but is not installed by RAC-DDT. For more information on RAC-DDT see RAC-DDT User Guide. OSWatcher is installed on each node where data is to be collected. Installation instructions for OSWatcher are provided in this user guide.

 

The OSWatcher Analyzer (oswbba) is a graphing and analysis utility which comes bundled with OSWatcher v4.0 and higher. oswbba allows the user to graphically display data collected, generate reports containing these graphs and provides a built in analyzer to analyze the data and provide details on any performance problems it detects. The ability to graph and analyze this information relieves the user of manually inspecting all the files.

 

NOTE: oswbba replaces the utility OSWg. This was done to eliminate the confusion caused by having multiple tools in support named OSWatcher. oswbba is only supported for data collected by oswbb and no other tool.

 

 

安裝OSWatcher

 

Linux平台的安裝簡單到了不能再簡單，如下所示，解壓安裝包生成oswbb文件夾。當然你最好將其部署或移動到合適的目錄。

 

[oracle@DB-Server monitoring]$ tar -xvf oswbb734.tar

[oracle@DB-Server monitoring]$ ls -lrt

total 6196

drwxr-xr-x 6 oracle oinstall    4096 Jul 25 22:22 oswbb

-rw-r--r-- 1 oracle oinstall 6318080 Nov  8 02:33 oswbb734.tar

[oracle@DB-Server monitoring]$ cd oswbb

[oracle@DB-Server oswbb]$ ls –lrt

 

 

 

卸載OSWatcher

 

 

卸載OSWatche也是非常簡單，只要稍懂Linux的應該都會。

 

1：卸載前先停止OSWatcher

 

./stopOSWbb.sh

 

2：刪除OSWatcher的目錄oswbb

 

   rm –fr oswbb

 

 

 

啟動OSWatcher

 

啟動OSWatcher也非常簡單，只需要執行startOSWbb.sh腳本即可，如下所示

 

 

  ./startOSWbb.sh 10  2

 

後面參數表示10秒采集一次數據，只保留最後采集2個小時的數據在歸檔文件中。首次啟動的日志最好做一下檢查，注意是否存在Warning信息，如下測試所示，檢查時發現ifconfig命令找不到（oracle賬戶下無法運行ifconfig命令）

[oracle@DB-Server oswbb]$ ./startOSWbb.sh 10 2

[oracle@DB-Server oswbb]$ Setting the archive log directory to/home/oracle/monitoring/oswbb/archive

Testing for discovery of OS Utilities...

VMSTAT found on your system.

IOSTAT found on your system.

MPSTAT found on your system.

Warning... IFCONFIG not found on your system. No IFCONFIG data will be collected.

NETSTAT found on your system.

TOP found on your system.

Testing for discovery of OS CPU COUNT

oswbb is looking for the CPU COUNT on your system

CPU COUNT will be used by oswbba to automatically look for cpu problems

CPU COUNT found on your system.

CPU COUNT = 4

Discovery completed.

Starting OSWatcher v7.3.3  on Sun Dec 4 08:01:57 EST 2016

With SnapshotInterval = 10

With ArchiveInterval = 2

OSWatcher - Written by Carl Davis, Center of Expertise,

Oracle Corporation

For questions on install/usage please go to MOS (Note:301137.1)

If you need further assistance or have comments or enhancement

requests you can email me [email protected]

Data is stored in directory: /home/oracle/monitoring/oswbb/archive

Starting Data Collection...

oswbb heartbeat:Sun Dec 4 08:02:02 EST 2016

oswbb heartbeat:Sun Dec 4 08:02:12 EST 2016

 

 ./startOSWbb.sh

 

如果沒有指定參數，默認參數為30,48，意味著30秒采集一次數據，只保留最後48小時的數據到歸檔文件當中。其實startOSWbb.sh可以定義四個參數

 

參數1： 指定多少秒采集一次數據。

參數2： 指定采集的數據文件在歸檔路徑保留多少個小時

參數3： 可選參數，打包壓縮工具， 在完成收集後OSW 將使用其來打包壓縮歸檔文件

參數4： 可選參數，指定采集歸檔數據的輸出目錄，默認為系統變量OSWBB_ARCHIVE_DEST的值。

 

 

 

使用上面方式啟動OSWatcher，會被輸出信息一直刷屏。基本上很少使用這種方式，一般使用nohup啟動，這樣可以讓OSW能夠在後台持續運行並在當前會話終止後不會被掛斷。當然如果系統重啟，那麼OSWatcher就會停止。

 

 nohup ./startOSWbb.sh 30 48 &

 

 

第一次啟動OSWatcher會在oswbb目錄下創建 gif、archive、tmp、locks目錄。其歸檔文件夾和osw<工具名> 子文件夾會被建

 

 

 

OSWatcher采集的數據都存放在archive（目錄路徑）下，下面會生成oswiostat、oswmeminfo、oswmpstat等10個目錄

 

[oracle@DB-Server archive]$ tree

.

|-- oswifconfig

|-- oswiostat

|   |-- DB-Server.localdomain_iostat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_iostat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_iostat_16.12.09.1500.dat

|-- oswmeminfo

|   |-- DB-Server.localdomain_meminfo_16.12.09.1300.dat

|   |-- DB-Server.localdomain_meminfo_16.12.09.1400.dat

|   `-- DB-Server.localdomain_meminfo_16.12.09.1500.dat

|-- oswmpstat

|   |-- DB-Server.localdomain_mpstat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_mpstat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_mpstat_16.12.09.1500.dat

|-- oswnetstat

|   |-- DB-Server.localdomain_netstat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_netstat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_netstat_16.12.09.1500.dat

|-- oswprvtnet

|-- oswps

|   |-- DB-Server.localdomain_ps_16.12.09.1300.dat

|   |-- DB-Server.localdomain_ps_16.12.09.1400.dat

|   `-- DB-Server.localdomain_ps_16.12.09.1500.dat

|-- oswslabinfo

|   |-- DB-Server.localdomain_slabinfo_16.12.09.1300.dat

|   |-- DB-Server.localdomain_slabinfo_16.12.09.1400.dat

|   `-- DB-Server.localdomain_slabinfo_16.12.09.1500.dat

|-- oswtop

|   |-- DB-Server.localdomain_top_16.12.09.1300.dat

|   |-- DB-Server.localdomain_top_16.12.09.1400.dat

|   `-- DB-Server.localdomain_top_16.12.09.1500.dat

`-- oswvmstat

    |-- DB-Server.localdomain_vmstat_16.12.09.1300.dat

    |-- DB-Server.localdomain_vmstat_16.12.09.1400.dat

    `-- DB-Server.localdomain_vmstat_16.12.09.1500.dat

 

10 directories, 24 files

 

采集的數據文件命名格式為OSWatcher 輸出文件格式為：<節點名>_<操作系統工具名>_YY.MM.DD.HH24.dat  。如下所示，

 

 

 

配置OSWatcher自啟動

 

OSWatcher在系統重啟過後，是無法自動重啟的，如果需要設置OSWatcher開機自啟動，需要安裝、配置osw-service這個RPM包。這個安裝包可以從How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service (文檔 ID 580513.1)出下載。

 

[root@DB-Server tmp]# rpm -ivh oswbb-service-7.2.0-1.noarch.rpm

Preparing...                ########################################### [100%]

   1:oswbb-service          ########################################### [100%]

[root@DB-Server tmp]#

 

安裝osw-service這個RPM包後，需要配置/etc/oswbb.conf文件後，才能配置OSWatcher自啟動。如下所示：

 

 

[root@DB-Server ~]#

[root@DB-Server ~]# /sbin/chkconfig oswbb on

[root@DB-Server ~]# /sbin/service oswbb start

Starting OSWatcher: [  OK  ]

[root@DB-Server ~]#

 

 

關閉OSWatcher

 

  關閉OSWatcher也是非常簡單，網上有種說法：stopOSWbb.sh 是Oracle唯一支持的用於停止OSW的方法，其實不然，應該說有兩種方法

 

  1：執行腳本stopOSWbb.sh關閉OSWatcher

 

./stopOSWbb.sh

 

  2：kill進程方法 

[root@DB-Server ~]# ps -ef | grep -i OSW

oracle   24863     1  0 16:02 pts/1    00:00:00 /bin/sh ./OSWatcher.sh 30 48

oracle   24904 24863  0 16:03 pts/1    00:00:00 /bin/sh ./OSWatcherFM.sh 48 /home/oracle/monitoring/oswbb/archive

root     25330 18699  0 16:05 pts/2    00:00:00 grep -i osw

[root@DB-Server ~]# kill 24863

[root@DB-Server ~]# kill 24904

[root@DB-Server ~]# ps -ef | grep -i OSW

root     25342 18699  0 16:05 pts/2    00:00:00 grep -i osw

[root@DB-Server ~]# 

 

 

如果你查看stopOSWbb.sh腳本，你會發現它停止OSWatcher也是通過kill OSWatcher program相關進程的方法，如下所示：

 

[oracle@DB-Server oswbb]$ more stopOSWbb.sh 

#!/bin/sh

######################################################################

# stopOSW.sh

# This is the script which terminates all processes associated with

# the OSWatcher program.

######################################################################

# Kill the OSWatcher processes

######################################################################

PLATFORM=`/bin/uname`

case $PLATFORM in

  AIX)

    kill -15 `ps -ef | grep OSWatch | grep -v grep | awk '{print $2}'` 

    ;;

  *)

    kill -15 `ps -e | grep OSWatch | awk '{print $1}'`

    ;;

esac

 

 

OSWatcher bba使用總結

 

OSWatcher now provides an analysis tool oswbba which analyzes the log files produced by OSWatcher. This tool allows OSWatcher to be selfanalyzing.

This tool also provides a graphing capability to graph the data and to produce a html profile. See the "Graphing and Analyzing the Output" section below.

 

oswbba is written in java and requires as a minimum java version 1.4.2 or higher. oswbba can run on any Unix X Windows or PC Windows platform. An X Windows environment is required because oswbba uses Oracle Chartbuilder which requires it.

 

OSWatcher bba 是一個Java語言寫的應用程序，需要安裝Java 1.4.2 或更高的版本。oswbba能夠在任何有X Windows的Unix平台或Windows平台上運行， X Windows環境是必須的，因為oswbba需要用到Oracle Chartbuilder組件，而這個組件需要它。

 

[root@DB-Server oswbb]# java -version

java version "1.8.0_111"

Java(TM) SE Runtime Environment (build 1.8.0_111-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

[root@DB-Server oswbb]# java -jar oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

或

[root@DB-Server oswbb]# java -jar Xmx512M oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

 

如果你只想生成某個時間段的報表，你可以使用參數-B  -E 如下案例所示

 

[root@DB-Server oswbb]#java -jar  -Xmx256m oswbba.jar -i /home/oracle/scripts/oswbb/archive  -B Dec  7 15:30:00 2016  -E Dec 7 17:00:00 2016

 

在處理過程OSWatcher有下面一些參數可供選擇，如下所示。根據具體需要選擇。 生成對應的html報表後，將其下載到本機，你就可以做一些分析了。

 

 

 

如下截圖所示，這些圖表僅僅是其中部分，OSWatcher給人印象最深的就是豐富的圖表，而不是一堆枯燥的數字。

 

 

 

 

 

 

 

 

當然oswbba也能生成一個非常詳細的分析報告，是文本格式，如下部分截圖所示。

 

 

 

其實使用OSWatcher的使用並沒有什麼難度，反而是你要理解報表或分析報告中的指標意義，能夠結合AWR、ASH以及其他數據庫診斷信息來分析、診斷問題。這個才是有難度的。關於具體指標的詳細說明，OSWatcher Analyzer User Guide (文檔 ID 461053.1) 中已有詳細解說，在此就不班門弄斧了。

 

 

 

 

OSWatcher常見應用問題

 

1： 如何判斷OSWatcher是否正在運行

 

    判斷OSWatcher是否正在運行，非常簡單，使用ps命令查看是否存在OSWatcher的相關進程即可。如下所示

 

[oracle@DB-Server ~]$ ps -ef | grep -i OSWatcher

oracle   23532     1  0 08:01 pts/2    00:00:14 /bin/sh ./OSWatcher.sh 10 2

oracle   23587 23532  0 08:02 pts/2    00:00:00 /bin/sh ./OSWatcherFM.sh 2 /home/oracle/monitoring/oswbb/archive

oracle   25808 24564  0 09:22 pts/3    00:00:00 grep --color=auto -i OSWatcher

[oracle@DB-Server ~]$

 

2: 沒有設置JAVA環境變量，會遇到下面錯誤：

 

[oracle@mylnx02 oswbb]$ java -jar oswbba.jar -i /home/oracle/oswbb/archive -B Dec 21 09:00:00 2016  -E Dec 21 10:00:00 2016

Validating times in the archive...

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

ERROR. You do not have a legitimate version of java in your PATH.

Linux users please download and install java from java.sun.com or 

see the oswbba README for instructions on how to use the version of java

that comes shipped with the Oracle database.

如下所示，設置JAVA環境變量後，然後執行oswbba的相關命令即可。

 

[oracle@mylnx02 oswbb]$  export PATH=$ORACLE_HOME/jre/1.4.2/bin:$PATH

[oracle@mylnx02 oswbb]$  java -jar oswbba.jar -i /home/oracle/oswbb/archive -B Dec 21 09:00:00 2016  -E Dec 21 10:00:00 2016

Validating times in the archive...

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...

Parsing file getlnx14.gfg1.esquel.com_iostat_16.12.21.0800.dat ...

Parsing file getlnx14.gfg1.esquel.com_iostat_16.12.21.0900.dat ...

 

 

3：運行"java -jar oswbba.jar ..."命令時報"Exception in thread "main" java.lang.OutOfMemoryError"

 

這個是因為JAVA的堆棧設置太小的緣故，加載到內存中的數據量過於龐大，需要設置參數-Xmx 可以用-Xmx256m或-Xmx512m 如下所示

 

java -jar  -Xmx256m oswbba.jar -i /home/oracle/scripts/oswbb/archive

 

oswbba parses all the archive files in memory prior to generating graphs or performing an analysis. If you have a large amount of files to parse you may need to allocate more memory in the java heap. If you experience any error messages regarding out of memory such as java.lang.OutOfMemoryError, you may have to increase the size of the java heap. To increase the size of the java heap use the -Xmx flag.

 

 

4：oswbba是否可以在命令行界面生成html報告文件？ 答案是可以，當然你會遇到很多雜七雜八問題，下面列舉了一些我遇到的問題

 

1： ”No X11 DISPLAY variable was set, but this program performed an operation which requires it“

 

 

 

如果是root賬號，只要執行export DISPLAY=:0.0命令後，即可在SecureCRT或 命令窗口生成報表文件。如下所示：

 

[root@DB-Server oswbb]# export DISPLAY=:0.0

[root@DB-Server oswbb]# java -jar oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...

Parsing file DB-Server.localdomain_iostat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_iostat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_vmstat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_vmstat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_netstat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_netstat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_top_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_top_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_ps_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_ps_16.12.04.0900.dat ...

Parsing Completed.

Enter 1 to Display CPU Process Queue Graphs

Enter 2 to Display CPU Utilization Graphs

Enter 3 to Display CPU Other Graphs

Enter 4 to Display Memory Graphs

Enter 5 to Display Disk IO Graphs

Enter 6 to Generate All CPU Gif Files

Enter 7 to Generate All Memory Gif Files

Enter 8 to Generate All Disk Gif Files

Enter L to Specify Alternate Location of Gif Directory

Enter T to Alter Graph Time Scale Only (Does not change analysis dataset)

Enter D to Return to Default Graph Time Scale

Enter R to Remove Currently Displayed Graphs

Enter A to Analyze Data

Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)

Enter P to Generate A Profile

Enter X to Export Parsed Data to File

Enter Q to Quit Program

Please Select an Option:1

Enter 1 to Display CPU Process Queue Graphs

Enter 2 to Display CPU Utilization Graphs

Enter 3 to Display CPU Other Graphs

Enter 4 to Display Memory Graphs

Enter 5 to Display Disk IO Graphs

Enter 6 to Generate All CPU Gif Files

Enter 7 to Generate All Memory Gif Files

Enter 8 to Generate All Disk Gif Files

Enter L to Specify Alternate Location of Gif Directory

Enter T to Alter Graph Time Scale Only (Does not change analysis dataset)

Enter D to Return to Default Graph Time Scale

Enter R to Remove Currently Displayed Graphs

Enter A to Analyze Data

Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)

Enter P to Generate A Profile

Enter X to Export Parsed Data to File

Enter Q to Quit Program

Please Select an Option:P

Enter a unique profile directory name or enter <CR> to accept default name:kkk.html

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Run_Queue.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Block_Queue.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Idle.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_System.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_User.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Wa.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Interrupts.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Context_Switches.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Swap.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Free.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Page_In_Rate.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_ST.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_RPS.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_WPS.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PB.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_1.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_2.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_3.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_TPS.gif

 

上面使用的是root用戶，如果是oracle用戶，還是有可能還會遇到下面問題

 

 

 

此時需要切換到root賬號，執行下面命令，然後切換到oracle用戶下執行命令

 

[oracle@DB-Server oswbb]$ su - root

Password: 

[root@DB-Server ~]# export DISPLAY=:0.0

[root@DB-Server ~]# xhost local:oracle

non-network local connections being added to access control list

 

 

 

當然，在SecureCRT中，你不能選擇1-5這幾個選項，否則依然會遇到 Can't connect to X11 window server using ':0.0'這個錯誤。這個只能在圖形化界面才能執行。例如通過VNC來生成。

 

 

 

java -jar  -Xmx256m oswbba.jar -i /home/oracle/oswbb/archive -B Mar 5 16:00:00 2016  -E Mar 5 16:30:00 2016

 

java -jar  -Djava.awt.headless=true -Xmx256m oswbba.jar -i /home/oracle/oswbb/archive -B Mar 5 16:00:00 2016  -E Mar 5 16:30:00 2016

 

 

 

 

參考資料：

 

OSWatcher（包括：[視頻]） (文檔 ID 1526578.1)

OSWatcher (Includes: [Video]) (文檔 ID 301137.1)

How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service(文檔 ID 580513.1)

OSWatcher Analyzer User Guide (文檔 ID 461053.1)

 

http://xxxxxx/Linuxjc/1181641.html TechArticle