linux json解析工具jq
# linux json 解析工具 jq
本文讲述 Linux 系统的 json 解析工具 jq 的用法。我们要临时解析一段 json 文本,通过需要手动开发程序,实际上没有必要, 因为有更加方便的方法,linux 系统已经提供了 json 解析工具,而且功能很强大。若您的日志是 json 格式写入的,那么利用这款工具,解析日志就会达到事半功倍的效果。
下载地址: jq (opens new window)。
# 安装 jq
# 通用安装方法
提示
国内访问官方提供的下载地址, 下载速度特别慢,不建议使用该方法。
建议查看后续章节, 根据不同的系统, 使用软件仓库安装。
此方法适用于所有 linux 版本, 只要把二进制程序下载到本地, 即可使用。
请前往官方文档 (opens new window), 查看下载地址。
# ubuntu 安装 jq
对于 ubuntu 系统, jq 存在于默认仓库中, 安装更为方便,执行如下命令即可。
sudo apt-get install jq
# centos 安装 jq
# 1. 安装 EPEL 软件仓库
yum install epel-release
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# yum install epel-release
已加载插件:fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.163.com
解决依赖关系
--> 执行事务检查
---> Package epel-release.noarch 0:6-8 will be 安装
--> 完成依赖关系计算
依赖关系解决
===================================================================================================================================================================
软件包 架构 版本 仓库 大小
===================================================================================================================================================================
正在安装:
epel-release noarch 6-8 extras 14 k
事务概要
===================================================================================================================================================================
Install 1 Package(s)
总下载量:14 k
Installed size: 22 k
确定吗?[y/N]:y
下载软件包:
epel-release-6-8.noarch.rpm | 14 kB 00:00
运行 rpm_check_debug
执行事务测试
事务测试成功
执行事务
Warning: RPMDB altered outside of yum.
正在安装 : epel-release-6-8.noarch 1/1
Verifying : epel-release-6-8.noarch 1/1
已安装:
epel-release.noarch 0:6-8
完毕!
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 2. 安装 jq
yum install jq -y
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# yum install jq -y
已加载插件:fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.aliyun.com
base | 3.7 kB 00:00
base/primary_db | 4.7 MB 00:00
epel | 4.7 kB 00:00
epel/primary_db | 6.1 MB 00:03
extras | 3.4 kB 00:00
extras/primary_db | 29 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 11 MB 00:05
解决依赖关系
--> 执行事务检查
---> Package jq.x86_64 0:1.3-2.el6 will be 安装
--> 完成依赖关系计算
依赖关系解决
===================================================================================================================================================================================
软件包 架构 版本 仓库 大小
===================================================================================================================================================================================
正在安装:
jq x86_64 1.3-2.el6 epel 94 k
事务概要
===================================================================================================================================================================================
Install 1 Package(s)
总下载量:94 k
Installed size: 230 k
下载软件包:
jq-1.3-2.el6.x86_64.rpm | 94 kB 00:00
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
Userid : EPEL (6) <epel@fedoraproject.org>
Package: epel-release-6-8.noarch (@extras)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
运行 rpm_check_debug
执行事务测试
事务测试成功
执行事务
正在安装 : jq-1.3-2.el6.x86_64 1/1
Verifying : jq-1.3-2.el6.x86_64 1/1
已安装:
jq.x86_64 0:1.3-2.el6
完毕!
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# jq 常用操作
# json 格式化
- jq 可实现 json 文本的格式化, 详细过程如下
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json
{"level":"INFO","time":"2020-08-06 20:49:59.132","caller":"controller/beauty_controller.go:109","message":"添加图片文件的disable后缀, 完成","serviceName":"beauty2","filePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg","newFilePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"}
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json | jq .
{
"newFilePath": "/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable",
"filePath": "/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg",
"serviceName": "beauty2",
"message": "添加图片文件的disable后缀, 完成",
"caller": "controller/beauty_controller.go:109",
"time": "2020-08-06 20:49:59.132",
"level": "INFO"
}
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 根据 key 解析出 value 值
语法: jq .key 的名称
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json
{"level":"INFO","time":"2020-08-06 20:49:59.132","caller":"controller/beauty_controller.go:109","message":"添加图片文件的disable后缀, 完成","serviceName":"beauty2","filePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg","newFilePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"}
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json | jq .newFilePath
"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 解析出指定的一组
json属性
先查看下内容完整的一行日志
[root@bj-s-21-v10 ~]# grep "用户访问日志, 完成" /data/log/newsBack/info.log | head -n 2
{"level":"INFO","time":"2020-12-10 09:35:41.579","caller":"middleware/api_log_middleware.go:23","message":"用户访问日志, 完成","serviceName":"news-back","uri":"/api/distributor/back/findByCondition?titleLike=","referer":"http://172.18.100.177:8004/distributor/distributorConfig","ip":"172.18.100.177","ipLocation":"局域网 对方和您在同一内部网","duration":"8.513198ms"}
{"level":"INFO","time":"2020-12-10 09:35:44.781","caller":"middleware/api_log_middleware.go:23","message":"用户访问日志, 完成","serviceName":"news-back","uri":"/api/distributor/back/getRevisitRedirectConfig?did=1134019964852768768","referer":"","ip":"172.18.100.177","ipLocation":"局域网 对方和您在同一内部网","duration":"2.454011ms"}
[root@bj-s-21-v10 ~]#
使用逗号,
表示要查询哪些json属性
[root@bj-s-21-v10 ~]# grep "用户访问日志, 完成" /data/log/newsBack/info.log | head -n 2 | jq .time,.message,.duration
"2020-12-10 09:35:41.579"
"用户访问日志, 完成"
"8.513198ms"
"2020-12-10 09:35:44.781"
"用户访问日志, 完成"
"2.454011ms"
[root@bj-s-21-v10 ~]#
# 解析 json 日志
- 批量解析 json 日志文件的内容
对于 json 日志文件,每一行都是一个 json 对象,jq 支持一次性解析多行。
(py3.6) wangshibiao@wangshibiao:~$ head -n 3 /tmp/log/gf-app/2020-11-13.log
{"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"estar接口有返回下载广告","time":"2020-11-13 14:11:02"}
{"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"estar接口有返回广告数据","time":"2020-11-13 14:11:02"}
{"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"完成","time":"2020-11-13 14:11:02"}
(py3.6) wangshibiao@wangshibiao:~$ head -n 3 /tmp/log/gf-app/2020-11-13.log | jq
{
"context": {
"requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo"
},
"level": "info",
"pTitle": "获取estar的视频广告",
"sTitle": "estar接口有返回下载广告",
"time": "2020-11-13 14:11:02"
}
{
"context": {
"requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo"
},
"level": "info",
"pTitle": "获取estar的视频广告",
"sTitle": "estar接口有返回广告数据",
"time": "2020-11-13 14:11:02"
}
{
"context": {
"requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo"
},
"level": "info",
"pTitle": "获取estar的视频广告",
"sTitle": "完成",
"time": "2020-11-13 14:11:02"
}
(py3.6) wangshibiao@wangshibiao:~$
- 对 json 日志文件实时解析
可以实时解析 json 日志。
(py3.6) wangshibiao@wangshibiao:~$ tail -f -n 1 /tmp/log/gf-app/2020-11-13.log | jq
{
"context": {
"requestId": "1nzbby6og30c71wxeew73vn800ore07s"
},
"level": "info",
"pTitle": "跟踪estar事件接口",
"para": {
"eventType": "xad_impression",
"platform": "android",
"adId": 170,
"imei": "869941043279996",
"slotId": "cHJpb3JpdHlfYWRfc2xvdA==",
"action": 5
},
"query": "event_type=xad_impression&package_name=com.zhangshangjianzhi.newapp&scop=None&userId=1315171366231&campaign_id=171&osv=0.1&isPlugin=1&git_last_time=20200318201149&box_pkg_name=c.l.a&ad_system=priority_ad&version_name=5.6.0.5&m_name=rtb&yid=1315171366231&cln=adx-14&deep_link=0&m_version=support_qq_public&total_ram=3GB&cpu_core_num=8&make=meizu&lon=-1&dsp=adx&platform=android&sdk_version=None&ad_user_id=105&time_stamp=20200320135422&device_id=6205ea67-ad97-3c34-97df-da5d3a9fc4e8&imsi&fp=-1&ad_id=170&sd_slot_id=103&city=%E5%86%85%E7%BD%91IP&brand=alps&androidid=81b4f0b18d137a02&des_pla=0&mac=02%3A00%3A00%3A00%3A00%3A00&hn=down.doumistatic.com&autp=7&imei=869941043279996&rp=-1&indus=ep&req_from=3&lat=-1&network=1&git_ci_time=3599&cop=None&cam_rt=14&tg_i=-1&gender&supr=0&ecs_name=unknown&git_br_name=support_qq_public&asign=a16f388884c2e1209636aacee144f765&slot_id=cHJpb3JpdHlfYWRfc2xvdA%3D%3D&fst=bhy&host_name=JS10XXX.local&request_id=3f6825f3-6a6f-11ea-b281-04d9f51f3141&action=5&sucop=None",
"sTitle": "",
"time": "2020-11-13 14:13:28"
}
- 解析 json 日志时,仅打印指定的字段
请见如下示例中tail -f /tmp/log/gf-app/2020-11-13.log | jq .para
和tail -f /tmp/log/gf-app/2020-11-13.log | jq .para.eventType
的打印结果。
(py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq
{
"context": {
"requestId": "1nzbby6og30c71xbvyyn8k3900evnqa7"
},
"level": "info",
"pTitle": "跟踪estar事件接口",
"para": {
"eventType": "xad_impression",
"platform": "android",
"adId": 170,
"imei": "869941043279996",
"slotId": "cHJpb3JpdHlfYWRfc2xvdA==",
"action": 5
},
"query": "event_type=xad_impression&package_name=com.zhangshangjianzhi.newapp&scop=None&userId=1315171366231&campaign_id=171&osv=0.1&isPlugin=1&git_last_time=20200318201149&box_pkg_name=c.l.a&ad_system=priority_ad&version_name=5.6.0.5&m_name=rtb&yid=1315171366231&cln=adx-14&deep_link=0&m_version=support_qq_public&total_ram=3GB&cpu_core_num=8&make=meizu&lon=-1&dsp=adx&platform=android&sdk_version=None&ad_user_id=105&time_stamp=20200320135422&device_id=6205ea67-ad97-3c34-97df-da5d3a9fc4e8&imsi&fp=-1&ad_id=170&sd_slot_id=103&city=%E5%86%85%E7%BD%91IP&brand=alps&androidid=81b4f0b18d137a02&des_pla=0&mac=02%3A00%3A00%3A00%3A00%3A00&hn=down.doumistatic.com&autp=7&imei=869941043279996&rp=-1&indus=ep&req_from=3&lat=-1&network=1&git_ci_time=3599&cop=None&cam_rt=14&tg_i=-1&gender&supr=0&ecs_name=unknown&git_br_name=support_qq_public&asign=a16f388884c2e1209636aacee144f765&slot_id=cHJpb3JpdHlfYWRfc2xvdA%3D%3D&fst=bhy&host_name=JS10XXX.local&request_id=3f6825f3-6a6f-11ea-b281-04d9f51f3141&action=5&sucop=None",
"sTitle": "",
"time": "2020-11-13 14:32:23"
}
^C
(py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq .para
{
"eventType": "xad_impression",
"platform": "android",
"adId": 170,
"imei": "869941043279996",
"slotId": "cHJpb3JpdHlfYWRfc2xvdA==",
"action": 5
}
^C
(py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq .para.eventType
"xad_impression"
- 从日志中过滤指定字段,且每条日志一行显示
示例如下:./jq-linux64 -c '{ts,level,msg}'
-c
表示每条日志一行显示
{ts,level,msg}
用来过滤指定的字段
- 从日志中排除指定字段
有些时候,每条日志中可能会包含大量的信息,又臭又长,这时我们可能希望排除不需要看到的日志内容。使用del函数即可解决,示例如下:
./jq-linux64 -c 'del(.level,."User-Agent",.caller)'
> 若某个字段包含-
, 那么需要用双引号引起来(见如上示例),否则报语法错误。
上次更新: 2022-08-17 22:28:19