Commit 52bf7adf authored by Hardy Simpson's avatar Hardy Simpson

feature: add pipeline support

parent 8e36e666
......@@ -12,10 +12,12 @@
[ ] redis对接,协议设计
[ ] 和rsyslog对接的问题
[ ] linux fsync->fdatasync, open..
[ ] 减少dynamic文件名open的次数,通过日期改变智能推断, file_table?
--- 1.2.0 ---
[ ] pipe, cronolog
[ ] fstat64
[ ] rotate和zip的方案
[o] pipe, cronolog
[o] record忽略第二参数
[o] __VA_ARGS__的跨平台
[o] 增加git/版本在代码中的标志(redis)
......
......@@ -4174,7 +4174,7 @@ aa(as it matches rules above)
\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="3">
<lyxtabular version="3" rows="8" columns="3">
<features tabularvalignment="middle">
<column alignment="left" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
......@@ -4300,6 +4300,35 @@ LOG_USER(default), LOG_LOCAL[0-7]
必填
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
管道输出
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
|cat
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
无意义
\end_layout
\end_inset
</cell>
</row>
......@@ -4435,6 +4464,108 @@ _buf))
所以,千万不要在守护进程的规则里面加上>stdout或>stderr。这会产生不可预料的结果……如果一定要输出到终端,用"/dev/tty"代替。
\end_layout
\end_deeper
\begin_layout Itemize
管道输出
\end_layout
\begin_deeper
\begin_layout LyX-Code
*.* | /usr/bin/cronolog /www/logs/example_%Y%m%d.log ; normal
\end_layout
\begin_layout Standard
这是一个将zlog的输出到管道后接cronolog的例子。实现的原理很简单,在zlog_init的时候调用popen("/usr/bin/cronolog
/www/logs/example_%Y%m%d.log", "w"),后面往这个文件描述符里面写指定格式的日志。使用cronolog来生成按天分割的日志效率比
zlog自己的动态路径的效率要高,因为通过管道,无须每次打开关闭动态路径的文件描述符。
\end_layout
\begin_layout LyX-Code
[rules]
\end_layout
\begin_layout LyX-Code
*.* "press%d(%Y%m%d).log"
\end_layout
\begin_layout LyX-Code
$ time ./test_press_zlog 1 10 100000
\end_layout
\begin_layout LyX-Code
real 0m9.234s
\end_layout
\begin_layout LyX-Code
user 0m2.860s
\end_layout
\begin_layout LyX-Code
sys 0m10.060s
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
[rules]
\end_layout
\begin_layout LyX-Code
*.* | /usr/bin/cronolog press%Y%m%d.log
\end_layout
\begin_layout LyX-Code
$ time ./test_press_zlog 1 10 100000
\end_layout
\begin_layout LyX-Code
real 0m1.911s
\end_layout
\begin_layout LyX-Code
user 0m1.980s
\end_layout
\begin_layout LyX-Code
sys 0m1.470s
\end_layout
\begin_layout Standard
不过,使用管道也是有限制的:
\end_layout
\begin_layout Itemize
POSIX.1-2001保证读写不大于PIPE_BUF大小的内容是原子的。linux上PIPE_BUF为4096。
\end_layout
\begin_layout Itemize
单条日志的长度超过PIPE_BUF的时候并且有多个有父子关系的进程写通过zlog写同一个管道,也就是在zlog_init之后fork多个子进程,此时只有一个cr
onolog的进程监听一个管道描述符,日志内容可能会交错。
\end_layout
\begin_layout Itemize
多个进程分别zlog_init,启动多个cronolog进程,写拥有同一个文件路径的日志文件,即使单条日志长度不超过PIPE_BUF,也有可能导致日志交错,因为
cronolog读到的文件流是连续的,它不知道单条日志的边界在哪里。
\end_layout
\begin_layout Standard
所以,总结一下,使用管道来输出日志而保证日志不交错,需要满足这样的条件。
\end_layout
\begin_layout Itemize
单进程写,单条日志长度不限制。单进程内内的多线程写日志的原子性已经由zlog保证了。
\end_layout
\begin_layout Itemize
有父子关系的多进程,单条日志长度不能超过PIPE_BUF(4096)
\end_layout
\begin_layout Standard
无父子关系的多进程同时写一个日志,无论单条日志长度是多少,都有可能导致日志交错。
\end_layout
\end_deeper
\begin_layout Itemize
文件
......
......@@ -4435,7 +4435,7 @@ Now, zlog supports some ways of output, the syntax is
\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="3">
<lyxtabular version="3" rows="8" columns="3">
<features tabularvalignment="middle">
<column alignment="left" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
......@@ -4557,6 +4557,35 @@ syslog facilitiy, can be: LOG_USER(default), LOG_LOCAL[0-7]
This is not optional.
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
pipeline output
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
| cat
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
no meaning
\end_layout
\end_inset
</cell>
</row>
......@@ -4695,6 +4724,122 @@ Then what will happen? The log will be written to the file whose fd is now
instead.
\end_layout
\end_deeper
\begin_layout Itemize
pipeline output
\end_layout
\begin_deeper
\begin_layout LyX-Code
*.* | /usr/bin/cronolog /www/logs/example_%Y%m%d.log ; normal
\end_layout
\begin_layout Standard
This is a example of how zlog pipeline its output to cronolog.
The implementation is simple.
popen("/usr/bin/cronolog /www/logs/example_%Y%m%d.log","w") is called at
zlog_init(), and forward logs will be write to the open descripter in the
"normal" format.
Writing through pipeline and cronnolog is faster than dynamic file of zlog,
as there is no need to open and close file descripter each time when logs
are write to a pipe.
\end_layout
\begin_layout LyX-Code
[rules]
\end_layout
\begin_layout LyX-Code
*.* "press%d(%Y%m%d).log"
\end_layout
\begin_layout LyX-Code
$ time ./test_press_zlog 1 10 100000
\end_layout
\begin_layout LyX-Code
real 0m9.234s
\end_layout
\begin_layout LyX-Code
user 0m2.860s
\end_layout
\begin_layout LyX-Code
sys 0m10.060s
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
[rules]
\end_layout
\begin_layout LyX-Code
*.* | /usr/bin/cronolog press%Y%m%d.log
\end_layout
\begin_layout LyX-Code
$ time ./test_press_zlog 1 10 100000
\end_layout
\begin_layout LyX-Code
real 0m1.911s
\end_layout
\begin_layout LyX-Code
user 0m1.980s
\end_layout
\begin_layout LyX-Code
sys 0m1.470s
\end_layout
\begin_layout Standard
Still, There are some limitations when using pipeline output:
\end_layout
\begin_layout Itemize
POSIX.1-2001 says that write(2)s of less than PIPE_BUF bytes must be atomic,
On Linux, PIPE_BUF is 4096 bytes.
\end_layout
\begin_layout Itemize
When a single log is longer than PIPE_BUF, and multi-process write logs
through one piple(father calls zlog_init(), and fork many child processes),
will cause log interlace.
\end_layout
\begin_layout Itemize
Multi-process who don't have relatives, start multiple cronolog processes
and write to the same log file.
Even a single log is not longer than PIPE_BUF, multiple cronologs will
cause log interlace.
As cronologs read log continuous, it doesn't know where is the egde between
logs.
\end_layout
\begin_layout Standard
In summary, pipeline is safe when:
\end_layout
\begin_layout Itemize
Single process writing, no limitation for length of one log.
Multi-threads in one process atomic writing is already assured by zlog.
\end_layout
\begin_layout Itemize
Multi-process who have relatives, the length of one log should not longer
than PIPE_BUF.
\end_layout
\begin_layout Standard
Multi-process who don't have relatives, no matter how long a single log
is, will cause interlace and is not safe.
\end_layout
\end_deeper
\begin_layout Itemize
file
......
......@@ -26,8 +26,7 @@ default.* >stdout; simple
my_.INFO >stderr;
my_cat.!ERROR "/var/log/aa.log"
my_dog.=DEBUG >syslog, LOG_LOCAL0; simple
my_dog.=DEBUG | /usr/bin/cronolog /www/logs/example_%Y%m%d.log ; normal
my_mice.* $record_func , "record_path%c"; normal
......@@ -119,6 +119,7 @@ zlog.o: zlog.c fmacros.h conf.h zc_defs.h zc_profile.h zc_arraylist.h \
$(DYLIBNAME): $(OBJ)
$(DYLIB_MAKE_CMD) $(OBJ)
cp -f $(DYLIBNAME) $(DYLIB_MINOR_NAME) # for test directory
$(STLIBNAME): $(OBJ)
$(STLIB_MAKE_CMD) $(OBJ)
......@@ -134,7 +135,7 @@ zlog-chk-conf: zlog-chk-conf.o $(STLIBNAME)
$(CC) -std=c99 -pedantic -c $(REAL_CFLAGS) $<
clean:
rm -rf $(DYLIBNAME) $(STLIBNAME) $(BINS) *.o *.gcda *.gcno *.gcov
rm -rf $(DYLIBNAME) $(STLIBNAME) $(BINS) *.o *.gcda *.gcno *.gcov $(DYLIB_MINOR_NAME)
dep:
$(CC) -MM *.c
......
#define ZLOG_GIT_SHA1 "312c091e"
#define ZLOG_GIT_DIRTY "9"
#define ZLOG_GIT_SHA1 "ff59f028"
#define ZLOG_GIT_DIRTY "13"
......@@ -47,7 +47,7 @@ void zlog_rule_profile(zlog_rule_t * a_rule, int flag)
zlog_spec_t *a_spec;
zc_assert(a_rule,);
zc_profile(flag, "---rule:[%p][%s%c%d]-[%s,%p,(%d)(%d:%ld*%d)][%d][%s:%s:%p];[%p]---",
zc_profile(flag, "---rule:[%p][%s%c%d]-[%s,%p,(%d)(%d:%ld*%d)][%d][%d][%s:%s:%p];[%p]---",
a_rule,
a_rule->category,
a_rule->compare_char,
......@@ -61,6 +61,8 @@ void zlog_rule_profile(zlog_rule_t * a_rule, int flag)
a_rule->file_max_size,
a_rule->file_max_count,
a_rule->pipe_fd,
a_rule->syslog_facility,
a_rule->record_name,
......@@ -187,7 +189,6 @@ static int zlog_rule_output_static_file_rotate(zlog_rule_t * a_rule, zlog_thread
zlog_buf_restart(a_thread->path_buf); \
\
zc_arraylist_foreach(a_rule->dynamic_file_specs, i, a_spec) { \
zc_error("%d", i); \
if (zlog_spec_gen_path(a_spec, a_thread)) { \
zc_error("zlog_spec_gen_path fail"); \
return -1; \
......@@ -282,6 +283,23 @@ static int zlog_rule_output_dynamic_file_rotate(zlog_rule_t * a_rule, zlog_threa
return 0;
}
static int zlog_rule_output_pipe(zlog_rule_t * a_rule, zlog_thread_t * a_thread)
{
if (zlog_format_gen_msg(a_rule->format, a_thread)) {
zc_error("zlog_format_gen_msg fail");
return -1;
}
if (write(a_rule->pipe_fd,
zlog_buf_str(a_thread->msg_buf),
zlog_buf_len(a_thread->msg_buf)) < 0) {
zc_error("write fail, errno[%d]", errno);
return -1;
}
return 0;
}
static int zlog_rule_output_syslog(zlog_rule_t * a_rule, zlog_thread_t * a_thread)
{
zlog_level_t *a_level;
......@@ -701,6 +719,19 @@ zlog_rule_t *zlog_rule_new(char *line,
}
}
break;
case '|' :
a_rule->pipe_fp = popen(output + 1, "w");
if (!a_rule->pipe_fp) {
zc_error("popen fail, errno[%d]", errno);
goto err;
}
a_rule->pipe_fd = fileno(a_rule->pipe_fp);
if (a_rule->pipe_fd < 0 ) {
zc_error("fileno fail, errno[%d]", errno);
goto err;
}
a_rule->output = zlog_rule_output_pipe;
break;
case '>' :
if (STRNCMP(file_path + 1, ==, "syslog", 6)) {
a_rule->syslog_facility = syslog_facility_atoi(file_limit);
......@@ -806,6 +837,11 @@ void zlog_rule_del(zlog_rule_t * a_rule)
zc_error("close fail, maybe cause by write, errno[%d]", errno);
}
}
if (a_rule->pipe_fp) {
if (pclose(a_rule->pipe_fp) == -1) {
zc_error("pclose fail, errno[%d]", errno);
}
}
free(a_rule);
zc_debug("zlog_rule_del[%p]", a_rule);
return;
......
......@@ -53,7 +53,6 @@ struct zlog_rule_s {
char file_path[MAXLEN_PATH + 1];
zc_arraylist_t *dynamic_file_specs;
int static_file_descriptor;
FILE *static_file_stream;
pthread_rwlock_t static_reopen_lock;
unsigned int file_perms;
......@@ -61,6 +60,9 @@ struct zlog_rule_s {
long file_max_size;
int file_max_count;
FILE *pipe_fp;
int pipe_fd;
size_t fsync_period;
size_t fsync_count;
......
......@@ -12,6 +12,7 @@ exe = \
test_leak \
test_mdc \
test_record \
test_pipe \
test_press_zlog \
test_press_zlog2 \
test_press_write \
......@@ -24,7 +25,7 @@ exe = \
all : $(exe)
$(exe) : %:%.o
gcc -O2 -g -static -o $@ $^ -L../src -lzlog -lpthread -Wl,-rpath ../src
gcc -O2 -g -o $@ $^ -L../src -lzlog -lpthread -Wl,-rpath ../src
.c.o :
gcc -O2 -g -Wall -D_GNU_SOURCE -o $@ -c $< -I. -I../src
......
/*
* This file is part of the zlog Library.
*
* Copyright (C) 2011 by Hardy Simpson <HardySimpson@gmail.com>
*
* The zlog Library is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* The zlog Library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with the zlog Library. If not, see <http://www.gnu.org/licenses/>.
*/
#include <stdio.h>
#include "zlog.h"
int main(int argc, char** argv)
{
int rc;
zlog_category_t *zc;
rc = zlog_init("test_pipe.conf");
if (rc) {
printf("init failed\n");
return -1;
}
zc = zlog_get_category("my_cat");
if (!zc) {
printf("get cat fail\n");
zlog_fini();
return -2;
}
zlog_info(zc, "hello, zlog");
zlog_fini();
return 0;
}
[formats]
simple = "%m%n"
[rules]
my_cat.* |/usr/bin/cronolog test_pipe_%Y%m%d.log;
......@@ -4,4 +4,4 @@
default format = "%D.%us %-6V %p:%T:%F:%L %m%n"
[rules]
*.* "press.log"
*.* "press%M().log"
......@@ -2,7 +2,7 @@
if [ "$1" = "" ]
then
echo "Usage: mktarball.sh <git tag, branch or commit>"
echo "Usage: mk_targz.sh <git tag, branch or commit>"
echo "Example: mktarball.sh 2.2-rc4"
exit 1
fi
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment