Commit 1988986f authored by Hardy Simpson's avatar Hardy Simpson

feature: fully rotate support, done!

parent e1078720
......@@ -15,7 +15,7 @@
[ ] 减少dynamic文件名open的次数,通过日期改变智能推断, file_table?
--- 1.2.0 ---
[ ] rotate和zip的方案
[o] rotate和zip的方案
[o] 把一部分static变量化为全局变量,减少接口参数
[o] fstat64
[o] pipe, cronolog
......
This diff is collapsed.
......@@ -320,7 +320,7 @@ type "mailto:"
\end_layout
\begin_layout Section
zlog 1.2 Release Notes:
Compatible Notes
\end_layout
\begin_layout Enumerate
......@@ -365,6 +365,41 @@ target "http://www.jhweiss.de/software/snprintf.html"
The file buf.c should be cracked, good luck!
\end_layout
\begin_layout Enumerate
zlog has no plan to support Windows.
As Windows already have tons of logging libraries.
\end_layout
\begin_layout Section
zlog 1.2 Release Notes
\end_layout
\begin_layout Enumerate
zlog 1.2 provide features:
\end_layout
\begin_deeper
\begin_layout Enumerate
support for pipeline.
Now zlog could ouput log through programs like cronolog
\end_layout
\begin_layout Enumerate
Full rotation support, see
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Rotation"
\end_inset
\end_layout
\begin_layout Enumerate
Other code compitable details, bug fix.
\end_layout
\end_deeper
\begin_layout Enumerate
zlog 1.2 is binary compatible with zlog 1.0.
The difference are:
......@@ -377,6 +412,23 @@ All zlog macros like ZLOG_INFO is shift to zlog_info, a lowercase version.
Becasue I think it is more easy for people's typewriting.
If you are pre-user of zlog, please using a script substitutes all macros,
and re-compile your program.
I provide one:
\end_layout
\begin_layout LyX-Code
sed -i -e 's/
\backslash
b
\backslash
w*ZLOG
\backslash
w*
\backslash
b/
\backslash
L&
\backslash
E/g' aa.c
\end_layout
\begin_layout Enumerate
......@@ -409,9 +461,8 @@ The reason is obvious: library is called by application, all time log library
\end_layout
\begin_layout Standard
If you want all these features, I recommend rsyslog, a excellent syslog
daemon implementation.
It is a independent process receives logs from other process or machines,
If you want all these features, I recommend rsyslog、zLogFabric、Logstash.
There are independent process receives logs from other process or machines,
and parse and store logs.
Still, it has a distance from user application.
\end_layout
......@@ -431,6 +482,15 @@ reference "sec:User-defined-Output"
\end_layout
\begin_layout Standard
One of my opinion is writing a zlog-redis client.
It send logs to redis on local or remote machines by user-define-output.
Then other process can read logs from redis and write it to disk.
What do you think abount this idea? I will be happy if you discuss it with
me.
\end_layout
\begin_layout Standard
\end_layout
......@@ -4612,15 +4672,19 @@ to file
\begin_inset Text
\begin_layout Plain Layout
file limitation, can be:
\end_layout
rotation.
see
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Rotation"
\begin_layout Plain Layout
1000, 1k, 2M, 1G...
\end_inset
for detail
\end_layout
\begin_layout Plain Layout
3m*2, 4k*3...
10MB * 3 ~ "press.#r.log"
\end_layout
\end_inset
......@@ -5090,6 +5154,303 @@ reference "sec:User-defined-Output"
for more details for $.
\end_layout
\begin_layout Section
Rotation
\begin_inset CommandInset label
LatexCommand label
name "sec:Rotation"
\end_inset
\end_layout
\begin_layout Standard
Why rotation? I have see more than once that in product-environment, the
hard disk is full of logs and cause system stop working, or a single log
file is too big to open or grep.
I conclude several ways of rotation and archive log files:
\end_layout
\begin_layout Enumerate
Split log by date or time.
\end_layout
\begin_deeper
\begin_layout Standard
For example, generate one log file per day.
\end_layout
\begin_layout LyX-Code
aa.2012-08-02.log
\end_layout
\begin_layout LyX-Code
aa.2012-08-03.log
\end_layout
\begin_layout LyX-Code
aa.2012-08-04.log
\end_layout
\begin_layout Standard
In this case, System admin knows how much log will be produced one day.
And after a few month, he hopes to find log file exactly of one day.
The best way for split is done by log library precisely.
Another choice is using cronosplit to analyse the content of log file and
split it.
A bad way is using crontab+logrotate to daily move log files, which is
not accurate, some logs will be put into file of the previous day.
\end_layout
\begin_layout Standard
In zlog, no need for rotating to complete that job.
Simplely setting time in name of log file works:
\end_layout
\begin_layout LyX-Code
*.* "aa.%d(%F).log"
\end_layout
\begin_layout Standard
or using cronolog for fast performace:
\end_layout
\begin_layout LyX-Code
*.* | cronolog aa.%F.log
\end_layout
\end_deeper
\begin_layout Enumerate
Split log by size.
\end_layout
\begin_deeper
\begin_layout Standard
Always fit for develop use.
In this case, program generate a lot of logs in a short period.
But text edit can't quickly open big files.
Althrough the split can be done by split tools after, but it aquires extra
steps.
So a good way is doing rotation by logging library.
There are two ways of rotation, as nlog describes, Sequence and Rolling.
In case of Sequence:
\end_layout
\begin_layout LyX-Code
aa.log (new)
\end_layout
\begin_layout LyX-Code
aa.log.2 (less new)
\end_layout
\begin_layout LyX-Code
aa.log.1
\end_layout
\begin_layout LyX-Code
aa.log.0 (old)
\end_layout
\begin_layout Standard
And in case of Rolling:
\end_layout
\begin_layout LyX-Code
aa.log (new)
\end_layout
\begin_layout LyX-Code
aa.log.0 (less new)
\end_layout
\begin_layout LyX-Code
aa.log.1
\end_layout
\begin_layout LyX-Code
aa.log.2 (old)
\end_layout
\begin_layout Standard
It's hard to say which one fit people's instinct.
\end_layout
\begin_layout Standard
OK, If only some of the newest log is useful to developers, logging library
shoud do the clean work.
Del the old log files.
External tools can't find out which files are older.
\end_layout
\begin_layout Standard
The most simple rotation configure of zlog is:
\end_layout
\begin_layout LyX-Code
*.* "aa.log", 10MB
\end_layout
\begin_layout Standard
It is Rolling.
When aa.log is larger than 10MB, will rename file like this:
\end_layout
\begin_layout LyX-Code
aa.log.2 -> aa.log.3
\end_layout
\begin_layout LyX-Code
aa.log.1 -> aa.log.2
\end_layout
\begin_layout LyX-Code
aa.log.0 -> aa.log.1
\end_layout
\begin_layout LyX-Code
aa.log -> aa.log.0
\end_layout
\begin_layout Standard
The configure can be more complex:
\end_layout
\begin_layout LyX-Code
*.* "aa.log", 10MB * 0 ~ "aa.log.#r"
\end_layout
\begin_layout Standard
The 1st argument represents when rotation will be trigger, in size.
\end_layout
\begin_layout Standard
The 2nd arguments represents how many archive file will be kept not delete(0
means keep all).
\end_layout
\begin_layout Standard
The 3rd argument represents archive file name.
#r is the number of archive files.
r is short for Rolling, and #s is short for sequence.
Archive file name must contain one of #r or #s.
\end_layout
\end_deeper
\begin_layout Enumerate
Split log by size, and add time tag to archive file.
\end_layout
\begin_deeper
\begin_layout LyX-Code
aa.log
\end_layout
\begin_layout LyX-Code
aa.log-20070305.00.log
\end_layout
\begin_layout LyX-Code
aa.log-20070501.00.log
\end_layout
\begin_layout LyX-Code
aa.log-20070501.01.log
\end_layout
\begin_layout LyX-Code
aa.log-20071008.00.log
\end_layout
\begin_layout Standard
In this case, log file is not focus frequently, but will be check out one
day.
Of course, when one day's log is more than 100MB, like 20070501, will be
stored in two files and add postfix numbers.
\end_layout
\begin_layout Standard
The configure of zlog is:
\end_layout
\begin_layout LyX-Code
*.* "aa.log", 100MB ~ "aa-%D(%Y%m%d).#2s.log"
\end_layout
\begin_layout LyX-Code
Do rotation every 100MB.
The archive file name support conversion strings also!.
#2s means the sequence number is at least 2 bytes wide.
Sequece from 00.
That's the most complex way of archive in zlog!
\end_layout
\end_deeper
\begin_layout Enumerate
Compress, move and delete old archive.
\end_layout
\begin_deeper
\begin_layout Standard
Compress should not be done by logging library, because compress need time
and CPU.
The mission of logging library is cooperation with compress programs.
\end_layout
\begin_layout Standard
For the 3 ways of split logs, way 1 and way 3 is easy to manage.
It is easy to find old log file by name or by modify time.
And then compress, move and delete old log files by crontab and shell.
\end_layout
\begin_layout Standard
For the way 2, compress is useless, delete is need and zlog already support
it.
\end_layout
\begin_layout Standard
If you really want to rotate and compress log file at the same time, I suggest
logrotate.
As it is a indepence program and will not confuse.
\end_layout
\end_deeper
\begin_layout Enumerate
zlog support for external tools like logrotate
\end_layout
\begin_layout Standard
The rotation support of zlog is very powerful, still there are several cases
zlog can not handle.
Like rotation by time, before or after rotation call some user-defined
shells...
That will make zlog too complex and ugly.
\end_layout
\begin_layout Standard
Under that cirumstances, maybe you like to use external tools like logrotate.
The problem is, on linux, when a tool rename the log file, the working
process will not automaticlly reopen the new file.
The standard way is send a signal to program, let it reopen the file, as
to syslogd the command is:
\end_layout
\begin_layout LyX-Code
kill -SIGHUP `cat /var/run/syslogd.pid`
\end_layout
\begin_layout Standard
For zlog as a library, it is not good to receive signals.
zlog provide zlog_reload(), which reload configure file and reopen all
log files.
So if you write a program and want to reopen log file mannully, you can
write some code to do job like this: after receiving a signal or command
from client, call zlog_reload().
\end_layout
\begin_layout Section
Configure File Tools
\end_layout
......
......@@ -152,6 +152,7 @@ zlog_conf_t *zlog_conf_new(const char *confpath)
zc_error("zlog_level_list_new fail");
goto err;
}
a_conf->formats = zc_arraylist_new((zc_arraylist_del_fn) zlog_format_del);
if (!a_conf->formats) {
zc_error("zc_arraylist_new fail");
......@@ -201,6 +202,7 @@ static int zlog_conf_build_without_file(zlog_conf_t * a_conf)
default_rule = zlog_rule_new(
ZLOG_CONF_DEFAULT_RULE,
a_conf->levels,
a_conf->default_format,
a_conf->formats,
a_conf->file_perms,
......@@ -466,6 +468,7 @@ static int zlog_conf_parse_line(zlog_conf_t * a_conf, char *line, int *section)
break;
case 4:
a_rule = zlog_rule_new(line,
a_conf->levels,
a_conf->default_format,
a_conf->formats,
a_conf->file_perms,
......
This diff is collapsed.
......@@ -31,12 +31,12 @@ typedef struct zlog_rotater_s {
char *base_path; /* aa.log */
char *archive_path; /* aa.#5i.log */
char glob_path[MAXLEN_PATH + 1]; /* aa.*.log */
size_t num_start_len; /* 3 offset to archive_path */
size_t num_end_len; /* 6 offset to archive_path */
size_t num_start_len; /* 3, offset to glob_path */
size_t num_end_len; /* 6, offset to glob_path */
int num_width; /* 5 */
int mv_type; /* ROLLING or SEQUENCE */
int max_count;
zc_arraylist_t files;
zc_arraylist_t *files;
} zlog_rotater_t;
zlog_rotater_t *zlog_rotater_new(char *lock_file);
......@@ -48,8 +48,8 @@ void zlog_rotater_del(zlog_rotater_t *a_rotater);
* 0 no rotate, or rotate and success
*/
int zlog_rotater_rotate(zlog_rotater_t *a_rotater,
const char *base_path, size_t msg_len,
const char *archive_path, long archive_max_size, int archive_max_count,
char *base_path, size_t msg_len,
char *archive_path, long archive_max_size, int archive_max_count,
int *reopen_fd, int reopen_flags, unsigned int reopen_perms);
void zlog_rotater_profile(zlog_rotater_t *a_rotater, int flag);
......
......@@ -58,8 +58,8 @@ void zlog_rule_profile(zlog_rule_t * a_rule, int flag)
a_rule->file_open_flags,
a_rule->file_path,
a_rule->dynamic_file_specs,
a_rule->static_file_descriptor,
a_rule->dynamic_specs,
a_rule->static_fd,
a_rule->archive_max_size,
a_rule->archive_max_count,
......@@ -74,8 +74,8 @@ void zlog_rule_profile(zlog_rule_t * a_rule, int flag)
a_rule->record_func,
a_rule->format);
if (a_rule->dynamic_file_specs) {
zc_arraylist_foreach(a_rule->dynamic_file_specs, i, a_spec) {
if (a_rule->dynamic_specs) {
zc_arraylist_foreach(a_rule->dynamic_specs, i, a_spec) {
zlog_spec_profile(a_spec, flag);
}
}
......@@ -91,7 +91,7 @@ static int zlog_rule_output_static_file_single(zlog_rule_t * a_rule, zlog_thread
return -1;
}
if (write(a_rule->static_file_descriptor,
if (write(a_rule->static_fd,
zlog_buf_str(a_thread->msg_buf),
zlog_buf_len(a_thread->msg_buf)) < 0) {
zc_error("write fail, errno[%d]", errno);
......@@ -101,8 +101,8 @@ static int zlog_rule_output_static_file_single(zlog_rule_t * a_rule, zlog_thread
/* not so thread safe here, as multiple thread may ++fsync_count at the same time */
if (a_rule->fsync_period && ++a_rule->fsync_count >= a_rule->fsync_period) {
a_rule->fsync_count = 0;
if (fsync(a_rule->static_file_descriptor)) {
zc_error("fsync[%d] fail, errno[%d]", a_rule->static_file_descriptor, errno);
if (fsync(a_rule->static_fd)) {
zc_error("fsync[%d] fail, errno[%d]", a_rule->static_fd, errno);
}
}
......@@ -140,26 +140,26 @@ static int zlog_rule_output_static_file_rotate(zlog_rule_t * a_rule, zlog_thread
}
len = zlog_buf_len(a_thread->msg_buf);
if (write(a_rule->static_file_descriptor, zlog_buf_str(a_thread->msg_buf), len) < 0) {
if (write(a_rule->static_fd, zlog_buf_str(a_thread->msg_buf), len) < 0) {
zc_error("write fail, errno[%d]", errno);
return -1;
}
if (a_rule->fsync_period && ++a_rule->fsync_count >= a_rule->fsync_period) {
a_rule->fsync_count = 0;
if (fsync(a_rule->static_file_descriptor)) {
zc_error("fsync[%d] fail, errno[%d]", a_rule->static_file_descriptor, errno);
if (fsync(a_rule->static_fd)) {
zc_error("fsync[%d] fail, errno[%d]", a_rule->static_fd, errno);
}
}
if (len > a_rule->archive_max_size) {
zc_debug("one msg's len[%ld] > archive_max_size[%ld], no rotate",
(long)msg_len, archive_max_size);
(long)len, (long)a_rule->archive_max_size);
return 0;
}
if (stat(a_rule->file_path, &info)) {
zc_warn("stat [%s] fail, errno[%d], maybe in rotating", file_path, errno);
zc_warn("stat [%s] fail, errno[%d], maybe in rotating", a_rule->file_path, errno);
return 0;
}
......@@ -170,7 +170,7 @@ static int zlog_rule_output_static_file_rotate(zlog_rule_t * a_rule, zlog_thread
a_rule->file_path, len,
zlog_rule_gen_archive_path(a_rule, a_thread),
a_rule->archive_max_size, a_rule->archive_max_count,
&(a_rule->static_file_descriptor),
&(a_rule->static_fd),
a_rule->file_open_flags | O_WRONLY | O_APPEND | O_CREAT, a_rule->file_perms)
) {
zc_error("zlog_rotater_rotate fail");
......@@ -189,7 +189,7 @@ static int zlog_rule_output_static_file_rotate(zlog_rule_t * a_rule, zlog_thread
\
zlog_buf_restart(a_thread->path_buf); \
\
zc_arraylist_foreach(a_rule->dynamic_file_specs, i, a_spec) { \
zc_arraylist_foreach(a_rule->dynamic_specs, i, a_spec) { \
if (zlog_spec_gen_path(a_spec, a_thread)) { \
zc_error("zlog_spec_gen_path fail"); \
return -1; \
......@@ -277,12 +277,12 @@ static int zlog_rule_output_dynamic_file_rotate(zlog_rule_t * a_rule, zlog_threa
if (len > a_rule->archive_max_size) {
zc_debug("one msg's len[%ld] > archive_max_size[%ld], no rotate",
(long)msg_len, (long) archive_max_size);
(long)len, (long) a_rule->archive_max_size);
return 0;
}
if (stat(path, &info)) {
zc_warn("stat [%s] fail, errno[%d], maybe in rotating", file_path, errno);
zc_warn("stat [%s] fail, errno[%d], maybe in rotating", path, errno);
return 0;
}
......@@ -463,8 +463,8 @@ static int syslog_facility_atoi(char *facility)
return -187;
}
static int zlog_rule_parse_path(const char *path_start, /* start with a " */
char *path_str, size_t path_size, zc_arraylist **path_specs)
static int zlog_rule_parse_path(char *path_start, /* start with a " */
char *path_str, size_t path_size, zc_arraylist_t **path_specs)
{
char *p, *q;
size_t len;
......@@ -475,7 +475,7 @@ static int zlog_rule_parse_path(const char *path_start, /* start with a " */
q = strrchr(p, '"');
if (!q) {
zc_error("matching \" not found in conf line[%s]", output);
zc_error("matching \" not found in conf line[%s]", path_start);
return -1;
}
len = q - p;
......@@ -519,11 +519,12 @@ static int zlog_rule_parse_path(const char *path_start, /* start with a " */
return 0;
err:
if (specs) zc_arraylist_del(specs);
if (a_spec) zlog_spec_del(a_spec):
if (a_spec) zlog_spec_del(a_spec);
return -1;
}
zlog_rule_t *zlog_rule_new(char *line,
zc_arraylist_t *levels,
zlog_format_t * default_format,
zc_arraylist_t * formats,
unsigned int file_perms,
......@@ -624,7 +625,8 @@ zlog_rule_t *zlog_rule_new(char *line,
p = level;
break;
}
a_rule->level = zlog_level_list_atoi(zlog_env_conf->levels, p);
a_rule->level = zlog_level_list_atoi(levels, p);
/* level_bit is a bitmap represents which level can be output
* 32bytes, [0-255] levels, see level.c
......@@ -715,32 +717,41 @@ zlog_rule_t *zlog_rule_new(char *line,
case '"' :
if (!p) p = file_path;
rc = zlog_rule_parse_path(p, a_rule->file_path, a_rule->dynamic_file_specs);
rc = zlog_rule_parse_path(p, a_rule->file_path, sizeof(a_rule->file_path),
&(a_rule->dynamic_specs));
if (rc) {
zc_error("zlog_rule_parse_path fail");
goto err;
}
memset(archive_max_size, 0x00, sizeof(archive_max_size));
nscan = sscanf(file_limit, " %[^*]*%d ~ %n", archive_max_size, &(a_rule->archive_max_count), &nread);
a_rule->archive_max_size = zc_parse_byte_size(archive_max_size);
nscan = sscanf(file_limit, " %s * %d ~",
archive_max_size, &(a_rule->archive_max_count));
if (nscan) {
a_rule->archive_max_size = zc_parse_byte_size(archive_max_size);
}
p = strchr(file_limit + nread, '"');
p = strchr(file_limit, '"');
if (p) { /* archive file path exist */
rc = zlog_rule_parse_path(p, a_rule->archive_path, a_rule->archive_specs);
rc = zlog_rule_parse_path(p, a_rule->archive_path, sizeof(a_rule->file_path),
&(a_rule->archive_specs));
if (rc) {
zc_error("zlog_rule_parse_path fail");
goto err;
}
if (strchr(a_rule->archive_path, '#') == NULL) {
zc_error("archive_path must contain #");
p = strchr(a_rule->archive_path, '#');
</