Tuesday 5 February 2013

Reducing Threads in the TimeAndSizeRollingAppender



Several people have commented that the TimeAndSizeRollingAppender could, where many instances were configured, launch many threads.  With date roll enforcement, backup log file scavenging, and backup log file compression all configured, there could be 3 threads per TASRA instance.  That simply didn’t scale.  The original design assumption that only one or two TASRA instances would be configured per application was clearly broken.

I have begun some work to minimise the number of threads launched where multiple TASRA instances are configured.  The first fruits of that have been released and can be downloaded from www.simonsite.org.uk.

The changes permit an arbitrary number of TASRA instances to share a single thread for all backup log file scavenging and backup log file compression tasks.  Date roll enforcement is still carried out on a separate daemon thread, for now.

The benefit is best illustrated with some numbers; taking an arbitrary 10 TASRA instances as opposed to just one to better illustrate the effect:

TASRA configuration
Daemon thread count per 10 TASRA instances
20120703-0826 Release
20130130-1840 Release
ScavengeInterval=300000, DateRollEnforced=true, CompressionAlgorithm=ZIP
30
11
ScavengeInterval=300000, DateRollEnforced=false, CompressionAlgorithm=ZIP
20
1
ScavengeInterval=300000, DateRollEnforced=false, CompressionAlgorithm=default (not set)
10
1
ScavengeInterval=-1, DateRollEnforced=false, CompressionAlgorithm=default
0
0

By default, all TASRA instances will now share the same thread.  Optionally, m TASRA instances may be configured to use n threads, where m >= n >= 1.  To do this, a common named object must be shared by all TASRA instances that will share the same thread, and distinct named object(s) for any other TASRA instances that will use different thread(s).  The following sample configuration illustrates two TASRA instances, “appender-1” and “appender-2”, each using its own thread, “executor-1” or “executor-2”, respectively, in this case for scavenging and compression tasks:

       <appender name="file-appender-1"
              class="uk.org.simonsite.log4j.appender.TimeAndSizeRollingAppender">
              <param name="File" value="${tasra.xml.test.log.file.1}" />
              <layout class="org.apache.log4j.SimpleLayout" />
              <param name="CompressionAlgorithm" value="ZIP" />
              <logTaskExecutorService class="uk.org.simonsite.log4j.appender.NamedLoggingTaskExecutorService">
                     <param name="Name" value="executor-1" />
              </logTaskExecutorService>
       </appender>
       <appender name="file-appender-2"
              class="uk.org.simonsite.log4j.appender.TimeAndSizeRollingAppender">
              <param name="File" value="${tasra.xml.test.log.file.2}" />
              <layout class="org.apache.log4j.SimpleLayout" />
<param name="CompressionAlgorithm" value="ZIP" />
              <logTaskExecutorService class="uk.org.simonsite.log4j.appender.NamedLoggingTaskExecutorService">
                     <param name="Name" value="executor-2" />
              </logTaskExecutorService>
       </appender>

3 comments:

Karl said...

Hi,

Do you have any references on using the element logTaskExecutorService with log4j ? I get an error from the log4j DOM parser when it processes it.

log4j:WARN The content of element type "appender" must match "(errorHandler?,param*,rollingPolicy?,triggeringPolicy?,connectionSource?,layout?,filter*,appender-ref*)".

Many Thanks for the appender by that way,
Karl

Simon said...

Perhaps you aren't using the latest version of the TASRA? Take a look at my site at www.simonsite.org.uk for downloads and documentation.

Simon

Unknown said...

Hello,
I have a question to ask you to please.
It can write separate file in tomcat server and system standard output by TimeAndSizeRollingAppender?

example:
[2015-11-18 12: 31: 45,891] INFO [Thread-2] org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write / read,
=>it will write to the file called catalina_yyyymmdd.log
and
Command line >>> C: \ Windows \ system32> cd \,
=>it will write to the file called stdout_yyyymmdd.log

I will provide my log4j.properties in the following:
log4j.rootLogger=INFO,R
log4j.appender.R=uk.org.simonsite.log4j.appender.TimeAndSizeRollingAppender

log4j.appender.R.DatePattern=yyyyMMdd_HHmm
log4j.appender.R.File=${catalina.base}/logs/stdout.
log4j.appender.R.MaxFileSize=30KB
log4j.appender.R.MaxRollFileCount=10
log4j.appender.R.DateRollEnforced= false
log4j.appender.R.RollOnStartup=true
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=[%d] %5p [%t] %c - %m%n

These are my questions. Thank you :)