The gzip (GNU zip) utility is older and less efficient than bzip2. Its flags and operation are very similar to those of bzip2. A file compressed by gzip is marked with a .gz filename extension. Linux stores manual pages in gzip format to save disk space; likewise, files you download from the Internet are frequently in gzip format.
If you work much with Unix and Linux systems you'll eventually run into the terrific file compression utilities, gzip and gunzip. As their names imply, the first command creates compressed files (by gzip'ing them), and the second command unzip's those files.
how to zip log files in linux
Before you stream logs in real time, enable the log type that you want. Any information written to the conosle output or files ending in .txt, .log, or .htm that are stored in the /home/LogFiles directory (D:\home\LogFiles) is streamed by App Service.
For Linux/custom containers, the ZIP file contains console output logs for both the docker host and the docker container. For a scaled-out app, the ZIP file contains one set of logs for each instance. In the App Service file system, these log files are the contents of the /home/LogFiles directory.
When you open a support case with F5 Support they will ask for a qkview and quite likely they will ask for logs from the system. The reason for this is because although the qkview utility will collect the logs from the device, by default it gathers up to 5MB of log files, meaning that the files can get truncated.
The Amazon EC2 instances in your Elastic Beanstalk environment generate logs that you can view to troubleshoot issues with your application or configuration files. Logs created by the web server, application server, Elastic Beanstalk platform scripts, and AWS CloudFormation are stored locally on individual instances. You can easily retrieve them by using the environment management console or the EB CLI. You can also configure your environment to stream logs to Amazon CloudWatch Logs in real time.
Bundle logs are full logs for a wider range of log files, including logs from yum and cron and several logs from AWS CloudFormation. When you request bundle logs, an instance in your environment packages the full log files into a ZIP archive and uploads it to Amazon S3.
Elastic Beanstalk stores tail and bundle logs in an Amazon S3 bucket, and generates a presigned Amazon S3 URL that you can use to access your logs. Elastic Beanstalk deletes the files from Amazon S3 after a duration of 15 minutes.
If your application generates logs in a location that isn't part of the default configuration for your environment's platform, you can extend the default configuration by using configuration files (.ebextensions). You can add your application's log files to tail logs, bundle logs, or log rotation.
You can find Elastic Beanstalk configuration files for logrotate in /etc/logrotate.elasticbeanstalk.hourly/. These rotation settings are specific to the platform, and might change in future versions of the platform. For more information about the available settings and example configurations, run man logrotate.
Elastic Beanstalk uses files in subfolders of /opt/elasticbeanstalk/tasks (Linux) or C:\Program Files\Amazon\ElasticBeanstalk\config (Windows Server) on the Amazon EC2 instance to configure tasks for tail logs, bundle logs, and log rotation.
You can use environment configuration files (.ebextensions) to add your own .conf files to these folders. A .conf file lists log files specific to your application, which Elastic Beanstalk adds to the log file tasks.
Use the files section to add configuration files to the tasks that you want to modify. For example, the following configuration text adds a log configuration file to each instance in your environment. This log configuration file, cloud-init.conf, adds /var/log/cloud-init.log to tail logs.
On Linux platforms, you can also use wildcard characters in log task configurations. This configuration file adds all files with the .log file name extension from the log folder in the application root to bundle logs.
To help familiarize yourself with log customization procedures, you can deploy a sample application using the EB CLI. For this, the EB CLI creates a local application directory that contains an .ebextentions subdirectory with a sample configuration. You can also use the sample application's log files to explore the log retrieval feature described in this topic. For more information about how to create a sample application with the EB CLI, see EB CLI basics.
As explained in Log rotation settings on Linux, Elastic Beanstalk uses logrotate to rotate logs on Linux platforms. When you configure your application's log files for log rotation, the application doesn't need to create copies of log files. Elastic Beanstalk configures logrotate to create a copy of your application's log files for each rotation. Therefore, the application must keep log files unlocked when it isn't actively writing to them.
On Windows Server, when you configure your application's log files for log rotation, the application must rotate the log files periodically. Elastic Beanstalk looks for files with names starting with the pattern you configured, and picks them up for uploading to Amazon S3. In addition, periods in the file name are ignored, and Elastic Beanstalk considers the name up to the period to be the base log file name.
For example, your application writes to a log file named my_log.log, and you specify this name in your .conf file. The application periodically rotates the file. During the Elastic Beanstalk rotation cycle, it finds the following files in the log file's folder: my_log.log, my_log.0800.log, my_log.0830.log. Elastic Beanstalk considers all of them to be versions of the base name my_log. The file my_log.log has the latest modification time, so Elastic Beanstalk uploads only the other two files, my_log.0800.log and my_log.0830.log.
1. Delete any previous test archives and check you still have the original three test files. Note that, unlike when creating the ZIP archive, you have to include the .zip file extension when referencing an existing archive.
1. Delete the archives inside the test_directory but keep the other files. To avoid confusion it would be good practice to delete the archives leaving only the original files that we created.
3. Create a ZIP archive containing the test_directory directory and its contents. Notice when you create the archive using the -r argument that you see a verbose output detailing each stage of the command as it descends into the directory and archives the files and folder.
We will create a ZIP archive called test_archive.zip which contains a directory test_directory which in turn contains 3 empty test files, test1.txt, test2.h, test3.c . To extract this archive we will use the unzip command.
2. Create a ZIP archive called test_archive.zip that contains the test_directory. Check that the archive has been successfully created using ls. The zip command has two arguments, the name of the archive that we wish to create, and the source of the files to be put into the archive.
With these few basic uses of the zip command you now have lots of options when creating ZIP archives in the linux terminal. Being able to select specific file types and being able to set the compression level in the terminal emulator gives quick access to these powerful tools that are often hard to find in a GUI application.
Any service installed on your Linux systems such as Apache web server or MySQL database server generates log files that are usually stored in the /var/log directory. If you check the contents of this directory, you will see contents similar to what we have below:
Over time, as additional information gets logged, the log files increase in size and take up more space on your hard drive. Before you even know it, the log files will have ballooned in size, gobbling up much of your hard drive space, and if you are not careful, you can easily run out of disk space.
Log rotation is a process that creates new log files and archives & removes old ones to save on disk space. The process renames a current log file. For example, apport.log becomes apport.log.1 and a new apport.log log file is created to log new log entries. Older log files are usually compressed and appear as apport.log.2.gz, apport.log.3.gz, apport.log.4.gz, and so on.
The log rotation process is facilitated using a utility called logrotate. This is a tool that facilitates the rotation of log files and archival & removal of old ones to free up disk space. In summary, logrotate accomplishes the following:
Logrotate runs daily as a cron job, going through various log files, rotating them, and purging older log files as defined in the configuration file. There are two main configuration sources that you need to pay close attention to:
The log files will be rotated on a weekly basis, with suppression of any error messages if any of the log files are missing. 14 log files will be backed up within the course of the month with the creation of a new log file after rotation of the current log file.
The statement implies that log files bigger than 40 Megabytes will be rotated without any regard for the time interval of rotation. This implies that for a log file that is due for rotation after 1 hour, it will be rotated before the specified interval if it exceeds the threshold of 40MB.
Using the crontab to call the logrotate script in addition to specifying the maximum size using the maxsize directive, makes for a perfect combination in ensuring that your log files are rotated in good time to avoid filling up your hard drive.
In this guide, we have shed light on the importance of log files, the menace that they can cause if left to increase in size, and how the logrotate tool can help manage the size of log files on your system. We also looked at some of the available options for use in logrotate configuration. 2ff7e9595c
Comments