Multipurpose Log Filter
From LokDoc
Multipurpose Log Filter (MPLF) is an application designed to quickly filter large line-based logs using regular expressions. It allows people to quickly sort out interesting information from logs.
The program's filters are defined by the user (using regular expressions) but one also ask Lokorin for special filters or use the built in regular expression generator if needed. The filters consist of several rules, documents are sequentially filtered based on the rules. The rules themselves specify what lines they should handle and what they should do to those lines. For instance a rule might remove a specific type of lines from the parse while another adds specific types of lines to the result.
Contents |
Installation
The application can be downloaded from http://www.lokorin.com/mplf/download.
The program requires Java (>=JRE 1.4.2) which is available for most platforms at http://java.sun.com (Direct link to the JRE 1.5 download section). Most platforms should already have Java installed though.
The program itself comes in a .jar file which can be launched directly by Java. The way to do this differs from platform to platform but the basic command (assuming that the jar file is named mplf.jar) is the folowing.
java -jar mplf.jar
Windows users should also be able to just double click the jar file (after installing Java) to get the program to start.
Interface
The interface is hopefully quite simple. The main window consists of three parts and a menu. The top part is where the results are displayed, the bottom-left part is where the currently selected filter's rules are displayed and the bottom-right part is where available filters are displayed.
Filter list
All filters are displayed in the bottom-right part of the window, it is an important part of the window as it controls all other parts of the window. By selecting a filter here one changes what is displayed in the result and in the rules list. The rules list (to the bottom-left) displays all of the currently selected filter's rules and therfor one views a filters contents by selecting it. One can create a new filter by clicking the Add Filter button and the currently selected filter can be removed by clicking the Remove Filter button.
Each filter defines name and a filter type, these can be editted by double-clicking on the filter that one wants to edit in the filter list. This brings up a new window containing the filter's name and filter type, clicking the Save button in that window will save the changes and close the edit window. Closing the window by other means will cause the changes to be ignored. Also it might be good to know that pressing the enter key while having the edit window open will have the same effect as pressing the Save button.
Rules list
The rules of the currently selected filter are shown in the bottom-left part of the window. The controls are the same as for the filter list, one adds new rules with the Add Rule button and removes the currently selected rule by clicking the "Remove Rule" button.
Double-clicking a rule in the list brings up a new window which one can use for editing the rule. Clicking the Save button (or pressing the enter key) in that window will cause any changes to be saved. Closing the edit window in any other way will cause the changes to be ignored.
Result window
The filtered results are shown in the upper half of the window. The result depends on the currently selected filter and on the contents of the source used. Only the currently selected filter is used, one can not use multiple filters at the same time and interlace the result (although that can be added in the future if requested).
Menu
File
- Open - Opens a source to be filtered.
- File - Opens a specified file to be filtered.
- Enter text - Opens a new dialog where one can directly enter text to filter.
- URL - Opens a new dialog where one can enter an URL, causing the contents that the URL points at (e.g. a website) to be filtered.
- Save result - Opens a dialog from which one can save the filtered results.
- Exit - Exits the program.
Edit
- Preferences - Opens a window from where users can edit some of the preferences used by the program.
Miscellaneous
- Serach for update - Searches for available updates and upgrades the program if any is found. The operation requires the user to restart the program afterwards for changes to take affect.
Help
- About - Opens an about box, it contains a bit of info regarding the program.
Filters
A filter describes exactly what should and should not be allowed to pass. The filter itself is made up of several rules which each defines a specific action for a specific type of text. So the filter can actually be seen as container for rules, the order in which the rules are placed in the filter is important though, the ones that are highest in the list are run first so they have a higher priority.
A filter has the following properties.
- Name - The name of the filter, the name has to be unique, there can not be two filters with the same name.
- Style - This property defines how the filter should act, there are currently two styles.
- Line by line - This is the default style, it takes each line by itself and applies all rules to the line as a whole.
- Multiline - This style views the whole text at once, but rules only affect the subsequence of the whole text that they match.
Here's some crude ASCII art to try to illustrate the flow, the lines flow from the top to the bottom and are filtered on the way:
| |
| document |
| |
rule1 ------------------------
_______________/ |
rejected lines /
___________________ /
| |
| |
| ignored |
| lines |
rule2 ------------------------
| \________________
\ accepted lines \
| _____________ |
| ignored / | |
| lines / | |
rule3 ------------------------ | |
| | | |
\ \_____________| |
| accepted lines |
|ignored_______________ |
|lines / | |
| / -------| |-------
____________________/ / < >
/ < >
rejected lines / < RESULT >
_______________________/ < >
<------------------->
In this example rule1 rejects everything that it matches, rule2 and rule3 accept everything that they match. The end result is the sum of all accepted lines.
Rules
A rule is a small unit which knows exactly what lines it should do something with and what that "something" is. The rules are defined by the user and are based on regular expressions. Writing a regular expression is not hard but it takes a while to get comfortable with.
Luckely one doesn't have to learn how to write regexps to use the program. One can use the built in regexp generator. The generator can be accessed through the rule edit window via the Generate button. A dialog asking for some samples will pop up into which one should enter plenty of examples of what the regexp should match, one can also enter examples of what the regexp should not match, but it is mainly a check to make sure that nothing went wrong in the generator. Once everything has been entered one can click Done and a regexp satisfying the requirements will be generated and entered. The generator is currently limited, a human will easily create better regexps with a bit of practise and it is up to the human to enter groupings. This might change in the future though, if there is an interest in generator being improved. In case the generator doesn't quite cut it then one can contact Lokorin.
Rules have the following properties
- Name - The name of the rule, this doesn't carry any improtance, it's just there so people can give their rules useful names.
- Action - This specifies what the rule should do with things that match the rule. There are currently three options.
- Nothing - Do nothing, this can be useful when one wants to temporarily deactivate a rule but otherwise one might as well just delete the rule if it's not going to do anything.
- Include - Includes the matched text into the result.
- Exclude - Excludes the matched text, the text will not be handled again, this effectivly makes it so that none of the following rules can include the text. This does nothing if the filter uses the multiline style.
- Regular expression - This specifies what should be handled by the rule. The regexp also specifies regexp groupings (which can then be used in the output format).
- Output format - This specifies how the output should be displayed. $n (where n=0,1,2...) can be used in the format to specify captured regexp groups. $1 is replaced with the first captured regexp group, $2 with the second one and so on. $0 is replaced with the whole subsequence (line in "line by line" style) that was handled by the rule. Therefor the default output format "$0" displays the subsequences that were matched in the result.
Checking for the latest version
The simplest way to get the latest version of the program is to use the built in version checker/updater. It can be accessed through the menu via Miscellaneous -> Search for updates which will open a new window. The new window will check with the server if there's a more recent version, if there is then the most recent version will be downloaded from the server. The new version will overwrite the old one, but you have to restart the program in order for the update to take affect.
Another way is to check the current version through the About box via the menu. Then compare that number to the latest release at the download page.
The online filter archive
The program has an online filter archive from which users can download already existing filters. Users can also upload their own filters there in order to help others. There are two ways to access the archive, via the program or via the web.
From inside the program
Open the archive from the menu via Miscellaneous -> Online Filter Archive. A new window will open, it might need a few seconds to download the latest info from the server. To the right you will see a tree structure of categories and filters, selecting a filter or category will bring up additional information.
You can upload one of your own filters to a category by selecting the category you want to upload to and then clicking on the Upload a filter button to the right. This will open a new dialog where you select the filter you want to submit and provide a brief description of what the filter is for. Please note that the filter will not appear imidentally in the archive, a moderator has to approve it first, which should only take about a day.
You can of course download filters too by clicking the filter and then clicking the Download button to the right. However you have to make sure that you don't have a filter with the same name as the one that you're about to download. The downloaded filter will appear in the filter list, ready for use.
You can also send suggestions about modifications to filter that you feel could be improved via the Suggest modification button.
Via the web
The web interface can be accessed at http://www.lokorin.com/mplf/archive . It provides the same functionality as the interface in the program and should be fairly easy to understand.
External links
- Multipurpose Log Filter - The home of the Multipurpose Log Filter project.
