Log parsing, explained

01 · Definition

Log parsing, defined

Here is one request from an nginx access log, before and after parsing. Each color is one field, and the same colors mark that field in the regex and config further down.

To a log store, the raw line is one string. Counting the 5xx responses from /api/checkout means running a pattern over every line each time you ask. Once url.path and http.response.status_code are fields, the same question is a filter and a group-by, and a status code stored as the integer 502 can be compared with >= 500.

Parsing before export does the work once, close to the source, so every backend receives the same structured, searchable fields and none of them needs its own parsing rules at query time.

02 · Formats

Log formats and parsing techniques

The format of a log decides the parser. Most logs are one of these:

Any text format without a dedicated parser falls back to a regular expression, where each named group, written (?P<name>...), becomes a field. Grok, popularized by Logstash, is regex with a library of ready-made named patterns: %{IP:client} matches an IP address, and %{HTTPD_COMBINEDLOG} matches a whole nginx or Apache line. The next section has a Collector config for each format in the table.

Log parsing tools

Maple stores logs that arrive over OTLP, so this guide parses them in the Collector before they are sent.

03 · Pipeline

Parse logs in the OpenTelemetry Collector

The OpenTelemetry Collector is a free, open-source program that receives logs, metrics and traces, processes them and sends them on to a backend. It is the usual place to parse logs:

• One config parses the logs of every app on the host, so no app needs its own parsing code.
• It works for software you can't change, such as nginx, databases and vendor daemons.
• It is vendor-neutral. Switching backends means changing the exporter, not rewriting the parsers.
• Your logs already pass through it on the way to the backend, so parsing there adds no extra hop.

The file log receiver

Most apps and servers write their logs to a file. The Collector's file log receiver, file_log in the config, watches that file and picks up each new line as it is written. A list of operators parses the line into fields, and an exporter sends the finished record to your backend.

1. Your appwrites log lines
2. access.loga file on disk
3. file_log receiverpicks up new lines, parses them
4. otlp_http exportersends records over OTLP
5. Your backendMaple in this example

The receiver ships in otelcol-contrib, the Collector build published as the Docker image otel/opentelemetry-collector-contrib. Older versions name it filelog and the exporter otlphttp; the configs here use the current names.

Pick your log format

Each tab has a sample line, a complete config.yaml for that format, and the record Collector 0.164.0 printed for the line in a test run.

receivers:
    file_log/nginx:
        include: [/var/log/nginx/access.log]
        resource:
            service.name: nginx
        operators:
            - type: regex_parser
              regex: '^(?P<client_address>\S+) \S+ \S+ \[(?P<time_local>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>[^ ?"]+)[^ "]* HTTP/[0-9.]+" (?P<status>\d{3}) (?P<body_bytes>\d+|-) "[^"]*" "(?P<user_agent>[^"]*)"$'
              timestamp:
                  parse_from: attributes.time_local
                  layout: "%d/%b/%Y:%H:%M:%S %z"
              severity:
                  parse_from: attributes.status
                  overwrite_text: true
                  mapping:
                      info:
                          - min: 100
                            max: 399
                      warn: 4xx
                      error: 5xx
            # Operators run top to bottom, so the renames come after the parser
            # that creates the fields. Dots are not allowed in group names.
            - type: move
              from: attributes.client_address
              to: attributes["client.address"]
              on_error: send_quiet
            - type: move
              from: attributes.method
              to: attributes["http.request.method"]
              on_error: send_quiet
            - type: move
              from: attributes.path
              to: attributes["url.path"]
              on_error: send_quiet
            - type: move
              from: attributes.status
              to: attributes["http.response.status_code"]
              on_error: send_quiet
            - type: move
              from: attributes.body_bytes
              to: attributes["http.response.body.size"]
              on_error: send_quiet
            - type: move
              from: attributes.user_agent
              to: attributes["user_agent.original"]
              on_error: send_quiet
            - type: remove
              field: attributes.time_local
              on_error: send_quiet

processors:
    transform/nginx:
        error_mode: ignore
        log_statements:
            - delete_key(log.attributes, "http.response.body.size") where log.attributes["http.response.body.size"] == "-"
            - set(log.attributes["http.response.status_code"], Int(log.attributes["http.response.status_code"])) where log.attributes["http.response.status_code"] != nil
            - set(log.attributes["http.response.body.size"], Int(log.attributes["http.response.body.size"])) where log.attributes["http.response.body.size"] != nil
    batch: {}

exporters:
    otlp_http/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        logs:
            receivers: [file_log/nginx]
            processors: [transform/nginx, batch]
            exporters: [otlp_http/maple]

Resource attributes:
     -> service.name: Str(nginx)
Timestamp: 2026-10-19 14:02:11 +0000 UTC
SeverityText: INFO
Attributes:
     -> client.address: Str(203.0.113.7)
     -> http.request.method: Str(GET)
     -> url.path: Str(/api/orders/42)
     -> http.response.status_code: Int(200)
     -> http.response.body.size: Int(1532)
     -> user_agent.original: Str(Mozilla/5.0 (X11; Linux x86_64))
     -> log.file.name: Str(access.log)

receivers:
    file_log/app:
        include: [/var/log/app/app.log]
        resource:
            service.name: app
        operators:
            - type: json_parser
              timestamp:
                  parse_from: attributes.ts
                  layout_type: gotime
                  layout: "2006-01-02T15:04:05Z07:00"
              severity:
                  parse_from: attributes.level
            - type: move
              from: attributes.msg
              to: body
              on_error: send_quiet

processors:
    batch: {}

exporters:
    otlp_http/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        logs:
            receivers: [file_log/app]
            processors: [batch]
            exporters: [otlp_http/maple]

Resource attributes:
     -> service.name: Str(app)
Timestamp: 2026-10-19 14:02:11.482 +0000 UTC
SeverityText: error
Body: Str(payment declined)
Attributes:
     -> ts: Str(2026-10-19T14:02:11.482Z)
     -> level: Str(error)
     -> order_id: Str(ord_8812)
     -> attempt: Double(2)
     -> log.file.name: Str(app.log)

receivers:
    file_log/worker:
        include: [/var/log/worker/worker.log]
        resource:
            service.name: worker
        operators:
            - type: key_value_parser
              timestamp:
                  parse_from: attributes.time
                  layout_type: gotime
                  layout: "2006-01-02T15:04:05Z07:00"
              severity:
                  parse_from: attributes.level

processors:
    batch: {}

exporters:
    otlp_http/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        logs:
            receivers: [file_log/worker]
            processors: [batch]
            exporters: [otlp_http/maple]

Resource attributes:
     -> service.name: Str(worker)
Timestamp: 2026-10-19 14:02:11 +0000 UTC
SeverityText: warn
Attributes:
     -> time: Str(2026-10-19T14:02:11Z)
     -> level: Str(warn)
     -> msg: Str(cache miss)
     -> key: Str(user:42)
     -> duration_ms: Str(18)
     -> log.file.name: Str(worker.log)

receivers:
    file_log/checkout:
        include: [/var/log/checkout/app.log]
        resource:
            service.name: checkout
        # A new entry starts at a line beginning with a date. Other lines, such
        # as stack frames, are appended to the entry before them.
        multiline:
            line_start_pattern: '^\d{4}-\d{2}-\d{2} '
        operators:
            - type: regex_parser
              regex: '(?s)^(?P<time>\S+ \S+) (?P<level>[A-Z]+)\s+\[(?P<thread>[^\]]+)\] (?P<logger>\S+) - (?P<message>.*)$'
              timestamp:
                  parse_from: attributes.time
                  layout: "%Y-%m-%d %H:%M:%S,%L"
                  location: UTC
              severity:
                  parse_from: attributes.level

processors:
    batch: {}

exporters:
    otlp_http/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        logs:
            receivers: [file_log/checkout]
            processors: [batch]
            exporters: [otlp_http/maple]

Resource attributes:
     -> service.name: Str(checkout)
Timestamp: 2026-10-19 14:02:11.482 +0000 UTC
SeverityText: ERROR
Attributes:
     -> time: Str(2026-10-19 14:02:11,482)
     -> level: Str(ERROR)
     -> thread: Str(http-nio-8080-exec-3)
     -> logger: Str(c.e.CheckoutController)
     -> message: Str(Checkout failed
java.lang.IllegalStateException: cart is empty
	at com.example.CheckoutService.submit(CheckoutService.java:88)
	at com.example.CheckoutController.post(CheckoutController.java:41))
     -> log.file.name: Str(app.log)

receivers:
    file_log/syslog:
        include: [/var/log/remote/hosts.log]
        resource:
            service.name: syslog
        operators:
            - type: syslog_parser
              protocol: rfc5424

processors:
    batch: {}

exporters:
    otlp_http/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        logs:
            receivers: [file_log/syslog]
            processors: [batch]
            exporters: [otlp_http/maple]

Resource attributes:
     -> service.name: Str(syslog)
Timestamp: 2026-10-19 14:02:11.003 +0000 UTC
SeverityText: crit
Attributes:
     -> priority: Int(34)
     -> facility: Int(4)
     -> facility_text: Str(auth)
     -> version: Int(1)
     -> hostname: Str(web-01)
     -> appname: Str(sshd)
     -> proc_id: Str(2211)
     -> message: Str(Failed password for invalid user admin from 198.51.100.4 port 52113 ssh2)
     -> log.file.name: Str(hosts.log)

The exporter at the bottom of each config sends the records over OTLP, the OpenTelemetry protocol, to Maple in this example. For another backend, change endpoint and headers; the receiver and its operators stay the same. For Maple, the key is the private ingest key (maple_sk_...) from the Ingestion page in Maple's settings, and organizations in the EU region use https://ingest.eu.maple.dev.

Run it

Check the parsing before anything is sent. debug.yaml swaps the exporter for one that prints each record; the Collector merges every --config file it is given, so config.yaml stays unchanged:

# debug.yaml: print parsed records instead of exporting them
exporters:
    debug:
        verbosity: detailed

service:
    pipelines:
        logs:
            exporters: [debug]

Start the Collector with Docker from the folder that holds both files. Mount the folder your log file is in at the path the config's include names; for nginx, that is /var/log/nginx:

docker run --rm \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v "$PWD/debug.yaml:/etc/otelcol-contrib/debug.yaml:ro" \
  -v /var/log/nginx:/var/log/nginx:ro \
  otel/opentelemetry-collector-contrib:0.164.0 \
  --config /etc/otelcol-contrib/config.yaml --config /etc/otelcol-contrib/debug.yaml

The receiver reads lines written after it starts, so send a request to nginx. A 502 from /api/checkout prints as:

Resource attributes:
     -> service.name: Str(nginx)
Timestamp: 2026-10-19 14:02:12 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(198.51.100.23 - alice [19/Oct/2026:14:02:12 +0000] "POST /api/checkout HTTP/2.0" 502 157 "-" "curl/8.5.0")
Attributes:
     -> client.address: Str(198.51.100.23)
     -> http.request.method: Str(POST)
     -> url.path: Str(/api/checkout)
     -> http.response.status_code: Int(502)
     -> http.response.body.size: Int(157)
     -> user_agent.original: Str(curl/8.5.0)
     -> log.file.name: Str(access.log)

When the records look right, start it in the background with your ingest key and without debug.yaml:

docker run -d --name otelcol \
  -e MAPLE_INGEST_KEY=YOUR_INGEST_KEY \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  -v /var/log/nginx:/var/log/nginx:ro \
  otel/opentelemetry-collector-contrib:0.164.0

See the parsed fields

Backends that accept OTLP store the parsed fields as log attributes. In Maple, for example:

1. 01

   Filter and open a record

   Open Logs and pick nginx under Service. Excluding INFO under Severity leaves the 4xx and 5xx responses. Open a record: its Attributes tab lists the parsed fields under Log Attributes.

   

2. 02

   Pin fields as columns

   Use Columns in the table toolbar to pin url.path and http.response.status_code next to the message. For charts, a dashboard's Query Builder over Logs can group by attr.url.path (Build dashboards).

   

04 · Your format

Parse your own log format, step by step

When no tab matches your logs, build the parser yourself. These steps work for any text format, with the nginx config as the worked example:

1. 01

   Copy a few sample lines

   Take real lines from the file, including an unusual one: an error, an empty field, a request from a scanner. For nginx, the line in the overview diagram.

2. 02

   Write a regex with one named group per field

   Match the text between fields literally and capture each field with (?P<name>...). For the nginx line, in the overview's colors:

   ^(?P<client_address>\S+) \S+ \S+ \[(?P<time_local>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>[^ ?"]+)[^ "]* HTTP/[0-9.]+" (?P<status>\d{3}) (?P<body_bytes>\d+|-) "[^"]*" "(?P<user_agent>[^"]*)"$

   \S+ matches up to the next space, [^\]]+ up to the closing bracket and [^"]* anything inside a pair of quotes. The path group stops at ?, which leaves the query string out. Put the regex in a regex_parser operator.

3. 03

   Rename the fields

   Group names can't contain dots, so a move operator renames each one to its OpenTelemetry semantic-convention name, such as status to http.response.status_code. Backends recognize the standard names.

4. 04

   Set the timestamp and severity

   The parser's timestamp block reads the time field with its layout, %d/%b/%Y:%H:%M:%S %z for nginx. Its severity block reads the level; nginx has none, so the mapping derives one from the status: 100 to 399 is info, 4xx is warn and 5xx is error.

5. 05

   Fix types, then check the output

   Every regex group is a string. The transform processor converts the ones you compare as numbers with Int(). Run with debug.yaml and compare each printed record with its sample line before you export.

05 · Best practices

Log parsing best practices and common mistakes

A parser that looks right on the sample line can still lose data in these ways:

06 · Command line

Parse a log file by hand

For one answer from one file, or to work out an unfamiliar format before you write a Collector config, the shell is enough. The commands run on Linux and macOS; the examples use the nginx access log.

1. 01

   Look at a few lines

   Print the first lines, then count the space-separated fields on each line. Different counts mean spaces alone can't split the format: here the request and the user agent contain spaces.

   head -n 3 access.log
   awk '{print NF}' access.log | sort -n | uniq -c

   203.0.113.7 - - [19/Oct/2026:14:02:11 +0000] "GET /api/orders/42?expand=items HTTP/1.1" 200 1532 "https://shop.example.com/cart" "Mozilla/5.0 (X11; Linux x86_64)"
   198.51.100.23 - alice [19/Oct/2026:14:02:12 +0000] "POST /api/checkout HTTP/2.0" 502 157 "-" "curl/8.5.0"
   203.0.113.7 - - [19/Oct/2026:14:02:14 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "https://shop.example.com/" "Mozilla/5.0 (X11; Linux x86_64)"
         1 10
         3 12
         2 15

2. 02

   Split on a separator that holds

   Look for characters that always enclose or separate a field: quotes, brackets, =, braces. In nginx lines the double quotes do, so awk -F'"' finds the status in the same place on every line. JSON and key=value logs bring their own separators:

   Text: awk JSON: jq key=value: grep

   # Split on the double quote: the status is the first word of piece 3
   awk -F'"' '{split($3, f, " "); print f[1]}' access.log | sort | uniq -c | sort -rn

         2 502
         1 404
         1 400
         1 204
         1 200

   # -R reads each line as text, fromjson? skips lines that aren't JSON
   jq -cR 'fromjson? | select(.level == "error") | {ts, msg, order_id}' app.log

   {"ts":"2026-10-19T14:02:11.482Z","msg":"payment declined","order_id":"ord_8812"}

   # The slowest duration_ms in a logfmt log
   grep -oE 'duration_ms=[0-9]+' worker.log | cut -d= -f2 | sort -n | tail -n 1

   240

3. 03

   Test a regex on the whole file

   Write the pattern with named groups and run it over every line. This script prints the fields of the first line and every line that doesn't match. Save it as check_pattern.py and run python3 check_pattern.py access.log:

   import re
   import sys
   
   PATTERN = re.compile(r'^(?P<client_address>\S+) \S+ \S+ \[(?P<time_local>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>[^ ?"]+)[^ "]* HTTP/[0-9.]+" (?P<status>\d{3}) (?P<body_bytes>\d+|-) "[^"]*" "(?P<user_agent>[^"]*)"$')
   
   misses = 0
   with open(sys.argv[1]) as f:
       for number, line in enumerate(f, start=1):
           match = PATTERN.match(line.rstrip("\n"))
           if match is None:
               misses += 1
               print(f"no match on line {number}: {line[:60]}")
           elif number == 1:
               print(match.groupdict())
   
   print(f"{misses} lines did not match")

   {'client_address': '203.0.113.7', 'time_local': '19/Oct/2026:14:02:11 +0000', 'method': 'GET', 'path': '/api/orders/42', 'status': '200', 'body_bytes': '1532', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64)'}
   no match on line 6: 192.0.2.44 - - [19/Oct/2026:14:02:17 +0000] "\x16\x03\x01\x0
   1 lines did not match

   Line 6 is a TLS handshake sent to the plain-HTTP port. In the Collector, a line like it passes through unparsed with its raw text.

4. 04

   Move it into the Collector

   Paste the pattern into regex_parser, in single quotes. Python and the Collector use the same (?P<name>...) syntax, but the Collector's Go regex engine has no lookarounds or backreferences, so leave those out. Then follow steps 3 to 5 of Parse your own log format.